Image <=> Imagine <=> Intelligence

Preface:

• The first step is to build a unified theory that can explain the observations in neuroscience. Thus I'm interested in the major theories of brain, such as infomax, sparse coding, attention and bias competition, perceptual learning and memory as well as the Bayesian brain hypothesis.

• The next two steps will iterate so as to steadily improve the unified theory. I'm also interested in principle problems of those theories mentioned above when they are applied in applications. These problems may indicate what the theory cannot explain and is thus worth refining. I'm working on semantic segmentation in which a major branch is energy-based semantic segmentation, grouping together pixels belonging to a semantic-meaningful object in an image via minimizing a defined energy function. The way of energy minimization is similar to how the brain-related theories mentioned above work. Thus it is worth exploring the potential of energy-based semantic segmentation as an application on brain theories, a unified theory combining these theories and its improvement via testing it, in the form of computational models, on the application of semantic segmentation. The comparison of the segmentation results between the computational model and the experimental psychology will show where the theory doesn't cover.

Chapter 1: Retina and LGN -- the Factory of INCEPTION?

	Project 1. Receptive Field for Salient Object Segmentation in Nano Scale
	Description: Existing segmentation methods often fail in noisy data (SNR ranging from 0.01 to 0.1), such as nano-scale data with intrinsic imaging limitations. This work explores the functions of simple receptive field model in retina and LGN for three-dimensional (3D) salient object localization in nano scale. The efficiency of this model is demonstrated on 600x1400x432 cry-electron tomograms, for which our method yields satisfied membrane segmentation that is similar to the segmentation of the state of the art specifically designed for nano-scale membrane segmentation. (Note: orientation-sensitive receptive field from V1 is involved to produce accurate segmentation)
	Publication: Chapter 3 of my dissertation

		Project 2. Depth of Field for Three-Dimentional Reconstruction
		Description: In LGN, temporal correlations/decorrelations (i.e.: focus-of-attension shift) and spatial correlations (i.e.: receptive field with different scale/depth) from both eyes are explored to benefit a 3D representation of object space. In this work, we explore the depth of field by developing a prototype system that allows efficient acquisition of 3D drosophila reconstraction. The system first acquires (multiple) microscopic image stacks and then estimates the underlying surface by estimating a range image for each stack based on the estimation of focusing. The range image is then segmented into different pieces of biological significance and parametric shape models (thin plate spline models) are derived to characterize the underlying surface of each component. We demonstrate the effectiveness of the proposed method by extracting the 3D shape of eyes and other parts from images stacks.
		Publication: Section 4.2 of my dissertation

Chapter 2: Visual Cortex -- the Battle between BRIGHT and PREJUDICE?

	Project 3. Reducing the Dimensionality of Data with a Hybrid Model
	Description: The feaure space in the visual cortex consists of feature responses from receptive fields organized in a hierachical or deep architecture. On one hand, simple receptive fields in low level capture information from visual input; on the other hand, more complicated receptive fields in higher level are more adaptive to prior information from Inferotemporal Stream and Parietal Stream. Thus the feature space is maintained in an optimized manner. In order to explicitly optimize discrimination performance in a more generative way, a hybrid dimension reduction model combining principle component analysis (PCA) and linear discriminant analysis (LDA) is proposed in this work. We also present a dimension reduction algorithm correspondingly and illustrate the method with several classification experiments. Our results have shown that there exists an optimized solution from the hybrid model that outperforms PCA, LDA and the combination of them in two separate stages.
	Publication: Nan Zhao, Washington Mio, and Xiuwen Liu. "A hybrid PCA-LDA model for dimension reduction." Neural Networks (IJCNN), The 2011 International Joint Conference on. IEEE, 2011. [Link][PDF][Poster]

		Project 4. Re-initilization Free Level Set Segmentation Based on Prior Propagation
		Description: Similar to Project 3, this work explores feature space optimization based on a hybrid model consisting of both low level features from visual input and high level features from prior information. However, the feature optimization is implicitly carried out on feature-based energy and is tested on segmentation of HIV membrane in 3D cryo-electron tomogram. Such energy consists of contributions from gradient information, shape prior of ideal level set function and spatial prior from nearby slices. Our results have shown that the optimizatoin of the feature based energy function also allows us to extract ideal segmentation in 3D.
		Publication: Sections 4.3, 4.4 of my dissertation

Chapter 3: Inferotemporal Stream (AIT/CIT/PIT) -- the library of objects?

Chapter 4: Parietal Stream (PP/MST/MT) -- THE MATRIX that correlates objects?

	Project 5. Nano-Scale Context-Sensitive Semantic Segmentation
	Description: Small object segmentation is still a challenging problem, especially when encountering low SNR, low contrast and large data size. It is hence a typical problem of big data. Following Project 1, this work proposes a new context-sensitive method for segmenting 3D volumes. By using robust context cues (relationships/interactions) between objects to efficiently narrow the search space of the target object, we achieve tractable and reliable nano-scale semantic segmentation. We demonstrate our method on a 600x1400x432 tomogram for segmenting microvilli spikes (12x12x20 in general), for which our method yields accurate spike segmentation in 1~2 hours, whereas the state-of-the-art semantic segmentation methods fail due to their inability to handle problems mentioned above.
	Publication: Nan Zhao and Xiuwen Liu. "Nano-Scale Context-Sensitive Semantic Segmentation." International Conference on Image Processing, 2015. (Oral Presentation) [Link][PDF]

		Project 6. Context-Sensitive Tattoo Segmentation and Classification
		Description: In object-centered semantic segmentation, information from the background is considered as noise. However, for some objects, it is difficult to described them in terms of their own appearance characters. In this work, we developed a context-sensitive semantic segmentation algorithm for extracting tattoos in an image. Segmentation of tattoo with unknown number of connected components and arbitrary shapes is transferred to a figure-ground (tattoo-skin) segmentation. We also applied our segmentation results on multiple tasks of tattoo classification and demonstrated the state-of-the-art performance using our algorithm.
		Publication: Allen, Josef D., Nan Zhao, Jiangbo Yuan, Xiuwen Liu. "Unsupervised tattoo segmentation combining bottom-up and top-down cues." SPIE Defense, Security, and Sensing. International Society for Optics and Photonics, 2011. [Link][PDF]