[O]ne sees a series of discontinuous dots upon a homogenous ground not as a sum of dots, but as figures. Max Wertheimer [1].

Many current theories for how we perceive and recognize shapes assert that their contour “features” are registered by orientation-selective neurons in primary visual cortex. That activity then passes to extrastriate brain areas wherein neurons respond to specific combinations of features, thus providing for recognition of a given shape. These concepts are not consistent with evidence from this laboratory that shapes can be identified when their boundaries are rendered using a string of widely spaced dots. It is hypothesized that a primitive shape-encoding mechanism summarizes the spatial relations among contour- or dot-marked locations. The visual abilities of vertebrates that have no cortex suggest that the retinotectal system is providing for shape encoding.

Greene E (2016) Retinal Encoding of Shape Boundaries. JSM Anat Physiol 1(1): 1002.

It is widely held that perceptual encoding of shapes begins with registering their contours, i.e., the lines and edges, with the outer boundary of the shape being especially important. It is commonly asserted that orientation-selective neurons of primary visual cortex are needed to register these attributes, and their activation feeds a hierarchical cascade that provides for shape recognition. But a number of reports from this laboratory demonstrate that humans can identify shapes wherein the boundaries are marked with a sparse set of widely spaced dots. Similarly, the continuous strokes that are normally used to produce letters can be replaced by a sparse pattern of dots, yet the letters are readily identified. The brain does not appear to be “constructing” the contour features that have been replaced by sparse dots, so a new theory of shape encoding is required.

Hypothesized cortical mechanisms for shape/pattern encoding

Most theories of shape recognition posit that extended contours, i.e., lines and edges, serve as fundamental units, these being described as features. Ostensibly, a given shape is comprised of an assembly of these features [2-6]. Hubel & Wiesel [7,8] provided evidence that neurons in primary visual cortex (V1) register contours, which is thought to provide the first step toward recognition of shapes. Some neurons respond optimally as a function of the location and orientation of an elongated stimulus, and these “orientation-selective” cells map every location across the visual field [9-11]. It is thought that the selectivity is based on converging anatomical connections that convey alignments of retinal ganglion cells, a model that has received extensive experimental confirmation [12-14].

Numerous neural network models have been proposed on the basis of this general framework and it is beyond the scope of this paper to review those proposals. In general, they can be described as embracing a combinatoric approach, meaning that the response of a given neuron within the system depends on the combination of inputs being provided to it. However, an exceedingly vast number of contour ombinations would be required to register all the shapes, with many variations of location and size, that we are able to identify. This “combinatoric explosion” militates against the idea that shapes are encoded and retrieved from memory on the basis of wired and weighted connections. Further, diverse shapes can be identified using exceptionally sparse dot patterns, wherein the dot spacing is too great to activate the orientation-selective neurons of primary visual cortex.

Recognition of shapes with low-density displays

Greene [15] displayed shapes on an array of LEDs, whereina chain of adjacent dots, similar to a silhouette, represented the outer boundary of each shape. There were 69 shapes in this inventory, depicting animals, vehicles, furniture, tools, and various other complex shapes. Experimental manipulation of the stimuli included display of low-density (sparse) versions, wherein the respondent might be shown only every 5th dot in the sequence, or every 9th dot, or every Nth dot. On average, an 18% density was needed for identification of inventory shapes, and many were identified with far fewer dots. A moth and a woman’s high-heel shoe required only 8% density (13 and 21 dots, respectively). A rooster was recognized with a 7% density (19 dots) and a boot required only 6% density (11 dots). Degree of symmetry did not determine the density required for identification. One might be concerned that the kinds of shapes that were include in the inventory would provide a covert cue for guessing the correct answer, but control tests showed that the probability of getting the answer by guessing was less than 1%.

A similar result was found for recognition of letters of the alphabet. Greene & Visani [16] examined the ability to identify letters with low-density samples that were displayed with a synchronized 10-microsecond flash of the dots forming the letters. These were “fat” letters, wherein the stokes forming the major limbs of each letter where three dots abreast. Mean recognition of letters was in the 80% range with a 25% density, and was still above chance at about 20% recognition with a density of 5%.

Greene [17] manipulated density with “thin” letters, these being formed with single-file chains of dots. Large and small letters were tested. Interestingly, both showed similar probability of recognition as a function of density, even though the large letters had three times the size of small letters. Both were recognition in the 100% range at about 27% density. At a 3% density level the respondents were able to identify 20% of the small letters, whereas there was 30% recognition for large letters. Chance performance was in the 4-7% range (depending on assumptions).

The ability to identify shapes and letters from low-density dot patterns provides a serious challenge to extant models for how shape features are registered by the nervous system. As noted above, the common view is that shapes provide a configuration of contours, e.g., lines and edges, and the visual system registers those contours. Responding selectively on the basis of length and orientation of the contour is especially critical. However, the lowdensity patterns are not providing any contours, per se, and it is doubtful that a neuronal basis exists for reconstructing a contour from the sparse dots that are able to elicit recognition of a given shape.

Sceniak et al. [18], examined the orientation-selective neurons in macaque V1 and found the length of the excitatory fields for 30 of 31 macaque neurons to be less than 2.5 arco . It is unlikely that receptive field length in humans is substantially longer than this. Greene [15] found that shapes could be identified when the spacing between adjacent boundary dots was much larger than this, i.e., greater than the longest receptive field of orientationselective neurons. Further, the recent work with letter recognition found that large letters at 3% density had a 3.6 arco mean spacing between adjacent dots [17]. Nonetheless, the 3% patterns elicitedrecognition on about 30% of the trials, this being well above chance. For patterns having such large separations between dots, no more than one dot would fall within the receptive field of an orientation-selective neuron. This likely would not be sufficient to activate cell firing. Each dot has a diameter of about 5 arc’ and the response of large orientation-selective cells is minimal for such small stimuli (Mosh Gur, personal communication). Even if a response could be elicited, the dot would not provide any information about the relative orientation of adjacent dots.

One can reverse the approach by sequentially displaying small subsets of dots that are randomly located around the boundary of a shape, versus subsets having close proximity that should activate orientation-selective neurons. If activation of orientation-selective neurons is a first step in the shape-encoding process, the later condition should be more effective at eliciting recognition. Results of this experiment show that the two conditions differ very little in their ability to elicit recognition of shapes [19]. If subsets containing randomly positioned dots are as effective as subsets that should activate orientation-selective neurons, this argues against the concept that orientation and linear extent are essential cues for shape recognition

Identifying unknown shapes from low-density displays

Letters of the alphabet are clearly over-learned, and one might wonder whether this makes it possible to identify a shape from a low-density display. Some new work from this laboratory, not previously reported, indicates that shapes are encoded quickly, and in a form that allows for immediate recognition on the basis of low-density samples of the boundary markers.

The experiment presented novel, i.e., unknown, shapes, such as the example shown in Figure (1). There were 450 of these shapes in the test inventory. This was a match-to-sample task, wherein two patterns were displayed, a “target” and a “choice option,” and the respondent had to decide whether or not the two were the same. Here the target display provided the full complement of boundary dots and the choice option was a lowdensity version of the target shape, designated as “matching, or a low-density version of a different shape, designated as “nonmatching.”

Eight respondents were tested individually. Each was successively shown 300 target shapes that were randomly selected from the inventory. Half of the trials provided “matching” choice options, these being low-density versions of the target shape. On the other 150 trials a low-density “non-matching” pattern was shown as the choice option. The non-matching choice options were the shapes that had not been chosen as targets. The target was shown in one of the quadrants of the display board and the choice option was shown in the opposite diagonal quadrant. Each target and choice option was displayed for 500 ms, with a 300 ms interval between displays.

Density levels for choice options were varied from 3% to 25%. Respondents were asked to say whether the choice option was derived from the target shape or not. Note that the experimental design provided only a one-time exposure to a given shape for only half-a-second, requiring that the shape be encoded by the visual system sufficient to decide if a low densitypattern provided a comparable rendering of the shape outline.

Figure 2A shows regression models for the eight respondents, with probability of correct response being plotted against dot density. Figure 2B shows the mean regression model alongwith the confidence interval for the model. At 25% density the decisions were roughly correct for 9 out of 10 trials, and each respondent manifested a linear decline in correct responding as the density was reduced. At 3% density the chance of being correct was about 65%, where chance performance was 50%. The decline in recognition was significant at p < 0.0001, and correct responding for the group was above chance at each level of density (p < 0.0001).

It is clear that complex shapes can be encoded within moments, and a decision of whether a subsequent display is a match requires display of only a small percentage of boundary markers. This supports earlier evidence [20] that each marked location makes an independent contribution to shape recognition, acknowledging that neighboring locations will have some degree of redundancy.

Shape recognition as a prerequisite for evolutionary success

Beyond the issue of how the nervous system registers basic shape features, there is the question of how those features are combined to provide a shape summary that can be stored and recalled. The aggregation of elemental features is generally ascribed to extrastriate regions, such as the lateral occipital complex and inferotemporal cortex [21-24]. The structure of the hierarchical models that have been developed to describe this process apparently assume that complex cortical circuits are required to accomplish the task. But if that were true, how could countless generations of vertebrates have survived and propagated their respective species across hundreds of millions of years prior to the evolution of mammalian cortical circuitry? Surely the ancestral species had evolved visual mechanisms for registering and identifying objects in order to more effectively interact with their environment.

We cannot evaluate the perceptual skills of animals that are long dead, but we can at least discuss the abilities of current species. Fish are clearly able to identify the shapes and patterns of their own species, as well as those of potential predators. Karplus and associates [25] reported that damselfish are able to distinguish between predators and non-predators on the basis of the shape of the mouth and the distance between the eyes. Siebeck’s laboratory [26] provided evidence that discrimination of the facial patterns is based on shape cues, not color cues. In the absence of evidence to the contrary, one might presume that the recognition of species-specific patterns is innate rather than learned. However, juvenile coral reef fish do not recognize predators whereas adults do [27], suggesting learning of the facial characteristics of the predators. Further, damselfish rapidly learn to discriminate various kinds of 3D stimuli [28]. Whether the pattern to be identified is innate or learned, the information must be distilled into a code that allows for recognition.

Retinotectal mechanisms for processing shape cues

Vision in fish, amphibians, and reptiles is mediated by the retina and optic tectum, which is homologous to the superior colliculus in mammals. Retinal loci project with good spatial precision to the superficial layers of the tectum [29,30]. Connections from this sensory map to deep motor layers providefor control of motor orienting responses to prey and predators [31].

Neurons in retina and tectum respond vigorously to small spots of light. Also, there are neurons that selectively respond to the direction of stimulus motion [30], orientation of gratings [32] and stimulus size [33]. Mechanisms for mediating predation and avoidance of predators have been found in larval zebrafish. The retinal ganglion cells that respond to small stimuli, such as a paramecium, map to circuits in tectum that elicit approach, whereas retinal ganglion cells that respond to large stimuli elicit avoidance [33,34].

However, the encoding mechanisms that have thus far been found cannot explain the shape and pattern discriminations that have been documented [25-28]. The shapes and patterns would have to be recognized irrespective of location within the visual field, and irrespective of changes in size. It is most plausible that the spatial relations among pattern markers, including those at the shape boundary, are collapsed into a summary that distinguishes a given pattern or shape. Once the initial distances among markers are combined into the summary, it could be normalized so that the same pattern or shape could be identified irrespective of size differentials. Also, having a summary that identifies the shape or pattern allows recognition or discrimination irrespective of where in the visual field the markers lie.

There is no known mechanism for registering the spans among widely spaced stimuli. As described above, shapes can be identified with dot-spans that are too large to be registered within the receptive fields of retinal ganglion cells, but the responses of wide-field amacrine cells might have a role in the encoding process. Greene [35,36] suggested that polyaxonal amacrine cells might register distances and relative orientation among stimulated loci. Localized stimulation of these neurons produces a spreading wave of action potentials, and the travel time of the waves to the centroid of the shape/pattern could provide a summary of the shape or pattern.

While it is possible that a shape/pattern summary would be generated in the tectum, at this point it seems more likely that the encoding is being done within retinal circuits. The retinotectal projections of mammals have various amounts of crossed and uncrossed optic nerve fibers that provide for binocular responses from each half of the superior colliculus. For primates (including humans) the left colliculus registers the right half of the respondent’s visual field and the right colliculus registers the left half of the visual field, as is the case for mapping into V1 [37]. Therefore a stimulus that provides pattern or boundary markers on both sides of the midline will register on opposite sides of the brain. At present there is no known mechanism for registering spans between or among marked locations in the two hemispheres, either in V1 or in the colliculus. The more conservative hypothesis is that the wide arborization of amacrine cells in the retina provides for encoding of spans and orientations among the markers. See [36] for additional discussion of this hypothesis.

Each half of the tectum in fish, amphibians, and reptiles does register the full visual field of the contra lateral eye. Therefore one might argue that in these vertebrate classes the tectumprovides for encoding of spatial relations, a function that has been replaced by cortical mechanisms in mammals. However, the anatomy of mammalian colliculus to higher brain centers suggests a continued role for stimulus encoding that passes through or is provided by tectal circuitry. In mammals the superior colliculus projects to extrageniculate visual thalamus [38], and also the pulvinar nucleus, which then connects to several ventral- and dorsal-stream visual regions [39-41]. One finds that even in a classic visual relay, the lateral geniculate nucleus of primates, the koniocellular layer is innervated by projections from the colliculus [42]. It has been suggested that the collicular projections provide the anatomical substrates for blindsight [42,44], instinctive behavioral responses [45], and rapid processing of socially relevant stimuli such as faces [46,47].

The interface of old and new mechanisms

There can be little doubt that neuronal systems contain substantial redundancy. This is especially true for the brains of mammals, where more newly evolved systems have been added to primitive ones. The older systems clearly provided early ancestors with an ability to effectively interact with the environment sufficient to survive and reproduce. Newer systems might expand the analysis of sensory information, provide better discrimination of alternative stimuli, and connect those discriminations to a larger repertoire of effective responses. But in the absence of evolutionary pressure to eliminate the older systems, they would be retained, perhaps as backup or to handle simpler stimulus conditions for which they were originally developed. This may well be the case for the relation of superior colliculus to primary visual cortex. Where the object to be identified contains not only an outer contour boundary, but also numerous interior contours and patterns, the summary may need to draw on more elaborate cortical mechanisms.

It is likely that primary visual cortex has highly evolved mechanisms for processing subtle and complex shape cues. These might include gray-on-gray boundaries and very thin lines. Being able to register small undulations or differentials in granularity of a contour might be relevant to survival. There is likely a benefit to being able to bridge gaps between collinear contour segments, or extrapolate a contour across open space. In short, the design of receptive fields in V1 and their connectivity can improve the ability to register contours, which in real-world images generally have linear extent. Nonetheless, the ability of sparse dot subsets to elicit identification of diverse shapes and letters indicates that contours are generally over-determined, meaning that the much of the information along their extent is redundant [20].

If a portion of space is surrounded by a line or edge, that space will often be seen as having a shape. The same is true if the space is surrounded by a set of discrete dots. The contrast that is provided by the line, edge, or dots serves to functionally mark locations on the retina. The hypothesis advanced here is that the spatial relations among these marked locations are summarized, and this summary is used to discriminate or recognize the shape even if the density of markers is reduced, or when they are displayed at a different location within the visual field.

At present we have no clear evidence for how the spatial relations among markers are registered and summarized. A shape may be identified when displayed using widely spaced dots, so neurons having very large receptive fields, such as retinal amacrine cells, may have a role in encoding the spatial relations. If so, this could explain the shape and pattern discrimination abilities of a number of vertebrate species that have no cortex.

Display equipment was designed, fabricated, and made operational by Jack Morrison, Digital Insight. He also wrote the applications for running the experiment. Respondents were tested by Joseph Younis. Statistical analysis was done by Dr. Wei Wang, Department of Medicine, Harvard Medical School and Brigham and Women’s Hospital.

Research funding for this laboratory was provided by the Neuropsychology Foundation and the Quest for Truth Foundation.

1. Wertheimer M, Ellis WD. Laws of organization in perceptual forms. A Source Book of Gestalt Psychology. London: Routledge & Kegan Paul, Ltd. 1938; 71-72.

2. Selfridge OG, Cherry C. Pattern recognition and learning. Information Theory. New York: Academic Press. 1957; 345-353.

3. Selfridge OG. Pandemonium: a paradigm for learning. Mechanisms of Thought Processes. London: National Physical Laboratory Symposium. 1959; 513-530.

4. Lindsay PH, Norman DA. Human Information Processing. Academic Press. 1977.

5. Marr D. Vision: A computational investigation into the human representation and processing of information. New York: Freemen. 1982; 51-79.

6. Grainger J, Rey A, Dufau S. Letter perception: from pixels to pandemonium. Trends Cog Sci. 2008; 12: 381-387.

7. Hubel DH, Wiesel TN. Receptive fields of single neurons in the cat’s striate cortex. J Physiol. 1959; 148: 574-591.

8. Hubel DH, Wiesel TN. Receptive fields, binocular interactions and functional architecture in the cat’s visual cortex. J Physiol. 1962; 160: 106-154.

9. Hubel DH, Wiesel TN. Ferrier lecture. Functional architecture of macaque monkey visual cortex. Proc R Soc Lond B Biol. Sci. 1977; 198: 1-59.

10.Blasdel GG, Salama G. Voltage-sensitive dyes reveal a modular organization in monkey striate cortex. Nature. 1986; 321: 579-585.

11.Bonhoeffer T, Grinvald A. Iso-orientation domains in cat visual cortex are arranged in pinwheel-like patterns. Nature. 1991; 353: 429-431.

12.Schwartz EL. Spatial mapping in the primate sensory projection: analytic structure and relevance to perception. Biol Cybern. 1977; 25: 181-194.

13.Carandini M, Demb JB, Mante V, Tolhurst DJ, Dan Y, Olshausen BA, et al. Do we know what the early visual system does? J Neurosci. 2005; 25: 10577-10597.

14.Xu X, Bosking WH, White L, Fitzpatrick D, Casagrande VA. Functional organization of visual cortex in prosimian bush baby revealed by optical imaging of intrinsic signals. J Neurophysiol. 2005; 94: 2748- 2762.

15.Greene E. Recognition of objects displayed with incomplete sets of discrete boundary dots. Percept Mot Skills. 2007; 104: 1043-1059.

16.Greene E, Visani A. Recognition of letters displayed as briefly flashed dot patterns. Atten Percept Psychophys. 2015; 77: 1955-1569.

17.Greene E. Information persistence evaluated with low-density dot patterns. Acta Psychol. 2016; 170: 215-225.

18.Sceniak MP, Hawken JJ, Shapley R. Visual spatial characterization of macaque V1 neurons. JNeurophysiol. 2001; 85: 1873-1887.

19.Greene E. Additional evidence that contour attributes are not essential cues for object recognition. Behav Brain Funct. 2008; 4.

20.Greene E, Ogden RT. Evaluating the contribution of shape attributes to recognition using the minimal transient discrete cue protocol. Behav Brain Funct. 2012; 8: 53.

21.Keysers C, Xiao DK, Foldiak P, Perrett DI. The speed of sight. J Cog. Neurosci. 2001; 13: 90-101.

22.Hung CP, Kreiman G, Poggio T, DiCarlo JJ. Fast readout of object identity from Macaque inferior temporal cortex. Science. 2005; 310: 863-866.

23.Rolls ET, Franco L, Aggelopoulos NC, Jarez JM. Information in the first spike, the order of spikes, and the number of spikes provided by neurons in the inferior temporal visual cortex. Vision Res. 2006: 46: 4193-4205.

24.Golarai G, Ghahremani DG, Whitfield-Gabrieli S, Reiss A, Eberhardt JI, Gabrieli JDE, et al. Differential development of high-level visual cortex correlates with category-specific recognition memory. Nature Neurosci. 2007; 10: 512-522.

25.Karplus L, Goren M, Algom D. A preliminary experimental analysis of predator face recognition by Chromis caeruleus (Pisces, Pomodentridae). Z Tierpsychol. 1982; 58: 53-65.

26.Siebeck UE, Parker AN, Sprenger D, Mathger LM, Wallis G. A species of reef fish that uses ultraviolet patterns for covert face recognition. Curr Biol. 2010; 20: 407-410.

27.Karplus L, Katzenstein R, Goren M. Predator recognition and social facilitation of predator avoidance in coral reef fish Dascyllus marginatus juveniles. Mar Ecol. ProgSer. 2006; 319: 215-223.

28.Siebeck UE, Litherland L, Wallis GM. Shape learning and discrimination in reef fish. J Exper Biol. 2009; 212: 2113-2119.

29.Holt CE, Harris WA. Position, guidance, and mapping in the developing visual system. J Neurobiol. 1993; 24: 1400-1422.

30.Gabriel JP, Trivedi CA, Maurer CM, Ryu S, Bollmann JH. Layer-specific targeting of direction-selective neurons in the zebrafish optic tectum. Neuron. 2012; 76: 1147-1160.

31.Gahtan E, Tanger P, Baier H. Visual prey capture in larval zebrafish is controlled by identified reticulospinal neurons downstream of the tectum. J Neurosci. 2012; 25: 9294-9303.

32.Nikolaou N, Lowe AS, Walker AS, Abbas F, Hunter PR, Thompson ID, et al. Parametric functional maps of visual input to tectum. Neuron. 2012; 76: 317-324.

33.Preuss SJ, Trivedi CA, vom Berg-Maurer CM, Ryu S, Bollmann JH. Classification of object size in retinotectal microcircuits. Curr Biol. 2014; 24: 2376-2385.

34.Abbas F, Martin MP. Fish vision: size selectivity in the zebrafish retinotectal pathway. Curr Biol. 2014; 24: 1048-1050.

35.Greene E. Retinal encoding of ultrabrief shape recognition cues. PLOS

36.Greene E. How do we know whether three dots form an equilateral triangle? JSM Brain Science. 2016; 1-1002.

37.Chalupa LM, Lia B. The nasotemporal division of retinal ganglion cells with crossed and uncrossed projections in the fetal rhesus monkey. J Neurosci. 1991; 11: 191-202.

38.Abramson BP, Chalupa LM. Multiple pathways from the superior colliculus to the extrageniculate visual thalamus of the cat. J Comp Neurol. 1988; 271: 397-418.

39.Glendenning KK, Hall JA, Diamond IT, Hall WC. The pulvinar nucleus of Galago senegalensis. J Comp. Neurol. 1975; 161: 419-458.

40.Lyon DC, Jain N, Kaas JH. The visual pulvinar in tree shrews II. projections of four nuclei to areas of visual cortex. J Comp. Neurol. 2003; 467: 607-627.

41.yon DC, Nassi JJ, Callaway EM. A disynaptic relay from superior colliculus to dorsal stream visual cortex in macaque monkey. Neuron. 2010; 65: 270-279.L

42.Hendry SHC, Reid RC. The koniocellular pathway in primate vision. Ann Rev Neurosci. 2000; 23: 127-153.

43.Brindley GS, Gautier PC, Lewin W. Cortical blindness and the functions of the non-geniculate fibers of the optic tract. J Neurol Neurosurg Psychiatry. 1969; 32: 259-264.

44.Pasik T, Pasik P. The visual world of monkeys deprived of striate cortex: effective stimulus parameters and the importance of the accessory optic system. Vision Res. 1971; 3: 419-35.

45.Sewards TV, Sewards MA. Innate visual object recognition in vertebrates: some proposed pathways and mechanisms. Comp Biochem Physiol A Mol Integr Physiol. 2002; 132: 861-891.

46.Johnson MH. Subcortical face processing. Nature Rev Neurosci. 2005; 5: 766-774.

47.Nguyen MN, Matsumoto J, Hori E, Maior RS, Tomas C, Tran AH, et al. Neuronal responses to face-like and facial stimuli in the monkey superior colliculus. Front Behav Neurosci. 2014; 10: 00085.