It was Christmas 1994. Some 10 members of my family were gathered at our New Hampshire home. And I offered to get a movie from the video store (remember those) as entertainment the day after Christmas. I brought home Ran, Akira Kurosawa’s 1985 samurai epic based on King Lear, the last and perhaps the greatest movie of that type Kurosawa ever directed.
My choice that day has remained a topic of derision in my family’s oral culture ever since. Even today, family members forcibly prevent me from choosing a movie on Netflix. I still maintain, doggedly, that it was a movie worth seeing.
Ran – the movie that forever sealed my reputation within my family. Image from IMDb.
I was reminded of Ran the other day as I was updating my memory on another of Kurosawa’s classics. In Rashomon, filmed in 1950 with Toshiro Mifune in the leading role, Kurosawa tells a simple tale of a rape and murder. It is widely acclaimed as a cinematic masterpiece, but I probably won’t ever subject my family to a viewing. The particular device used in Rashomon was to tell the simple story three times, each time as a remembrance by one of the people present at the events. The result is three radically different versions. Each is supported by at least some of the ‘facts’ and not contradicted by any of the other ‘facts’ and each is plausible, but the viewer is left not knowing what really happened. And Rashomon brings me to today’s topic: what is reality, really?
Rashomon – Kurosawa’s 1950 classic film. Image from IMDb.
I want to explore reality because we live in a time when Rashomon is perhaps more relevant than it was in 1950. We live in a time in which it is common for different people in the same society to have radically different conceptions of reality. Usually, individuals are quite convinced that their reality is, well, real, and that alternative views are not only incorrect, but deliberately false. I also sense that the rigidity with which people adhere to their views of events is stronger than it used to be in 1950 (although Kurosawa’s three observers were quite convinced of the veracity of their own memories).
But, let me bring this all down to an explicit example: In a TV newscast last month, a journalist was sampling the thoughts of voters assembling for the caucuses that are used in Iowa to select candidates to stand for election to political office. I was stunned to hear one voter say, “Donald Trump is the best President the United States has ever had.” Yes, I have a radically different view on this matter.
Now this voter, a kindly-looking woman, probably a mother with young children who she loved dearly, a law-abiding citizen, with a family and a career, was clearly not a deranged psychopath with several screws loose. But her world view was so different from mine on the topic of Mr. Trump, that I still cannot get over those 5 seconds of video. I felt exactly the same in 2017 when, in the middle of being interviewed, Kelly Ann Conway, then one of Mr. Trump’s staff as the new President, said, approximately, “well, you have your facts, but we have alternative facts.” When it comes to politics, people in many western democracies these days not only differ strongly in their viewpoints, but they have a deep conviction that their views are correct, unassailable, rooted in reality while the differing views of others are not just a different perspective, but an absolutely false distortion of the facts, and probably a deliberate distortion by those dishonest, disloyal, or evil others.
Let’s get away from politics, which has its special problems because there are so many vested interests, and so much money to fund the provision of both information and disinformation. The fact remains that different people can have quite different understandings of reality even when provided with the same information. Why that is so ties into what I want to consider here.
A lot has been written in recent months concerning the ways in which social media channel us into information ghettos by feeding us the kind of information the algorithms judge us to like. We are being deliberately provided with access to a biased range of information in ways that did not happen in the past. This may account for the apparently increased degree of conviction with which we hold our beliefs to be correct. And if this is the case, we who live in democracies should be extremely concerned because this fact opens us up to being manipulated in ways that never existed in the past. But again, that topic is not what I want to explore today, although I do recommend that you support the traditional press by buying subscriptions.
Today I want to explore how it can be that three witnesses to a rape and murder saw completely different stories unfold. Let’s explore how we access the world around us. I’m not thinking about whether we use a library, Wikipedia, social platforms, or our own capacities to explore the world. I’m thinking about how our sensory systems work.
The Human Eye is Not a Camera
The human eye is not a camera. The human visual system, which includes the eye, the optic nerve and the several visual centers within the brain is not a camera. No matter what you have been taught before, the human visual system is not a camera. It is in some ways much more crude than a camera, and in other ways far more sophisticated than a camera. I’d say the same thing about our olfactory system, auditory system, tactile system, and gustatory system except that we have yet to invent camera-like devices that might capture olfactory, auditory, tactile, or gustatory views of the world around us.
The analogy between a camera and the human eye falls apart the moment the light reaches the retina/film.
Photo © Canon.
A camera captures information contained in light from the environment, and, by focusing it with a lens onto a sensitive film or sensor array, produces a two-dimensional image of what is in view. The human eye captures information contained in light from the environment and focuses it onto the retina, but then all hell breaks loose. Because the various sensory cells in the back of the retina do not all respond to light in the same way, and do not simply respond by becoming more excited the more light there is. Nor are they wired up nicely in the optic nerve to take an image captured at the retina to be projected on some light-table analog within the visual cortex. Because the human visual system is not about capturing an image of the world outside and displaying it within the brain.
There are two major ways in which what happens after light reaches the retina differs from what happens after light reaches a camera’s film or sensor array. And there are a few minor ways in which these differ as well. I won’t waste time on the minor differences except to note that we all possess a blind spot almost in the middle of our visual field because our eye is built basackward. (Yes, that is a word…basackward). The retina has the sensory cells arrayed deep within its structure, almost up against the outside wall of the back of the eye, with their sensory axons (that will begin the information’s journey to the brain) interfacing with other neurons within the retina. After a couple of such intra-retinal links, axons emerge to the front of the retina (into the eye), drape themselves across its surface, and plunge through it close to the center of the field of view to get out the back of the eye and become the optic nerve. This is about as far from ‘intelligent design’ as can be imagined because those axons get in the way of light reaching the sensory cells in the retina, and near the fovea (the most sensor-dense part of the retina) there is essentially a hole in the retina through which the axons emerge – ergo a blind spot. Funny thing is, we are quite unaware of this blind spot nearly all the time. And it is smack in the middle of our field of view. Functioning like a camera? If cameras worked this way you could accidentally put your thumb over the lens and it would not be seen in the image!
The two major differences between visual system and camera are that 1) our brain tells the visual system what to attend to, and 2) our visual system has a remarkable capacity, which it uses all the time, to complete patterns based on minimal information from the eye. While a camera passively gathers information about the world outside, our visual system gathers information in an active but biased way; it attends to those aspects of the field of view that the brain wants information about, and it makes stuff up.
On Seeing and Not Seeing
Marshall McLuhan once said, “If I had not believed it, I never would have seen it” referring to the ways in which our brain directs our sensory systems to attend to specific things. To illustrate this, think back to a time when you were expecting to meet a friend outside the library. She has not yet arrived. There are lots of people milling about, and a steady stream of new people coming over the hill, walking towards the front of the library, and you are scanning the crowd. And then you see her, raise your arm to wave, and shout out her name, “Stella!”
Now let’s examine what just happened. How big an image on your retina did Stella make at the time you first picked her out from the crowd? Could your eyes resolve her facial features? How much of her could you see in that crowd, anyway? Maybe just head and shoulders? You saw your friend and waved to her long before she was close enough to actually discriminate her features from any of the other people walking towards you (well, maybe she looked different to the 2.3-meter-tall basketball player in his team uniform, or the clown with a big red mouth and a white face). How did you do that? Your visual system was actively looking for Stella and it used the limited information available to build a picture of her. If someone had asked you to watch that mass of people, but without looking for anyone in particular, you probably would not have noticed Stella until she was a lot closer to you.
Come to think of it, how often have you run up to someone you thought was a good friend, blurted out, “Hi, Joe, you old bugger” and then realized that this person is not Joe at all? And did you feel immensely confused as well as embarrassed? That is a case of your eyes taking the limited information they had and ‘completing’ it incorrectly to signal ‘Joe’. And you were embarrassed because your visual system had told you, with utter certainty, that this is Joe.
There is a delightful psychology experiment which shows how little we notice if we are not “looking for” specific people or things. This experiment asks the subjects (usually University psychology students!) to watch a short video of six people, three in black shirts and three in white, passing basketballs slowly back and forth. Subjects are asked to count the number of times the people in the white shirts pass the ball. If you want to try this test before reading further, go here.
(Spoiler alert.) At the end of the demonstration, the subjects are asked whether they saw the gorilla. Most of them have not a clue what is being asked of them. A re-run of the video shows that halfway through, a man dressed in a gorilla suit wanders into the group from the left, moves around, and exits stage right. Could not be more obvious. But because the subjects were paying attention to how often the people in white passed the basketball, most never even noticed. (The version I linked to has a couple of additional surprises, so even if you already know about the gorilla, it is worth a watch.)
The ability to miss obvious things when you are not “looking for” them, and the ability to “see” your friend in a crowd long before you could possibly really see her are two sides of the same phenomenon – the ability of your brain to tell the visual system what to look for. So is the failure to see the pitcher of milk in a fridge because, on being asked to “fetch the milk,” you have gone to your friend’s fridge looking for a carton of milk. Or my failure to notice the car I backed into a couple of weeks ago because I was in a quiet parking lot and was thinking about other matters as I looked quickly into the rearview mirrors and the backup camera not expecting any cars to be around. That failure was embarrassing as well as expensive. In all these cases, it is not that our eyes malfunction, it is because our brain is using our eyes for specific purposes, or in my case in that parking lot, attending to something other than visual information.
Data Processing vs Capturing an Image
Now, what about that other major difference between our visual system and a camera? I’ve already stated that our retinas do not deliver a pixel-by-pixel image to some light table in the visual cortex. But what do they deliver?
They deliver data. The analysis of the visual information begins in the retina as sensory cells interface with bipolar neurons which then connect with ganglion cells (whose axons become the optic nerve). Individual sensory cells interact with multiple bipolar neurons and these interact with multiple ganglion cells. There are also horizontal cells and amacrine cells which interconnect bipolar and ganglion cells. In other words there is a complex network within the retina that enables complex comparison and inhibition of the signals from nearby sensory neurons. The information is being processed even before it leaves the eye.
This diagram from Encyclopedia Britannica of a small portion of the retina shows the arrangement of sensory cells (the rods and cones), bipolar cells and ganglion cells. It omits the horizontal and amacrine cells that connect across multiple bipolar or ganglion cells. In the diagram, the inside of the retina is at the top and the outside is at the bottom – definitely basackwards.
Once in the brain there is further mingling of data from separate sensory neurons. Data from one eye are even combined with data from the other eye. This process of analysis is carried out in multiple parts of the visual pathway with the result that by the time we come to neurons in the anterior or posterior visual cortex we find cells that are responsive to very specific stimuli in various parts of the field of view. There are cells that respond to edges, to circular objects, to moving objects travelling in particular directions across the field of view. There are even neurons that respond specifically to known faces, Stella’s face, perhaps. It is unclear whether there is any single place within the brain in which all this information is brought together. In fact, that appears unlikely; we build the image of reality by assembling the various bits into a coherent image in our consciousness. (This should perhaps not be surprising. We regularly build stories out of seen symbols we call the alphabet.)
One of my favorite optical illusions nicely illustrates the way our visual system ‘completes’ images. It is one I discovered by accident one day when I was fiddling with an overhead transparency of a pattern of random dots (who remembers the overhead projector, it comes from a time before floppy disks). By chance I had two copies of the same random array and I placed one over the other aligning all the dots. Then I twisted it slightly and an incredible thing happened – a set of concentric circles appeared out of nowhere. Here is the pattern aligned and twisted.
The lower image is the same random array of dots as the upper image, but duplicated and then the alignment of the two sets is twisted slightly.
Thinking logically, the lower image consists of randomly placed pairs of dots. The only circularity in the image is that the pairs are oriented with respect to each other in a direction that depends where relative to the axis of rotation they are located. Pairs above or below the axis of rotation are beside each other while pairs at the left or right of the axis of rotation are positioned one above the other. Each of the concentric circles you see consists of just two dots close together; your visual system has created the rest of the circle. (If I was to animate this image, you’d see the incredible rapidity with which the non-existent concentric circles jump out as the image is twisted. And twist it too far and the circles disappear again – apparently the visual system finds it easier to construct a circle based on two dots close together but cannot do this when they are too far apart.)
But what about the creation of a detailed image of the world outside us? There is complex image analysis going on in the visual system, beginning in the retina and continuing in the brain, but how does our mind take all this analyzed data and create the very real view of the world that we see in front of us? I don’t think neuroscience can yet explain that model building process. But the important thing to know is that it IS a model. We are not inside our heads looking out at the world through our eyes (and out other sense organs), despite the fact that it feels precisely like that is happening. In a very real sense we are in Plato’s cave looking at images of the world cast as shadows on the cave wall, and thinking we are looking at the world itself. (Unlike Plato’s people, we are not even sure where the cave wall, the light table, or the model of reality resides within ourselves.)
I was reminded of how our visual system works to complete, enhance, or generally tidy up the spatial information it gets from the eyes during my hunt for my perfect beach. I found my beach in Australia on 3rd December 2023. When I first saw it I was standing at the top of a stairway down to the shore. The beautifully deserted beach stretched far to my right, and a wave formed and began to crest. The wave was nearly linear and uniformly high, moving in towards the shore first at my left and then to my right as it propagated along the shore. And as the wave began to break right in front of me an eagle ray breached, completely clear of the water including its long tail. What a magical sight! And, needless to say, I could not capture a picture of it despite having my camera in my hands, lens cap on.
Not my eagle ray, this one was photographed off the West Australian coast. Photo © FishtasticScuba.
What happened over the next few hours as I thought back to that wonderful beach was a lesson in photo editing. In my memory now, I have suddenly developed the eyes of an eagle, zoomed in to the cresting wave, and the breaching eagle ray stretches almost completely across my field of view. I can see the droplets of water running down its body and along its tail, each droplet rainbowed by the sun which has partially backlit the ray so it glistens. It hovers in the air a moment before dipping back into the wave. I was at least 100 meters from that ray, and I do not have the eyes of an eagle, but my memory has built a vivid image of an event that has become important to me. An image that is far more real than what I actually saw.
Did the water droplets stream along the underside of the ray’s tail to drip off the end? Were they each rainbowed by the light from the sun, which also backlit the ray beautifully? I do not know. I do not know how much of that scene was decoded from information captured by sensory neurons in my retina and how much was invented by the powerful algorithms that my visual system uses hour by hour, minute by minute as it builds its model of the scene in front of me. Our visual system is a whole lot more wonderful than any camera yet devised (and our other sensory systems, though less developed than our eyes, are similarly marvelous).
Reflecting on how my own sensory equipment works helps me understand how perfectly sane individuals can have completely different perspectives on the world. Start with particular memories (accurate or otherwise), add in some beliefs, add in whatever it is that determines what the sensory systems should be attending to at any given time… I can begin to comprehend how it is possible for perfectly sane people to doubt the reality of climate change, to consider Donald Trump to have been the greatest US President ever, to have complete faith in an economic theory based on maintaining perpetual economic growth on a finite planet. And, comprehending, I am at a loss to provide solutions for how to change peoples’ minds, although I do finally understand why “just giving people the facts” does not work. In many ways, I wish our reality was more real.