Learning is Predicting

This post is a recap of an Ignite talk delivered by Aptima Corporate Fellow Webb Stacy, PhD.  Ignite talks are ITSEC’s shorter version of a TEDTalk – a fast, fun 5-minute presentation that focuses on a fascinating, thought-provoking topic.  Here, Webb explains how it is that prediction is central to human learning, how prediction is an increasingly pervasive model in modern neuroscience, and the implications for improving learning and performance.

In the split second when a baseball leaves the pitcher’s hand, how does a batter know to swing or not? Or, when approaching an aircraft carrier, what do expert pilots do differently to anticipate and line-up a safe landing compared to the novice?  These and other examples show that at the heart of human performance and learning is prediction.

We’ll explore what that means for the two ends of the learning curve – expert and novice.  We’ll Look at how modern neuroscience understands prediction and the brain, and what the implications are for training and performance measurement.

Learning as prediction happens in many domains.

In baseball, batters have about a quarter of a second to decide whether to swing at a pitch, and if so, where to swing to make contact with the ball.

It turns out that batters cope with this situation with prediction—they learn to predict the kind of pitch that will be thrown. This goes well beyond pitchers who give away information with explicit behaviors. The prediction the batters learn is mostly implicit.

Which leads to the question, can you improve this prediction, and if so, how? 

We’ll discuss this more, but the basic idea is to:

• show the student a video of a pitcher making a pitch
• stop the video before the pitch arrives, and
• ask the student what kind of pitch it is going to be

Using this technique, called Temporal Occlusion Training, you can explicitly and directly train prediction. Unsurprisingly, students can learn to do this.  When one researcher¹ (Peter Fadde) used this technique on a college baseball team, batting averages went from .187 to .274

The takeaway:  Baseball batters learn to encode the future into their representation of pitches.

Carrier landing is a difficult task. 

U.S. Navy photo by Photographer’s Mate 3rd Class Christopher B. Long (RELEASED)

The ‘runway’ is small and moving, not just in the direction of travel, but also as the ocean affects level of the runway.

Pilots train long and hard to be able to accomplish carrier landings. Experienced pilots anticipate the way their landing will unfold with a skill they call proactive flying.

With proactive flying, they take into account their current position in space, the current energy state of the aircraft, and environmental conditions to predict what adjustments they need to make as they land. They position themselves to make those adjustments with maximum smoothness.

Image © 2019 Aptima, Inc.

So what does it mean to proactively fly?  To assess that my colleagues and I conducted an experiment² in 2014 with expert and novice Navy pilots.

We showed them a series of 8-second videos of some portion of a landing (recorded from a simulator), and asked them to predict whether a carrier landing situation required a standard or aggressive correction. This was a way of asking them whether or not they thought they were in good shape for the landing.

Prediction in Carrier Landing

Image © 2019 Aptima, Inc.

In the graph above, the green bars show reaction time—lower is better. The red lines show percent correct (as judged by a panel of Subject Matter Experts) – the higher the better.

The takeaway is not surprising:  Experts (on left) were faster and more accurate than novices. What this means: Pilots learn to encode the future into their cognitive representation of the landing. We looked at these variables before and after. Both groups benefitted from training after a simulator-based training session.

Example:  Cancer Screening

Now let’s consider a domain different from the others: medicine. 

In a study by Brennan³ and colleagues this year, Radiologists were asked to predict whether patients had breast cancer from just a ½ second glimpse of mammogram images. 

Obviously, radiologists are more careful than that when they interpret real mammograms—but this was about prediction, not diagnosis.

They were shown images with tumors and normal images, but were also shown images that would develop cancer but did not show it yet, or showed it so little that it had been missed initially.

Image courtesy Brennan, Gandomkar, Ekpo, Tapia, Trieu, Lewis, Wolfe, & Evans (2018).3 [Annotations added]

The red box outlines the groups that did not have obvious lesions but were going to develop cancerous tumors.

For those conditions, the radiologists were much better than chance. This is even more remarkable when you consider they only saw the images for ½ second.

The takeaway:  Skilled radiologists encode future tumors into their representation of mammograms.  

Meaning, that after reading thousands of mammograms, a radiologist gains an intuition or prediction of what may happen (although not as well as a diagnosis).

So what’s happening in the black box of the brain?  Recent work in neuroscience supports the idea that the brain is a predictive machine. 

Lisa Feldman Barrett, professor of Neuroscience at Northeastern⁴, said this:

Your brain is predictive, not reactive…all your neurons are firing constantly, stimulating one another at various rates … this brain activity represents … millions of predictions of what you will encounter next in the world, based on your lifetime of past experience.

…and she is not alone. Many cognitive neuroscientists are beginning to focus on the extent to which the brain functions by making predictions.

Camera Theory of Vision

Image courtesy Mads00 [CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0)]. https://commons.wikimedia.org/wiki/File:Neural_pathway_diagram.svg

For years, people thought that vision and brain processes worked in a bottom-up fashion, similar to a digital camera.

Meaning, light hit the “sensor” (the retina), and the sensor array was communicated wholesale to the next “image processing” waypoint (the lateral geniculate nucleus), which added some processing and sent the whole data set to the main “visual computer” (the visual cortex).  The metaphorical “photographer” (the rest of the brain) then proceeded to make sense of the “developed image.”

Not so. We now know this is not how the brain works. 

The Brain is a Top-Down Organ

There is a very strong top-down component.

There are many more downstream connections in the brain than there are upstream connections. This strongly suggests that higher level processing must be somehow influencing lower level processing.

Higher brain centers must be influencing the visual cortex, which in turn must be influencing the lateral geniculate nucleus, which may even be influencing the retina.

The best candidate to describe this influence is prediction: higher-level brain centers predict what lower level centers should be experiencing, and the lower levels only need to communicate any deviation from the prediction.

This scheme is a considerably more efficient way for the centers in the brain to communicate with each other.

Image © 2019 Aptima, Inc.

On the left is a depiction of the increasingly burdensome communications that would have to take place in a bottom-up scheme.

On the right is a depiction of how the top-down communications might look.

• If there is no change in prediction for a node, and if the prediction has been correct, there is no need to communicate either up or down. (far right)
• If there is a new prediction for a lower-level node, that prediction is communicated by the higher-level node. (middle?)
• If a lower-level node’s prediction is wrong, the error is communicated to this higher-level node. (left?)

The result is a much lower requirement for communication bandwidth (and therefore biological energy) in the brain.

• Prediction is more efficient than purely bottoms-up processing
• There’s only a need to convey predictions and errors, not the full sensory array
• Much lower “communication” cost: e., fighter pilots that don’t say anything to teammates if everything is going well.

So, what are the implications for training? 

The biggest one is to consider direct prediction training. We know that predictive skill develops in most (or all) cases, but it is generally treated as a side-effect.

Image © 2019 Aptima, Inc.

Why not train it directly? 

This would allow the development and refinement of more dynamic knowledge, skills, and abilities as the predictive models develop.

So how do we train prediction directly?

Direct Prediction Training: Occlusion

Photo courtesy DVIDS https://www.dvidshub.net/image/2784997/59th-mdw-upgrades-flight-training-technology-advances-pilot-safety (Released)

One approach that we discussed briefly is occlusion training. 

The idea is to:

• Present the to-be-predicted environment with some portion blocked or occluded.
  • Time – temporal occlusion, as in the baseball example of the just released pitch.
  • Space – spatial occlusion, as in the map above, to infer activity in the masked area.
  • Other – cue occlusion, context occlusion, others limited by imagination
• Ask the student to make predictions about the occluded aspect.
• Give the student feedback.

But how do we measure the quality of the student’s prediction?

There’s three broad categories of prediction quality. The first, and probably the most obvious, is accuracy.

Image © 2019 Aptima, Inc.

In the diagram above, we see a performance surface describing predictions in three stages of learning—on the left is a novice, on the far right (high peak) an expert, and in the middle, between those two.

Accuracy, then, is a measure of how close the prediction error is to 0, on average.

But accuracy isn’t the whole story.

Another important aspect of prediction quality is prediction variability. This is a measure of the consistency of the student’s predictions.

Image © 2019 Aptima, Inc.

We can see in the diagram that novice variability is high but expert variability is low. The expert is more consistent than the others.

It is desirable both to be accurate and to have low variability. This is what experts develop.

A final category to measure prediction quality is the prediction’s time horizon.

That is, how far into the future can the student predict or see ahead?

In the diagram below, we see the expert can predict further into the future than the novice.

Image © 2019 Aptima, Inc.

So, the ideal would be to have high accuracy with low variability and a long time horizon.

Think how this benefits a pilot approaching a landing. A novice pilot may only think ahead a few seconds at a time, whereas an expert makes determinations further in advance in order to set up their approach for an easy landing.

However, these dimensions of quality are logically independent, so all combinations are possible, and those combinations could recommend specific additional experience for a given student.

For example, if a student was generally accurate with low variability but had a limited time horizon, we could recommend that they engage in specific training that involved predictions over longer time periods.

The Takeaways:

1. Experts from many domains learn to predict well.
2. Prediction skill grows as expertise grows.
3. The brain is a prediction machine.
4. Direct prediction training and prediction measurement can exploit and improve these effects.

References

1 Fadde, P.J. (2006). Interactive video training of perceptual decision-making in the sport of baseball. Technology, Instruction, Cognition and Learning, 4, 237-255.

2 Stacy, W., Walwanis, M., Bolton, A., Beaubien, J., & Wiggins, S. (2014).  Using temporal occlusion to assess carrier landing skills.  Proceedings of the 2014 Interservice/Industry Training, Simulation, and Education Conference, Orlando, FL.

3 Brennan, P.C., Gandomkar, Z., Ekpo, E.U., Tapia, K., Trieu, P.D., Lewis, S.J., Wolfe, J.M., & Evans, K.K. (2018). Radiologists can detect the ‘gist’ of breast cancer before any overt signs of cancer appear. Nature Scientific Reports,  https://www.nature.com/articles/s41598-018-26100-5.

4 Lisa Feldman Barrett, https://www.edge.org/response-detail/26707, 2016.