This is Part 2 on a short series on sensemaking. You may read Part 1 here.
In the previous instalment we talked about one way to make sense of AI (without losing your head). At the end of that piece I wrote that the approach is a good start, but it is not enough. In order to examine why, however, we need to introduce more concrete language about how humans make sense of situations. In particular, we want to know how experts sensemake differently from novices. Only after introducing these new ideas will we be able to talk about how to improve at sensemaking when faced with an uncertain new technology.
In this instalment we will examine the best theory for sensemaking that we currently have. This theory comes out of research done for the US military, primarily funded by a contract with the Army Research Institute for the Behavioral and Social Sciences. It has proven itself useful in all sorts of domains, but it is a relatively young theory — which is to say that it has survived falsification for only two decades. (This is good enough for me, though you might want to calibrate your expectations appropriately).
If we take a step back, however, you can easily imagine why the US military would be interested in a theory of sensemaking. A huge part of intelligence-gathering and warfighting is making sense of ambiguous information, under conditions of extreme uncertainty. (It’s called the ‘fog of war’ for a reason!)
And what is true for sensemaking in war is true also for sensemaking in business and in investing. The sensemaking processes that skilled warfighters use in battle is the same one that a business leader uses when deciding what to do when faced with a new competitive threat. It is the same process that an investor uses when coming up with an investment thesis for a specific company (or when the investor decides that a previous thesis has been invalidated.) And it is the same process technology leaders and engineering managers must use when faced with a revolutionary new technology.
If you’re interested in only what is immediately applicable, you might scoff at theory. But I assure you that there are plenty of things here that you can use. If nothing else, it will give you better language for your own cognition — concepts that are more useful than accounts like ‘selection bias’ or ‘confirmation bias’. That is: you will be able to notice things about your own thinking that you won’t before reading this piece.
And most importantly: you will know how to get better. Which means you’ll be better equipped when AI impacts your corner of the world.
The theory we’re going to examine is the Data-Frame Theory of Sensemaking, originally published in 2007 by Gary Klein, Jennifer K. Phillips, Erica L. Rall, Deborah A. Peluso. (If you are a Commoncog member, you may download a cleaned-up version of this paper in PDF and ePub formats here).
Sensemaking is defined as “the deliberate effort to understand events”. For instance, the thinking and actions involved with the question: “how will AI impact my career?” is a form of sensemaking. So is “how may I use AI to gain a competitive advantage against my competitors?” But many other activities are also considered sensemaking, and use the same cognitive processes in the human brain. For example: “where are we right now and how do we get to the train station?” is a sensemaking process, just as “I am a doctor; what is going on with this patient and what should I do next?”
To give you a taste of what experts may accomplish when sensemaking, I’ll start with two real world examples, both taken from Klein et al’s paper. Here is example one (all bold emphasis mine):
During a Marine Corps exercise, a reconnaissance team leader and his team were positioned overlooking a vast area of desert. The fire team leader, a young sergeant, viewed the desert terrain carefully and observed an enemy tank move along a trail and then take cover. He sent this situation report to headquarters. However, a brigadier general, experienced in desert-mechanized operations, had arranged to go into the field as an observer. He also spotted the enemy tank.
But he knew that tanks tend not to operate alone. Therefore, based on the position of that one tank, he focused on likely overwatch positions and found another tank. Based on the section’s position and his understanding of the terrain, he looked at likely positions for another section and found a well-camouflaged second section. He repeated this process to locate the remaining elements of a tank company that was well-camouflaged and blocking a key choke point in the desert. The size and position of the force suggested that there might be other higher and supporting elements in the area, and so he again looked at likely positions for command and logistics elements. He soon spotted an otherwise superbly camouflaged logistics command post. In short, the brigadier general was able to see and understand and make more sense of the situation than the sergeant. He had much more experience, and he was able to develop a fuller picture rather than record discrete events that he noticed.
And here is example two, which comes from an account of a expert nurse in a Neonatal Intensive Care Unit (NICU) — again, all bold emphasis mine:
This baby was my primary; I knew the baby and I knew how she normally acted. Generally she was very alert, was on feedings, and was off IVs. Her lab work on that particular morning looked very good. She was progressing extremely well and hadn’t had any of the setbacks that many other preemies have. She typically had numerous apnea episodes and then bradys [short for bradycardia — a common, usually temporary slowing of a newborn’s heart rate, frequently caused by immature brain development rather than disease], but we could easily stimulate her to end these episodes. At 2:30 her mother came in to hold her and I noticed that she wasn’t as responsive to her mother as she normally was. She just lay there and half looked at her. When we lifted her arm it fell right back down in the bed and she had no resistance to being handled. This limpness was very unusual for her. On this day, the monitors were fine, her blood pressure was fine, and she was tolerating feedings all right. There was nothing to suggest that anything was wrong except that I knew the baby and I knew that she wasn’t acting normally. At about 3:50 her color started to change. Her skin was not its normal pink color and she had blue rings around her eyes. During the shift she seemed to get progressively grayer. Then at about 4:00, when I was turning her feeding back on, I found that there was a large residual of food in her stomach. I thought maybe it was because her mother had been holding her and the feeding just hadn’t settled as well. By 5:00 I had a baby who was gray and had blue rings around her eyes. She was having more and more episodes of apnea and bradys; normally she wouldn’t have any bradys when her mom was holding her. Still, her blood pressure hung in there. Her temperature was just a little bit cooler than normal. Her abdomen was a little more distended, up 2 cm from early in the morning, and there was more residual in her stomach. This was a baby who usually had no residual and all of a sudden she had 5 cc to 9 cc. We gave her suppositories thinking maybe she just needed to stool. Although having a stool reduced her girth, she still looked gray and was continuing to have more apnea and bradys. At this point, her blood gas wasn’t good so we hooked her back up to the oxygen. On the doctor’s orders, we repeated the lab work. The results confirmed that this baby had an infection, but we knew she was in trouble even before we got the lab work back.
Hold these bolded sections in abeyance; we’re going to come back to them.
The Data-Frame Theory of Sensemaking starts out with a common-sense observation. Humans notice facts only in the context of a frame. What is a frame? At its most basic level, a frame is a story or a cognitive structure we construct to help explain the situation. Frames organise the relationship between pieces of information in our heads — and in so doing, allow us to use that information. These relationships may include spatial relationships (“where are we on a map?”), causal relationships (stories and scenarios), temporal accounts (stories and scenarios) and functional relationships (‘scripts’ — which in psychology is taken to mean ‘a regularly occurring sequence of events or activities that can be formulated as a template’ (Schank & Abelson, 1977)).
The two examples quoted earlier already give us multiple examples of frames. But even before we examine those examples, I suspect this account of ‘data and frame’ should make intuitive sense to you. For instance, have you ever had the following experiences? Let’s say that you’re a programmer. You read an error log and say “ok, I think the problem is such-and-such library …” and after a few minutes of debugging, you turn out to be correct. Meanwhile, your intern looks uncomprehendingly at the error log and spends two hours digging in exactly the wrong place in the codebase. Or, let’s say that you’re an exec in your company. You’re looking at a company dashboard, spot some data that seems odd to you (and only to you!), and a hunch forms in your head. Two dashboard checks, three SQL queries and one conversation later, you discover that 20% of trial customers have been silently dropping off for the past two months because the onboarding flow is broken. Meanwhile, your lead data analyst has completely missed this, despite looking at the raw data every week in preparation for the weekly company-wide data report.
The difference between you, the experienced programmer, or you, the senior exec, with someone more junior is not the data you are looking at. In both cases your programming intern and your lead data analyst had access to the same information that you did. The difference is the frame with which you evaluated that data. In both cases, you drew inferences based on the causal mental models in your head (i.e. your frame), whilst the more junior person either drew the wrong inferences from the data, or treated it as irrelevant. In simpler terms, you made more sense of the data than your subordinate because of your superior frame construction abilities, which resulted in different information-gathering actions, and eventually different decisions.
This difference lies at the heart of expertise. I don’t want to put too fine a point on this, because this is important. The observation that experts construct different frames compared to novices is actually profound. Frame construction is the bit of tacit knowledge that matters. It is the bit that accelerated expertise training programs attempt to train for. If that isn’t enough, frame construction is directly linked to insight generation. Insight generation is really important! If you study cases of business strategy, every winning strategy boils down to a few small insights that frame the problem advantageously — which the rest of the strategy is then built around. Richard Rumelt calls this the ‘kernel’. Roger Martin calls this answering the twin questions of “where to play” and “how to win”. The point is: if you can frame the problem you’re facing properly, you have half the problem solved.
And so you might understand why I believe frame construction is ridiculously important to understand, and to improve at.
For now, let us return to the two examples I listed above. We may describe the different frames in those examples as follows:
The frame we pick informs the data we notice. But the data we notice is also used to construct the frame. This implies the relationship between data and frame is recursive: both frame and data affect each other. Let’s examine a third example which demonstrates this relationship (which also comes from Klein et al’s paper):
An accident happened during an Army training exercise. Two helicopters collided. Everyone in one helicopter died and everyone in the other helicopter survived. Our informant, Captain B., was on the battalion staff at the time.
Immediately after the accident, Captain B. suspected that because this was a night mission there could have been some complications due to flying with night-vision goggles that led one helicopter to drift into the other.
Then Captain B. found out that weather had been bad during the exercise, and he thought that was probably the cause of the accident; perhaps they had flown into some clouds at night.
Then Captain B. learned that there was a sling on one of the crashed helicopters, and that this aircraft had been in the rear of the formation. He also found out that an alternate route had been used, and that weather wasn’t a factor because they were flying below the clouds when the accident happened. So Captain B. believed that the last helicopter couldn’t slow down properly because of the sling. The weight of the sling would make it harder to stop to avoid running into another aircraft. He also briefly suspected that pilot experience was a contributing factor, because they should have understood the risks better and kept better distance between aircraft, but he dismissed this idea because he found out that although the lead pilot hadn’t flown much recently, the copilot was very experienced. But Captain B. was puzzled about why the sling-loaded helicopter would have been in trail. It should have been in the lead because it was less agile than the others. Captain B. was also puzzled about the route—the entire formation had to make a big U-turn before landing and this might have been a factor too. So this story, though much different than the first ones, still had some gaps.
Finally, Captain B. found out that the group had not rehearsed the alternate route. The initial route was to fly straight in, with the sling-loaded helicopter in the lead. And that worked well because the sling load had to be delivered in the far end of the landing zone. But because of a shift in the wind direction, they had to shift the landing approach to do a U-turn. When they shifted the landing approach, the sling load had to be put in the back of the formation so that the load could be dropped off in the same place. When the lead helicopter came in fast and then went into the U-turn, the next two helicopters diverted because they could not execute the turn safely at those speeds and were afraid to slow down because the sling-loaded helicopter was right behind them. The sling-loaded helicopter continued with the maneuver and collided with the lead helicopter.
Notice how the protagonist in this example, Captain B., uses data points as anchors to construct possible frames. Each frame serves as a hypothesis: a plausible explanation to make sense of the facts he has been given:
At first, Captain B. had a single datum, the fact that the accident took place at night. He used this as an anchor to construct a likely scenario. Then he learned about the bad weather, and used this fact to anchor an alternate and more plausible explanation.
Next he learned about the sling load, and fastened on this as an anchor because sling loads are so dangerous. The weather and nighttime conditions may still have been factors, but they did not anchor the new explanation, which centered around the problem of maneuvering with a sling load. Captain B.’s previous explanations faded away. Even so, Captain B. knew his explanation was incomplete, because a key datum was inconsistent—why was the helicopter with the sling load placed in the back of the formation?
Eventually, he compiled the anchors: helicopter with a sling load, shift in wind direction, shift to a riskier mission formation, unexpected difficulty of executing the U-turn. Now he had the story of the accident. He also had other pieces of information that contributed, such as time pressure that precluded practicing the new formation, and command failure in approving the risky mission.
Captain B. only stops with his sensemaking process when all the information he has received about the accident is coherent with his frame (in this case a story he constructed to explain the incident). Thus satisfied, he moves on from sensemaking and begins to think about taking action. Note that if he receives a new piece of information that is inconsistent with his current frame and too difficult to explain away, he will drop back to sensemaking once again.
We now have enough concepts to describe the Data-Frame theory in full.
Figure 1 shows the four cycles of the Data-Frame Theory of Sensemaking. We shall examine each of these sensemaking cycles separately, as they involve different strategies:
All of this seems obvious — you might already recognise this in your own cognition — but bear with me. We need to go through each of these cycles in order to get to the good stuff.
The basic sensemaking act is the data-frame symbiosis. As previously discussed, your frame informs what is counted as data, but the data you examine extends and modifies your frame. In practice, what happens is that you switch back and forth between examining data and constructing frame.
Let’s talk details.
Sometimes you have committed to an initial frame, but you still have unanswered questions. One common move during sensemaking is to take action to fill in gaps in your current frame. You may:
I’ll illustrate with an example from the previous instalment in this series. One possible frame in the current AI revolution is “it is possible to produce complex, usable software at high velocity with little to no manual human intervention, but it requires around six months of iterating on the agent harness.” I’ve mentioned this frame in passing in the previous instalment, as an example of “What are the possible outcomes here?” But there remain many unanswered questions. For instance: are these teams actually producing good software? What kinds of software? How do I know? And then: how are these teams accomplishing this? What goes into their harness engineering? What do they know that I don’t?
In response, you may take action to elaborate the frame:
The key thing is that I have to commit to a frame before elaboration starts in earnest. Humans cannot investigate what they do not believe can be true. Someone who rejects the very idea of the frame, who has the opinion “this is bullshit, my experience of software engineering and the crappy results I’ve been getting from Claude Code tells me this isn’t even possible” means that they will either dismiss data points, or they will not take steps to seek out new confirming (or disconfirming) information.
In many ways, they cannot seek out disconfirming information at all. Recall that all data must be constructed, and all data are constructed within a frame. If you do not have (or refuse to commit to) a frame — you will not even see the relevant data. You would — quite literally — be blind.
At some point, you may encounter new data that does not fit into your frame. At this juncture, you may begin to question the frame you’ve committed to.
Here you are presented with two options:
Questioning a frame consists of the following mental activities:
The preservation cycle occurs when you choose to ignore data. You may also decide to discard most of the data but record a new gap in your current frame (which you may then act on: you may seek new information to plug that gap or to extend your existing frame). For instance, one outcome that might occur as I investigate the “AI coding with minimal human intervention” is that it turns out this is only possible for certain kinds of software. This is an outcome that nobody can predict in advance — but that’s par the course when you’re dealing with the uncertainty of a new technology.
I think all of us have had experiences where we considered an alternative perspective, only to reject the evidence and recommit to our current frame. We’ll revisit this step when we talk about expert novice differences — in the next section.
The reframing cycle is where things get interesting. Let’s say that you’re seeing new data that is simply too incongruent with your current frame. If enough incongruent data accrues, you will experience a ‘frame breaking’ moment. You may decide to discard your current frame.
Here you have two options:
These four cycles explain the core of the Data-Frame Theory. But of course, if we want this to be useful, we should ask: what do experts do differently from novices?
Expert-novice differences are useful because they tell us how to improve. If we know what experts do differently from novices, we may a) evaluate our skill level, b) design training programs for ourselves and for others, and c) we may avoid the worst of the novice errors.
With this in mind, it was a little surprising for me to learn that according to the Data-Frame theory, experts do not use different sensemaking strategies from novices. Both experts and novices use all four cycles in the Data-Frame model: they construct frames, question frames in response to sufficiently incongruent data, elaborate frames and perform reframing in the exact same ways.
It sure seems like the Data-Frame model is a core part of human cognition!
What differs is this:
As I’ve mentioned earlier, the Data-Frame theory appears to get at something universal about human cognition. And it gets there from naturalistic study of experts (and novices) working on real world tasks. This implies that improvement suggested by this theory should be easier: if real-world experts all converge on the same set of superior strategies, we have proof that it’s doable to train novices in them.
But this is perhaps too easy. You might think, “Okay, on with it! This all seems very obvious. Let’s talk about how to put this to practice! Let’s apply this to sensemaking AI!”
Yes, we’ll get there. But first, I want to poke at your newfound knowledge, in ways that might unsettle you. I want to talk about …
I suspect that if you’ve reached this point, you’ve found yourself mostly nodding along. All of the elements of the Data-Frame theory can seem obvious, even trivial. In fact, I’m willing to bet that you recognise the various cycles in your own cognition. The only thing that might be new to you are the claims about the specific number of anchors and frames (e.g. that it will take no more than three-to-four anchors to construct a frame, that it is difficult to reuse an anchor for a different frame, that experts hold at most two-to-three frames in parallel, beyond which performance will degrade).
But if you feel that the Data-Frame model is common sense, you are mistaken. The easiest way to work out why the theory should be challenging to you is to go through a few other beliefs you might have, and then talk about why those other beliefs are invalidated by implication. We should start with …
Confirmation bias is a cognitive bias that is extremely well-replicated, and confirmed in literally thousands of psychology studies and reviews. The American Psychological Association defines it as a tendency to gather evidence that confirms preexisting expectations. Psychologist Raymond Nickerson’s widely cited review describes it as a pervasive phenomenon that appears in many forms across judgment and reasoning. He even goes so far as to describe it as a “weakness” of human cognition — an assessment that puts him in good company. This view of ‘confirmation bias as human weakness’ is as close to mainstream consensus as we can get; it is held by some of the brightest stars in psychology: Daniel Kahneman and Amos Tversky and Phillip Tetlock included.
The following paragraph comes from the concluding section in Nickerson’s review paper:
The question of the extent to which the confirmation bias can be modified by training deserves more research than it has received. Inasmuch as a critical step in dealing with any type of bias is recognizing its existence, perhaps simply being aware of the confirmation bias—of its pervasiveness and of the many guises in which it appears—might help one both to be a little cautious about making up one’s mind quickly on important issues and to be somewhat more open to opinions that differ from one's own than one might otherwise be.
What a reasonable recommendation! Note that I say this unironically: the weight of evidence for the cognitive process that produces what we call ‘confirmation bias’ is beyond doubt. Humans naturally want to confirm beliefs that we already have. Because confirmation bias is so pervasive, the logical thing to do is to tell folks to delay forming a conclusion, to keep an open mind.
I want to be completely honest with you: I believed this argument completely. I swallowed it hook, line, and sinker. Like many of my generation, I read Thinking: Fast and Slow and many other popular science books on psychology. By the early 2010s, the heuristics and biases view had so completely permeated our world that it is difficult to imagine a universe of ideas without it. Hell, you can’t read a self-help book today without a reference to some cognitive bias.
Let’s agree that the confirmation bias describes a real cognitive phenomenon. Humans really do prefer to seek confirming evidence over disconfirming evidence. Most folks find it difficult to seek out disconfirming evidence without practice. But is this tendency really a weakness?
I want you to think about Example Two — the one with the expert NICU nurse — above. Go back to that example and read all the bolded bits. When the nurse started elaborating a second frame in parallel (“this baby has sepsis”), she sought out confirming evidence for the competing frame. Was she engaging in confirmation bias? No. Her performance was in the top percentile of NICU nurses. She was simply using the reframing cycle of the Data-Frame theory.
Klein et al write (bold emphasis mine):
In natural settings, skilled decision makers shift to an active mode of elaborating the competing frame once they detect the possibility that their frame is inaccurate. (…) A person uses an initial frame (hypothesis) as a guide in acquiring more information, and, typically, that information will be consistent with the frame. Furthermore, skilled decision makers such as expert forecasters have learned to seek disconfirming evidence where appropriate.
It is not trivial to search for disconfirming information—it may require the activation of a competing frame. Patterson, Woods, Sarter, and Watts-Perotti (1998), studying intelligence analysts who reviewed articles in the open literature, found that if the initial articles were misleading, the rest of the analyses would often be distorted because subsequent searches, and their reviews were conditioned by the initial frame formed from the first articles. The initial anchors affect the frame that is adopted, and that frame guides information seeking. What may look like a confirmation bias may simply be the use of a frame to guide information seeking. One need not think of it as a bias.
When you call something a ‘bias’, the natural response is to do bias correction, or error reduction. This is why Nickerson, in his paper, and in fact hundreds of other psychology papers and intervention experiments and popsci books all recommend the same thing: “keep an open mind”, “don’t form a conclusion too hastily.” What could be more reasonable?
Unfortunately the recommendation by the confirmation bias folks backfires completely: in just about all natural settings that the authors examined, folks who use the ‘keep an open mind’ or ‘delay forming a view too early’-type strategies underperformed the experts. In fact, Klein et al lays this out as one of the falsifiable assertions of the Data-Frame theory: if you can find a single expert in a naturalistic setting using the ‘keep an open mind’ approach, then the Data-Frame theory is falsified and needs updating. (And if you think you’ve found someone who does this, you should check: is this really a top percentile performer? Or is there someone — or many someones — who have better performance?)
What do the experts do instead? In domain after domain, Klein and his colleagues have found that experts form frames quickly, with specific, concrete expectancies that may be violated. This allows them to course-correct when they encounter abnormalities (that is, when their expectancies are violated) much faster than someone who hasn’t committed to a frame.
Note that I’ve elided the evidence Klein et al cite to justify this assertion; you may read the original papers here, here and here. But you may also do a literature review search to see how this claim has held up over the subsequent 19 years.
Why did the confirmation bias folks get things so wrong? The answer — which is a bit of an open secret — is that the entire field of cognitive biases and heuristics is built around a flawed methodology. The field conducts experiments by administering toy problems to unskilled test subjects (undergrads, mostly) in artificial lab environments. All the cognitive processes demonstrated by the participants in these studies are then labelled as ‘reasoning errors’ or ‘biases’ whenever they result in the wrong answers. And make no mistake: these cognitive processes are real; the vast majority of them are reliably replicated in lab study after lab study, over the course of decades. But if you study practitioners solving real problems in naturalistic environments — problems that they have actual expertise in — you will find that all the same cognitive processes that result in ‘reasoning errors’ on lab tests are suddenly deployed in ways that produce excellent performance. This has been the finding of the Naturalistic Decision Making (NDM) branch of applied psychology — which Klein helped start — in study after study, ‘bias’ after ‘bias’, for the past 30 years.
I can write a longer essay about this ‘open secret’. Perhaps I will. But I want to leave you with the following observation: whenever you see a cognitive bias, you should understand that there are two ways to get better performance. You may do error reduction, or you may fix it by gaining expertise.
Unfortunately, the first path, on error reduction, is a dead end. Kahneman himself has argued that he did not believe you can correct cognitive biases . To be fair, some of his colleagues do not agree with him, and have demonstrated that training interventions for bias correction can improve performance. But the improvements are comparatively minor, and the results are slow in coming. In 2021, two giants of the Judgment and Decision Making field, David Weiss and James Shanteau, published a paper titled The Futility of Decision Making Research. They were lamenting the uselessness of their findings, despite publishing decades of research. Meanwhile the field of NDM continues to produce useful results for the military and others, including effective decision training for the armed forces, better user interfaces for military and industrial control systems, and accelerated expertise training programs.
It doesn’t take a genius to see why. Which should result in better performance: an error correction training program built on top of the findings of unskilled test subjects solving toy problems in artificial lab environments? Or an expertise acceleration training program built on top of how experts actually achieve a low error rate in their performance on real world scenarios? Exercise left for the alert reader.
Here is a second implication of the Data-Frame model. Most of us have the following view of analysis:
The common view of analysis is that data gets turned into information, which gets turned into insight, which gets turned into decisions, which gets turned into (in this case) ‘alpha’ — which is investment terminology for “the return your investment earns beyond what the market gives you for free.”
In this model of analysis, more data results in more insights, which leads to more decisions, which results in more alpha. Therefore LLMs and other decision support systems will increase decision quality and quantity by increasing the data points that a single analyst can process.
The Data-Frame theory immediately tells us this model cannot be true.
I’ve spent much time in my exposition talking about how ‘data must be constructed’ and that ‘something is only considered data in the context of a frame.’ I’ve spent some time telling you about a hypothetical data analyst who fails to spot a data abnormality because of expertise: he lacked the right frame. The Data-Frame theory tells us that the high order bit for analysis isn’t data processing — it can’t be: data does not arrive fully formed in packets, ready for consumption. The high order bit for analysis is frame construction. It’s only by constructing the right frame that an analyst can pick out the right data from an event stream. Without the right frame, the analyst is effectively blind.
And in fact, in study after study, Klein and his colleagues have found that performance degrades when an analyst is fed too much data.
So what do you do, if you want to build a analysis augmentation or decision support system?
Klein et al suggests that you build a system that keeps track of anchors. There are two types of anchors: anchors that are used to construct your current frame, and ‘possible anchors’ — data points that are regarded as potential anchors by the analyst, but ultimately discarded in the course of analysis. Since frame construction happens with only a small number of anchors, keeping track of these anchors is trivial.
But the potential impact is large: if the analyst encounters a frame-breaking moment, they may go back to their initial set of three-to-four anchors and reassess them. They may attempt to construct alternative frames from previously discarded possible anchors. We’ve discussed how humans find it difficult to take data points used to anchor one frame and reuse them as anchors for a different frame. Having a system that externalises anchor categorisation opens the analyst up for collaborative critique. It allows a team of intelligence analysts to do collective sensemaking.
The Data-Frame theory also suggests a method for measuring the effectiveness of decision support systems: how long does an analyst take to detect faulty data? Say we insert bad data into a series of simulated events — often described as a ‘garden path’ intelligence testing scenario. A good system should allow the analyst to catch inconsistencies faster and reframe. A bad system will make it more difficult to recognise that one’s current frame is compromised.
There are other recommendations from the theory, and you should read the original paper if you want to dive deeper. (There have also been extensions worked out in the subsequent 19 years; a lit review should uncover them. I suspect some of you might already have ideas on how to augment frame construction with LLMs by this point.)
But I’ll stop here.
Let’s return to the overarching topic of this series. We want to talk about how to sensemake uncertain new developments that are likely to affect our careers, businesses, and our lives. Right now that’s a new technology: AI. In the future it might be some other thing: tariffs, perhaps, or — god forbid — war.
I’ve taken you on a long detour to talk about the Data-Frame theory because I think it’s actually more important to understand how we make sense of things, in order to improve at it. (I believe that AI is just one of many uncertain things that will affect us over the course of our lives. Things have gotten more volatile, not less, in the past decade, and there’s no reason to assume it’ll get better in the coming years. As I write this, the Straits of Hormuz is closed to most ships. This may or may not develop into a widespread energy crisis. Is that not uncertainty that one must handle?)
But I digress. Now that we have a better language around the process, we may talk about how to apply this.
First, let’s discuss the most obvious way your sensemaking might fail. The biggest danger we face when sensemaking a revolutionary new technology is frame fixation: that is, we stick to our current frame — by which I mean our current understanding of our domain — and as a result ignore new data that can only make sense in the context of a new frame. This is especially dangerous if the new frame contradicts the way we’ve worked before.
It is possible that you’ve already realised this. In the previous instalment I gave you a simple method of sensemaking AI:
And that’s all well and good, but the problem is that if you are not sensitive to alternative new frames, you may dismiss data points that indicate new outcomes, new actions and new causal mechanisms. Which means my recommendation is as good as useless.
Fortunately, the Data-Frame theory has one obvious answer: be more willing to elaborate a new, alternative frame. You don’t have to commit totally to this new frame — it’s logical to hold on to your current frame whilst you elaborate the new frame in parallel. In truth, my frame of “it’s possible to build good software with AI without much human intervention” is an alternative frame I’m actively elaborating. At this point I am not yet sure if I should commit completely to it.
The broader danger is clear now, I think: you hold on to your current frame too tightly that you don’t even consider alternatives. Which means you’ll be blind to certain data, and therefore blindsided when the best practices of your domain shifts under you.
We’ll have more to say about this failure mode in the next instalment.
There is one other recommendation that falls out of the Data-Frame theory. The theory states that you cannot improve by learning better ‘sensemaking skills’. Those don’t exist; novices sensemake in the exact same way that experts do. Instead, you improve by:
This may all seem a little theoretical, so here’s an example.
I shall give you a fragment: an example of a new technology leading to a competitive advantage for a specific company. Observe your own cognition as you’re reading this fragment; I’m willing to bet that your sensemaking will change by the end.
The PC revolution occurred over the course of the 80s. Initially, demand for personal computers was insatiable. The first killer app for the personal computer was the spreadsheet — VisiCalc, originally written for the Apple II. This caused sales of the Apple machine to take off. It’s a little hard to imagine today, but the interactive digital spreadsheet represented something remarkable: before desktop spreadsheet software, accountants and businesspeople had to update the ‘cells’ of a literal spreadsheet (on paper) manually. Nobody who saw the interactive spreadsheet in action could forget it. Demand was instantaneous. When Lotus 1-2-3 — a better, more powerful spreadsheet package — launched for the IBM PC, the same thing happened for the entire IBM PC-compatible market.
This meant that dozens of PC manufacturers sprung up to sell machines to consumers. And despite the average PC costing in the thousands of dollars, consumers and businesses simply couldn’t get enough of them.
Spreadsheet software was quickly followed by desktop word processing software. The market leader was WordPerfect. Microsoft veteran Steven Sinofsky was a graduate student at the time; in his memoirs, he wrote:[In the 90s] Windows Office followed Mac Office but with a slightly bumpier journey. Externally, with Windows, the challenge was first winning critical acclaim and customer love over the category competition. The journey would take years for some customers—not only were the MS-DOS leaders [Lotus 1-2-3 and Wordperfect] loved, but those products were hard to learn and customers invested a lot in the keystrokes, macros, plugins, and in the existing files, which were difficult to import and export with Excel and Word. The MS-DOS PC era [in the 80s] was characterized by investing in a PC and software to the tune of $3,000 ($7,000 in 2019 dollars) or more, and then literally taking classes and buying books to learn to use the computer. I spent two summers in the mid-80s at Martin Marietta teaching people how to use WordPerfect and 1-2-3 (but never both to any one person as “secretaries” learned WordPerfect and managers learned 1-2-3).
The classifieds of all the major newspapers at the time were plastered with ads for computer classes. Course creators, then as now, were making bank in response to this new technology, capitalising on widespread excitement, but also widespread fear of missing out. The opinion pages were filled with breathless speculation about the ‘paperless office’; tech pages were filled with PC reviews; even non-technology writers weighed in on how the computer would change business, and society.
If this sounds familiar, it is because it is. In the early stages of a new technological revolution — especially with a general purpose technology like the computer, or with AI today — you should expect to see rabid, widespread adoption. Then, firms were purchasing computers by the boatload and paying for training. Today, companies are mandating ‘AI-usage throughout the enterprise’ and course creators are teaching “how to use AI” courses and are raking it in. Often, you only have to slap ‘AI’ on a product label to get buy-in.
Did any of these companies win? No, of course not. Mass adoption meant that everyone upgraded within the same decade, which meant nobody had a permanent advantage against their competitors. In the long run, PCs made certain activities — like accounting, finance, desktop publishing, typesetting, and so on — permanently easier. In some cases, PC adoption let smaller companies punch above their weight. Barcode scanners, for instance, allowed medium-sized retail chains the ability to fight back against the national chains … for a time. Similarly, desktop publishing software led to a short flourishing of new weekly newspapers, monthly periodicals, and even daily newspapers in 1985 to 1990 (a trend that would soon reverse with the advent of the Internet). But PC adoption by itself did not lead to sustained competitive advantage.
So who won as a result of the PC revolution? Who built a sustained competitive advantage and crushed their competitors?
The answer: Walmart.
Walmart was an early adopter of personal computer technology, but not the earliest adopter. The first barcode scanner, for instance, was installed at an outlet of Marsh Supermarkets in 1974. Marsh was a medium-sized regional grocer. Over the course of the 70s and the 80s, the barcode scanner helped many smaller retailers compete with national chains. (I should note that it did not help the smallest of retail shops — the mom-and-pops — since computer equipment was still expensive and mom-and-pops lacked the bargaining power necessary to get manufacturers to print bar-codes on their packages). But for a ‘mere’ decade, at least, the barcode scanner helped many medium size chains resist total dominance by the nationals — a notable thing, since the nationals had superior economies of scale. Walmart adopted barcode scanners in their stories starting from 1983. It was not a laggard by any means. But it was only in the late 80s that Walmart had barcode scanners in all of their distribution centres. This second thing was more important, as we shall soon see.
Walmart was also not the only consumer of information technology. During this period, Sears — then the largest retailer in the US — was a major spender, and in many ways the leader, on retail technologies. In his 2022 book, The New Goliaths, economist James Bessen points out that Sears was IBM’s largest customer in the late 80s. They also partnered with IBM and spent $1 billion to create the jointly-owned Prodigy system, pioneering e-commerce in the process.
Today Sears is effectively defunct. Kmart, the second largest retailer during the 80s, is also dead. In 1979, Kmart was the king of the discount retailing industry — a segment it helped create. At times the company even surpassed Sears in revenue. It had 1891 stores and average revenues per store of $7.25 million; that same year, Walmart had only 229 stores and per-store revenues that were half of the average Kmart store. A mere 10 years later, in 1989, Walmart had achieved the highest sales per square foot, inventory turns, and operating profit of any discount retailer. In 1990, Walmart overtook Kmart as the second-largest US retailer by revenue.
What happened?
Walmart invested in information technology, but did so in a particular way. Barcode scanning generated a torrent of purchase data. This allowed stores to calculate store finances and update inventory data on-the-fly. This benefit was obvious, and was touted by the makers of barcode scanning systems; many regional chains built or bought software to accomplish exactly this set of outcomes. When the national chains hopped onto the barcode scanner wagon, they built their own versions of this software but grafted it onto their own centralised company structure. This was logical: the national chain store format was originally built on the premise that centralised purchasing and streamlined product line ups would mean increased purchasing power, and therefore increased economies of scale.
Walmart noticed this benefit of barcodes and the affordances of cheap computers and took it to its logical extreme.
The problem with stocking new product lines is that you have to coordinate purchasing and restocking with your suppliers. A larger selection of SKUs and services results in increased complexity for store managers. Walmart built custom software to handle this complexity. By the late 1970s, all of Walmart’s distribution centres were linked by a computer network. In 1987, Walmart built its own $24 million satellite network for communication between stores and headquarters. In 1990, Walmart launched Retail Link — custom software that connected its stores, distribution centres, and suppliers and provided live inventory data to all parties. With this final piece, even Walmart’s suppliers could see when inventories at specific stores were running low, and initiate a new purchase order and delivery.
More importantly, the net result of all of Walmart’s information technology investments inverted the chain store command structure: they redirected the flow of information to enable store managers and even suppliers to make stocking decisions. This reduced the cost of managing and therefore adding additional product lines. As a result, Walmart began expanding its offerings. In 1988 it introduced the Supercenter format. In Supercenter stores, Walmart not only sold general merchandise but also dry and frozen goods, meat, poultry, fresh seafood and produce, pharmacies, optical stores, photography services, tire and lube services, hair and nail salons, and eventually cell phone stores, banking products and fast food. This increased variety of products and services drew customers to Walmart; the company touted ‘one-stop shopping’ as the reason “customers choose our Supercenters.” One study even found that consumers were willing to pay a premium to shop at Walmart due to this advantage.
But Walmart didn’t stop there. It abandoned the typical chain store warehouse model in favour of a logistics practice called cross-docking. In a traditional chain store warehouse, suppliers would deliver goods to the company warehouse, where it would be shelved and inventoried. Later, when stores needed to be resupplied, the company would take products out of the warehouse and deliver it to the stores. Walmart got rid of this practice. Instead, when supplier trucks arrived at a Walmart distribution centre, goods would be unloaded from the supplier’s truck and immediately loaded onto various Walmart trucks. These Walmart trucks would then drive to each individual store. This meant that both store trucks and supplier trucks must arrive at the distribution centre at the exact same time — and that loaders needed to know how much of each shipment needed to be put into which Walmart truck. This was only possible with (again) custom software Walmart had built.
The impact of cross-docking was that Walmart could enjoy the same economies of scale that came with purchasing in bulk, but without the usual inventory and handling costs. It passed on all of this cost savings on to the consumer. The savings from cross-docking meant that in the grocery segment, products in Walmart Supercenters cost about 10% less in the same markets relative to traditional supermarkets. Walmart’s competitors simply couldn’t compete. In every market that Walmart entered, they began dying. They were toast.
Bessen calls this “winning through economies of scope” — the ability to service vastly increased product variety due to investments in information technology.
So whilst all of Walmart’s competitors were investing in ‘everyone gets a computer’ and ‘everyone needs to use a computer’ and ‘here are stipends for computer courses’, Walmart was using information technology to do something very, very different.
Bessen concludes (lightly paraphrased, bold emphasis mine): “Walmart married information technology to a new type of organisation. It turned the chain store model around and used information technology to decentralise decision-making, allowing countless decisions to be made quickly and efficiently, creating stores that better met customer needs. It was this combination of technology and organisation that allowed Walmart to grow while Sears and Kmart and countless smaller retailers struggled and often failed.
So? Did your thinking about AI change in response to this?
I suspect yes. Why?
The short answer is that I’ve given you a new causal mental model with this fragment. The causal model goes something like this: “When mass adoption of a new technology happens, nobody wins — or at least not for long. The way to truly beat competitors with new technology is to use that new technology in a way that cannot be easily copied.”
You may also draw certain other inferences, such as “‘how to use AI’ courses will not lead to sustained competitive advantage, or “top-down ‘everyone must use AI’ mandates will not lead to winning.” Perhaps if you are more sophisticated, you might think “new technology is best when it is used to be enforce an existing competitive advantage.” But in truth these inferences are not that important. What is important is that the fragment lives in your head.
The way you win with AI will — naturally — be different from the way that Walmart won with PCs. AI is a different technology from personal computers, and the competitive advantages available to you will be different compared to the competitive advantages available to Walmart in retail.
But because this fragment about Walmart lives in your head, you will be sensitive to new data that hints at a path to sustained competitive advantage for your specific context. Your brain will be able to assemble alternate new frames from fragments of what you know about your industry, combined with this Walmart fragment. Some of the frames you construct will be genuinely new insights — frames that nobody has ever constructed before. Hopefully, you’ll have a shot at competitive advantage that your peers do not see coming.
This is why it is important to collect cases of companies winning with new technology. Yes, I’m making a meta point here: the way to improve your sensemaking with regard to AI is to collect fragments of companies adopting revolutionary new technology and winning (as well as companies adopting new technology and losing). The more fragments in your head, the better the frames you will be able to construct. The more data you will be able to notice. The quicker you can self-correct from frame fixation. And the better prepared you will be if disruption comes for you and yours.
In the next instalment, we’ll discuss concrete details of how to improve the sensemaking approach that I outlined in the previous instalment. Specifically, we’ll talk about preventing frame fixation, and about hunting for new fragments that will help you see.
See you there.