We can all agree: the incentives are bad. Everywhere you look, people are being rewarded for doing the wrong thing. This has become the go-to diagnosis for every problem, from climate change to political polarization to scientific malfeasance—“sorry pal, you’ve got a bad case of the incentives.”
I don’t disagree with this analysis, exactly. Who could? It’s basically tautological. “People do bad things because they are encouraged and rewarded for doing bad things.” Well, why else would you do them?
My real gripe with this line of thinking is it implies that incentives should be obeyed, that it is normal or even noble to do so, and that is perhaps impossible to do otherwise. How could you be expected to break a rule, forego a promotion, or give up decimal of your grade point average, for the sake of—what? Doing the right thing?
There’s a point here that should go without saying, but instead it has simply gone unsaid. Which is: incentives are for losers. Doing something simply because an arbitrary reward system will give you a lil treat—that’s for pigeons, not people. If you build your life around satisfying incentives, then you’re a dummy and a schmuck.1
When I played Little League baseball for a few disastrous years, my least favorite part was going up to bat, because you had to wear one of these ratty old batting helmets supplied by the local rec department. None of them fit right, and they all smelled like a hundred sweaty 10-year-olds had worn them before you, because a hundred sweaty 10-year-olds had worn them before you. And so every at-bat carried not only the risk of striking out, but also the risk of contracting head lice.2
A few kids, the serious ones, brought their own helmets—custom fitted, tastefully decorated with decals, and presumably containing zero lice. These kids ran the bases like little princes. Any errant pitches bounced harmlessly off their heads, leaving their perfect skulls un-dented. The outfielders would perk up whenever a kid with a bespoke helmet stepped up to the plate, all of them thinking to themselves, this kid might actually hit the ball.
I say all this because incentives are like batting helmets: if you don’t bring your own, you’ll have to use the crappy ones that are provided for you.
A critical part of human development—in fact, the most human part of human development—is to acquire a purpose. That means refining your sense of right and wrong and figuring out how you, personally, can increase the right and decrease the wrong. Once your neural tube has ballooned into a brain, once you can poop without issue, conjugate irregular verbs, and feed yourself solid foods, you ought to ask yourself, “Why am I doing these things?” What’s it all for? How can I make myself useful?
It’s fine if you’re still working on your answers to those questions. It’s fine—and good!—if you have multiple answers, or if your answers change over time. But it’s not fine to pass on the questions entirely. We owe these answer to each other, and we owe them to ourselves, because a sense of purpose is both prosocial and protective. If you have a void where your moral convictions should be, then that cavity of your personality will get filled up with tumorous desires—for money, power, fame, status. This is the kind of person who “responds to incentives”.
It took me a long time to understand this. In my early 20s, when I was at the bottom of every ladder that I aspired to ascend, I assumed the people at the top of those ladders must be pretty great. How wise these incentive-maxxers must be, how cool and smart, to have the things that everybody else wants!
But over the years, as I had the opportunity to meet some of these captains of industry, these Grand Pooh-Bahs and Distinguished Scholars, I’ve realized how wrong I was. They only seemed enviable because I was looking at them from a distance, so all I could see was what they had. Up close, when you can see what they are, it becomes clear that only a few of these ladder-toppers are indeed wise and cool. More often, they’re sniveling little twerps. They’re spiritually chinless. They have no convictions, no passions, no purpose other than ladder-climbing. It’s sad to watch them, these people who have everything except the one thing that actually matters.
Often, these are the people who are most insistent that the incentives ought to be obeyed. They are quick to swear that good grades, cash, prestige, etc. are inherently valuable, or that they are useful proxies for value. The incentive-obsessives feel this way because they have to. Like the government of a failing nation, they must swear that their currency is real, and they must lash out at anyone attempting to participate in an alternative economy, lest the whole fraud come crashing down upon itself.
Without any formal instruction, approximately 100% of humans will learn to speak, and approximately 0% of humans will learn to read. Talking comes easy and early, enabled by ancient mental structures that hoover up words and extract the meaning behind them. These are the original LLMs, and still the best in the known universe. Reading, on the other hand, comes late and laboriously, piggybacking on bits of brain that were never meant to decode squiggles on paper, let alone comprehend Finnegan’s Wake.
So when thinking about any human ability, it’s useful to ask: is this more like speaking, or is it more like reading? Is this an ancient capability that we get for free just by living long enough, or is this a modern skill that you can’t get without working for it?
Developing a value system seems to be a lot more like reading, because research suggests that many people don’t make it very far.
In the early 2000s, the sociologists Christian Smith and Melinda Lundquist Denton asked hundreds of American teenagers about their religious beliefs. What they got in return was a bunch of gobbledygook. Regardless of their supposed faith, few of these teens could say anything specific about the divine. Most of them believed that God is like, this cool dude who floats in the sky and wants everybody to be happy and to get along, and if you ask him nicely he might help you achieve your goals, but otherwise he’s mostly busy floating. Fortunately, this celestial butler is not super specific about what you should and shouldn’t do, and he doesn’t really speak unless spoken to.3
Smith and Denton came up with a name for this thin soup of theology: moralistic therapeutic deism. They only studied teenagers, but they suspect that many people remain moralistic therapeutic deists for their whole lives—after all, the teens probably picked up this loose handful of beliefs from their parents. People will claim on surveys that they belong to a certain religion, or they’ll get riled up about specific bits of dogma. But if you dig into their beliefs at all, you’ll soon hit a gooey core of underbaked religious mishmash.4
I first encountered this research a decade ago, but when I revisited it recently, I realized something remarkable: this appears to be a rare case where sociologists and psychologists have discovered the same phenomenon by approaching it from different directions. It’s like we started excavating on opposite sides of the Earth, and now we’ve finally clanged shovels.
On the psychology side, we called this discovery the illusion of explanatory depth. As I’ve written before, people seem to understand the world exactly as well as they need to, and no better. Everyone knows how to use a toilet, for example, but they mistakenly think they also know how a toilet works.
I think this illusion is inevitable, given the architecture of the mind. According to predictive processing theory, the mind is always trying to guess what’s going to happen next, and it only updates its guesses when its expectations don’t match its experience. If you run that algorithm for a while, you’ll end up with a model of the world that is good enough, even if it’s not particularly good. You don’t notice the errors in your model until you run smack-dab into them, and every time you correct one, you assume that you’ve gotten them all.
From the sociology side, this same phenomenon can show up as something like moralistic therapeutic deism. People’s folk morality and folk religion, just like their folk physics, their folk biology, etc., are exactly as detailed as they need to be. Don’t hit, don’t steal, take turns, say please and thank you—almost everyone achieves this basic level of moral development, because otherwise you’ll end up friendless, if not incarcerated. Similarly, almost everyone acquires some vague notion of an all-seeing, sky-dwelling God through simple osmosis. (After all, He is kind of everywhere, literally and metaphorically.)
But beyond a handful of high-level principles, most folks remain mushy on the particulars. Smith and Denton didn’t perform the whole illusion of explanatory depth protocol with their participants, but I’m willing to bet that everyone feels like they have a coherent system of ethical and religious beliefs right up until you ask them any specific questions.
Our value systems are flimsy for the same reason our scientific reasoning is flimsy: we don’t fix our mental models until something tells us that they’re broken. So unless your pastor is popping quizzes during the sermon and cold-calling people in the pews, and unless you find yourself thrust into various Trolley Problems on a regular basis, there’s nothing that tells you that you need to firm up your moral and theological convictions. Why spend your days wondering about right and wrong, good and evil, sacred and profane, etc. when you don’t have to? You’re just going to get yourself into trouble, or even worse, into a philosophy PhD program.
This could explain why our sense of right and wrong is so easily overwritten by any half-decent system of rewards and punishments: we just don’t have much written there in the first place.
So how does anyone ever develop a value system that’s more than a few bullet points? I think, in short, that they look at a fish and they go insane.
You can summarize most of H.P. Lovecraft’s short stories as “Guy looks at something so messed up that he goes totally bonkers.” “Dagon” was among Lovecraft’s first: it’s about a guy who gets marooned at sea and encounters a really freaky fish, which upsets him so much that he ultimately throws himself out a window.
This is a useful metaphor, because I think at the center of everyone’s psyche is some version of the Big Horrible Fish: some unexamined assumption that, if you fully considered it, would require you to upend a significant portion of your life.
I saw my fish in 2021 in San Fransisco, during the annual meeting of the Society for Personality and Social Psychology.5 Inside the conference hotel were dozens of talks about stereotyping, prejudice, inequality, and so on. That had never struck me as strange until that year, when outside of the hotel, which was in the middle of SF’s Tenderloin district, people were sprawled out on the sidewalk, dying of overdoses. Scholars were scurrying past homeless people to give talks on social justice, which is ironic because “scholars will scurry past homeless people to give talks on social justice” is literally the main finding from one of the most famous psychology studies of all time.6
Something un-clicked for me that day. I had assumed that the answer to every dilemma was “more research”, and that eventually, all of the experiments being presented inside the hotel would solve the problems outside of it. But as I ate canapés in a ballroom mere feet from so much misery, I started to wonder whether any of us were even trying to make progress, or whether we were just publishing papers. I still believe that the barrier between us and a better world is a lack of knowledge; I’m just no longer convinced it’s that knowledge. I don’t think we will cure the human suffering visible on the streets of San Francisco by running another thousand studies on stereotype threat, ego depletion, and growth mindset, no matter what wonders it does for our CVs.
I saw many more fish after that, of species varied and grotesque. That’s why, on the rare occasions that I enter the conference hotel these days, my name badge does not have an academic affiliation on it, but instead simply says “guest”.
When I meet someone of uncommon integrity, someone willing to defy the incentives at any cost, I find they almost always have seen the fish—big, hulking fish, far larger than any I’ve ever encountered. They have undergone some kind of moral reckoning that undid their illusion of philosophical depth and gave them the chance to forge their own values. They’re the kind of person who could go to the gallows with a witticism on their lips, the opposite of a loser.
I used to see a guy like that every day, although he died a long time ago. Near the psychology department, there was a statue of Senator Charles Sumner, who is most famous for being beaten nearly to death by a congressman from the South in 1856. But that bit of historical trivia doesn’t do justice to the story of his life.
Sumner grows up poor in Boston, living amongst the city’s considerable Black population. On a trip to France in the 1830s, he’s astounded to see Black and white students studying together as equals at the University of the Sorbonne. He writes in his diary: “It must be then that the distance between free blacks and whites among us is derived from education, and does not exist in the nature of things.” He later becomes the most outspoken white abolitionist of his generation, taking positions even his friends think are crazy: the complete and immediate freeing of all enslaved people, followed by Black suffrage7 and racially integrated schools.
Then comes the infamous part: a few days after one of Sumner’s most inflammatory speeches, a congressman named Preston Brooks approaches Sumner at his desk on the Senate floor, accuses him of libel, and then to beats him with a cane until it breaks. Sumner, caught by surprise, is tangled up in his desk and cannot escape; he nearly dies. But then he recovers, and when he does, he becomes even more provocative. In 1861, he pushes Lincoln to abolish slavery in the Confederacy, which causes the newspapers in Sumner’s home state of Massachusetts to call him “a candidate for the insane asylum”. (Two years later, Lincoln issues the Emancipation Proclamation.) Sumner writes bills, arguments, and amendments that are often defeated or defanged, only for them to reappear 100 years later in Brown v. Board of Education and the Civil Rights Act of 1964.8
All that to say: I am guessing somewhere on Beacon Hill, at the Sorbonne, and in the puddle of his own blood on the Senate floor, Sumner saw the fish, and he refused to look away.
I understand why so many people are so concerned with incentives. It’s infuriating to watch villains succeed. And look, if we can stop rewarding people for doing the wrong things, we obviously should.
But I don’t know if we can. Goodhart’s Law, Campbell’s Law, and their corollaries state that every incentive is destined to be corrupted eventually, and so far, no one has proven these conjectures wrong. That’s why discussions around incentives often sound to me like we’re talking about a monkey’s paw that will grant our wishes, but only in evil, ironic ways. “The last time we made a wish on the Monkey’s Paw, it backfired horrifically. To rectify this, we have decided to make another wish on the Monkey’s Paw.”
In the eternal game of cat and mouse between the goodhearted people trying to fix the incentives and the Goodhart’d people trying to exploit the incentives, I am rooting for the cats. We should check whether our incentive systems are having their intended effects, and we should revisit and revise them accordingly. But for some reason, the second we start talking in terms of incentives, we also start acting like courage doesn’t exist and can’t exist. Bad incentives have become our generation’s version of “I was just following orders”, an excuse that should be an indictment.
So we should try to get our incentives right, but also—incentives be damned, do the right thing. If we seek an end to our permanent problems, we must cultivate in ourselves the same three things Charles Sumner said we should look for in a leader: “The first is backbone, the second is backbone and the third is backbone.”
Shout out to this post from Tal Yarkoni and this tweet from Alice Maz, which I saw long ago and no doubt spiritually inspired this post:
Only morons and cowards “respond to incentives.” Healthy people do what they want to do if reasonably practical, great people do what they want to do no matter what.
It was understood that these helmets, with their gnawed and rotted foam, would offer little protection if you actually got beamed in the head with a baseball. Their role was more ceremonial than functional, like a chef’s toque or a pope’s mitre.
As one participant in Smith and Denton’s study, a Muslim boy from California, put it:
God is like an entity that decides when, if, he wants to intervene with a lot of things. To me God is pretty much like intervention, like extreme luck. Say you’re $50 away from something and you find $50 on the floor, then that’s probably God’s intervention or something like that. But other than that it just seems like he’s monitoring. He just kind of stays back and watches, like he’s watching a play, like he’s a producer. He makes the play all possible and then he watches it, and if there’s something he doesn’t like, he changes it.
Survey research reaches a similar conclusion. For example, Catholics and Protestants spent several centuries murdering each other over doctrinal disputes. Today, however, only about half of Catholics know that they, unlike the Protestants, are supposed to believe in transubstantiation—that the bread and wine used in mass become the literal body and blood of Jesus. Meanwhile, about half of Protestants don’t know who Martin Luther was.
It took place in February, just before covid really kicked off in the US. I remember my advisor cautioning me not to go, and I dismissed him, citing the standard armamentarium of psychological biases: focusing illusion, scope neglect, availability heuristic. When I delivered my apology to him, it had to be over Zoom.
In the 1970s, a pair of psychologists recruited a bunch of seminary students to give a short lecture, either about career opportunities for seminary graduates, or about the story of the Good Samaritan. Some of them were told that they were late to give their lecture and they should hurry, or they were told they were on time or early. On their way to the building where they were supposed to deliver their talk, all the participants passed by a man slumped over in a doorway, groaning and coughing in the December cold. The participants in the “hurry” condition were less likely to stop and help; the content of their talk did not matter. As is common for studies from this era, there aren’t enough participants to be certain these results are robust, but I’m optimistic you could find an effect like this again.
For men, anyway. Sadly, Sumner was never much of an advocate for women. His biographer suggests he was probably gay, and he may have had some resentments toward the opposite sex. He exchanged saucy letters with a male friend of his, and was despondent when that friend married. Sumner himself remained a bachelor until in his 50s, only to have his wife immediately start cheating on him openly and then dump him a few months later. She told everybody he was impotent, which gave his political enemies no end of ammunition.
Sources for this section here, here, and here. For more, see this 600-page biography that came out last year.