Why Greatness Cannot Be Planned (#34)
In exploring “habits, projects, and goals,” I realized the need for planning and reflection even in an open-ended pursuit. For my sabbatical, time isn’t infinite; I can’t afford to wander aimlessly without any sense of direction. But what kind of planning is needed? How much? These questions led me to the book Why Greatness Cannot Be Planned by AI researchers Kenneth Stanley and Joel Lehman.
Deceptions
They argue that in ambitious pursuits, an objective-driven approach is not just useless but harmful.
Embarking on an ambitious endeavor means navigating without a clear map. If we mechanically pursue the objective, we would predict and pick the stepping stones toward that goal. Yet, they rarely resemble the final product. For example, people set out to invent the first computer won’t meticulously study vacuum tube, but the tubes turned out to be critical. Stepping stones or measures of success are often deceptive. If we focus only on them, they can lead us away from the goal.
The logic of discovery
Rather than focus on the final destination, it helps to collect stepping stones for their own sake. They’re interesting, and they can lead us to more.
Why does chasing interestingness get us closer to ambitious goal? How is it different from a random search?
People want to measure their progress because it provides information. But as mentioned, objectives can be deceptive. Interestingness provides no lack of information. It’s just a different kind. It’s based on the past, on what we’ve seen, rather than the unknown future. The past usually provides richer information.1
Another way to look at this is that searching for interestingness results in a certain order. The order is not based on good or bad in light of the objective — again, such judgment can be premature — it’s a relative measure because what was novel doesn’t stay novel. The ordering is from simple to complex.
The evolution from simplicity to complexity entails accumulation of knowledge, as finding something new requires learning about the world. I use the word evolution because it works similarly:
As Stephen Jay Gould has pointed out in evolution, once all the simple ways to live are exhausted, the only way to create a new species or niche is to become more complex. In other words, there are only so many ways of being a bacteria. That’s why increasing complexity is almost inevitable if evolution is to continue. But these increases in complexity are not arbitrary. Rather, they reflect the properties of the world in which evolution takes place: Eyes represent the presence of light in the universe. Ears signify mechanical vibration. Legs are reflections of gravity, and lungs of oxygen.
Personally, following my curiosity makes sense because I end up learning about the world and myself. I don’t need the objective to keep my discovery bounded because the world provides its own constraints.2
One caveat is that this approach doesn’t guarantee success when the goal is too ambitious, just like objective-driven approach. (The authors tested it with their novelty search algorithms.) The bright side though, is that we will find something interesting; we just can’t predict what it is.
Be a treasure hunter
I find the logic of discovery illuminating. It prompts me to be more attuned to my curiosity and follow its lead. It also tells me to focus more on the stepping stones I’ve collected and decide where to go based on where I am, rather than where I should or want to be.3 Successful inventors don’t waste effort on far-off grandiose visions but concentrate on the edge of what’s possible today.
This approach challenges us to let go of the comforting notion of a “right path.”
It’s tempting to think of progress as a set of projects, some of them on the wrong path and some on the right. […] Instead of judging every activity for its potential to succeed, we should judge our projects for their potential to spawn more projects. If we really behave as treasure hunters and stepping stone collectors, then the only important thing about a stepping stone is that it leads to more stepping stones, period. […] The more stepping stones we find, the more opportunities there are to depart to somewhere greater.
Criticisms
How do we decide what’s interesting then? It seems to say nothing more than follow your gut feelings, which is one criticism of the book.
I find the major weakness is that it can explain away any successful objective-driven pursuits as unambitious. The Manhattan project is an example: was it not ambitious enough? This makes the theory irrefutable. I think in practice, the difficulty lies in judging whether something is just one stepping stone away, or so ambitious that we should dismiss objectives completely.
Implications for other areas
The authors explore the problems of objective thinking and how we can do better following a treasure-hunting strategy.
Education
Measures come from an objective-driven mindset. We already know that measurement ceases to work once they’re known — map is not the reality, and people find ways to get away from it or even make things worse.
But still, we were taught to stay away from those who have no concrete goals for the future — we’re supposed to have a plan and a purpose, and pursue it wholeheartedly.
Education falls under the problem as it heavily relies on assessment. The authors envision an alternative where teachers have more freedom to try different ideas rather than rely on uniform measurement. Progress can be measured in a peer-driven way: schools are assessed based on their own situation. They believe that in societal efforts like education, we might make better progress by exposing each other to potential stepping stones to new ideas.4
Innovation and consensus
The authors discuss how granting agencies for scientific research are often based on consensus, and caution that it works against exploring stepping stones. A different approach might be to reward maximal disagreement (e.g. a project that receives “excellent, excellent, poor, poor” could be more interesting than “excellent, excellent, excellent, excellent.”)
Judgmentalism is the natural habitat of the objective-seeker, always worried about where everyone else will end up. But we are all better off in the end if we end up at different places. Otherwise, everyone would be standing on the same stepping stone. That’s why we need to beware of the seduction of consensus. Of course, if we all want to end up in the same place, the right place, then it makes sense to push towards consensus. But that should be the last thing we want. Disagreement and divergence are virtues that deserve to be protected.
One such example is to judge projects based on the importance of their predicted impacts. The authors argue that even if we can reliably estimate impact, such “importance” can be deceptive, as important discoveries are often serendipitous. It’s better to pursue interesting projects that lead to further unanticipated experiments, rather than focusing on their own importance.5
Why is objective thinking so alluring? One reason is our fear of risk. But risk-averse objective thinking limits progress.
Consensus, perceived importance, alignment with national interests—these are “objective” parachutes for escaping the great unknown when we should be rocketing further into it.
Natural Selection
People interpret “survival of the fittest” differently, with one view linked to objective thinking: competition leads to fitter organisms. Under this view, evolution is progressive — we’re objectively superior to bacteria. But the authors argue that if evolution has the goal, it was already accomplished because bacteria survived and reproduced. Rather, selection restricts exploration — a Gentle Earth with no selection pressure would produce more creative organisms.
“Survival and reproduction” is more of a constraint than objective, and evolution is more like a treasure hunter than an optimizer. As the authors argue, the main engine behind evolution’s creativity is searching for many ways to solve the same problem, and competition only plays a secondary role.
Why doesn’t competition cause evolution to converge on an optimal solution? One reason is that competition in evolution isn’t global; each organism doesn’t compete with every other organism in the same way. Global competition seeks the best overall, while local competition encourages the founding of new niches to escape competition.
While the book didn’t entirely shift my perspective, as I didn’t subscribe to the all-powerful objective thinking to begin with, I enjoyed reading it and learning about why an objective-less approach works and how it connects to a wide range of things.
I hope you find it helpful. If there’s anything you’d like to share, feel free to say hi on X or leave anonymous feedback here.
Until next time,
Weichen
The idea that the past provides more information is related to Taleb’s insight that systems learn by removing parts, not adding.
Another parallel to Taleb’s idea: we should use the constraints from the world to our advantage, by having skin in the game.
The idea of starting from here and now aligns with Designing Your Life, which suggests as the first step. It also reminds me of mindfulness concepts.
The idea of “good reply game” can be thought of as giving others stepping stones.
The idea to pursue interesting projects aligns with Karl Popper’s idea that good theories should produce more interesting problems to solve.
Thumbnail photo by Birmingham Museums Trust on Unsplash