Design Is How It Works
Software is manifested theory: an insightful opinion, often contrarian, about the matter at hand. This view says something about the team-to-product relationship and how to make better products.
Is Software Magic?
You can still speak with people whose career included punch-card data crunching. I worked for years in a phase where software was firmly tied to the hardware that ran it. For a while we had the Application Service Provider (ASP) model in which all that was still true but you paid someone else to do it and used a crappy remote connection software like Citrix to get to the software1.
Then there was single-instance multi-tenant Software as a Service (SaaS) tightly coupled to hardware scalers (you paid per hardware usage, essentially). Now there are serverless offerings where a customer has no actual relationship to the physical layer. That’s a long way in one career: from manipulating laminated paper like a kid’s craft project to ubiquitous and hidden calculation.
As software becomes further abstracted from calculation it is worth pausing a moment and reflecting on what it is. At this point software is starting to resemble magic spells. We move our wand (mouse) and say the incantation (type something into a webform) and stuff just gets done. But SaaS isn’t magic: it is theory.
I’m not the first or only to propose that software is theory. This line of thinking has been here all along. Consider the 1985 Programming as Theory Building by Peter Naur. He posited that developing software is not about the execution of sequential build steps, but primarily about forming an insights or theory of the matter at hand of their desired product. In practical terms, if you make an app about habit tracking (for example) the engineering is less about completing a cascade of necessary steps to code it so much as forming a novel and correct insight which would organize the resulting habit-tracking experience. Peter Naur chose to describe this process as theory building, and connects it back to philosophical notions such as Karl Popper’s world three and two realized objects.
Both Peter Naur and later writers emphasized the role of the software creators (i.e. the team) as the most interesting aspect of viewing software as theory. For example, here is Baldur Bjarnason2.
Software is the insights of the development team made manifest. ... The written code is how software interacts with end users and other software systems. The insights and knowledge that exist in the minds of the developers are how software lives, changes, and grows. Without the code, it has no way to interact with the real world. Without the knowledge in the minds of the developers, it has no way to adapt and survive.
Ken Kocienda's book Creative Selection describes this approach as central to Apple's software development during his tenure there and when working on blockbuster products like iPad and iPhone. Their approach to product as theory manfests in things like a focus on taste over data. As he says when contrasting Goole and Apple:
Google factored out taste from its design process. At Apple, we never would have dreamed of doing that, and we never staged any A/B tests for any of the software on the iPhone. When it came to choosing a color, we picked one. We used our good taste—and our knowledge of how to make software accessible to people with visual difficulties related to color perception—and we moved.
To restate Peter Naur, Ken Kocienda, and Baldur Bjarnason view in brief, I believe they see the software as an incomplete manifestation of theory, noting that only the team who built it contain the full theory. They would see software products as akin to experimental results that were accurately predicted rather than the theory that predicts them. I agree with this perspective, and note some confirming facts it explains:
Software can be experienced without being easily duplicated. If it has a theory, that theory isn’t fully clear to the users.
Engineers coming to a product without a chance to speak with its originators often struggle to internalize what it does, how it works, and why it was made as it was. As per point 1, if the software has a theory it is also not obvious to later engineers who can access its code.
Probably the easier analogy to use here is music. The songs of artist like Bob Dylan or Taylor Swift manifest their theory about music, but we recognise that the songs are incomplete artifacts vis-à-vis this theory: one cannot listen to a Bob Dylan album and make its natural follow-up. Bringing it back to software, this is similar to how Apple’s focus on taste results in outstanding user experience in their products which are nonetheless not imitable by competitors even when such choices have made them the most valuable tech company in the world.
Most of the proponents of this view go on to focus on the higher importance that should be placed on the team who makes products. If you believed the creation and survival of software products was akin to artistry, you’d simply behave differently as a leader of such people. It would call for different recruitment, advancement, collaboration, and retention strategies. I agree with these conclusions, but want to focus the rest of the article on something else that this view of software-as-theory leads to. I ask myself: if software creation is theory-building then what can we learn from the way scientific theory is formed, competes for domination of a field, and is eventually discarded?
How Does Theory Advance?
I’m no scholar of the history or philosophy of science. But we’ve all heard of paradigm shift as proposed by Thomas Kuhn. His book The Structure of Scientific Revolution is the most cited source in social science publishing. I’ll just quote his Wikipedia article directly:
Kuhn made several claims concerning the progress of scientific knowledge: that scientific fields undergo periodic "paradigm shifts" rather than solely progressing in a linear and continuous way, and that these paradigm shifts open up new approaches to understanding what scientists would never have considered valid before; and that the notion of scientific truth, at any given moment, cannot be established solely by objective criteria but is defined by a consensus of a scientific community. Competing paradigms are frequently incommensurable; that is, they are competing and irreconcilable accounts of reality.
There are a few key points I’d like to draw attention to.
While continuous refinement does happen, real progress is infrequent, abrupt, and transformational
Scientific truth is in part a collective agreement among domain participants.
Competing theories are difficult (or perhaps just pointless) to compare in their own terms.
I think these points are incredibly germane to theory building for software products. Let’s start with the paradigm shift aspect. Successful products create discontinuities in their markets. Your phone's SMS didn’t get better a bit at a time until it was free to use around the world and could handle GIFs, videos, group chats, and provide real-time location pings. WhatsApp did those things. In product creation we call this by terms like disruption, winner-take-most, power-law distribution of success, etc. Its why VC funds flow to high-risk high-reward software start-ups, and it maps nicely to how theory advances in infrequent, abrupt, and transformational steps.
Second, Kuhn’s point about scientific truth being a consensus of the community is something I also see in software products. A great example is how software vendors and buyers self-organize around a category: its name, its minimum requirements, and the axes on which competitors should fight for market share. Gartner collects a tidy tax on this tendency by just typing it up as Magic Quadrants and other memorabilia. My point here is that the existence of a category like Transport Management System (TMS) and the basis for comparing various TMS vendors are all there by consensus of the community. There is nothing to say objectively that functionality for transport planning and execution should be together in a package, and this is exactly the sort of thing that a community believes collectively at a moment and then later changes their mind about (again, in consensus).
Third, Kuhn describes theories as difficult to compare in their own terms. He’s not saying it’s all subjective, just that it’s kind of pointless to expect theories to have meaningful perspectives on each other. This again aligns to my experience with software products, and most concretely with their teams as the full owners of the theory. Have you ever wondered why a team whose product is failing doesn’t simply attempt to fast-follow a winning product? This may be ill advised for business strategy reasons (never run your competitor’s playbook), but I think it’s more basically ruled out because the failing product’s team cannot yet see the winning product’s theoretical basis because they are trying to do so from the lens of their existing theoretical framework. For example, I have seen product teams that had a foundational belief in the primacy of the desktop user not be able to properly internalize the importance of multi-screening or mobile access. At first this can look like poor decision making driven by lax prospect research, fallacious sunk-cost thinking, a fear of the internal politics of blame, or just avoiding taking on a hard piece of workload. But a deeper reason seems to be that the worldview that leads to a decision is simply not well suited to judge other worldviews accurately.
How to Apply This to Product Creation?
Bring a theory
The most valuable tip to take from this article is the obvious one: your product should be a manifestation of a theory on the subject matter it purports to deal with. I am astounded by how few product teams realize this and act on it. Kuhn proposed a set of five points which he'd expect a new paradigm to achieve.
Accurate – it aligns with reality
Consistent – internally consistent, but also externally consistent with other theories
Broad – a theory's consequences should extend beyond that which it was initially designed to explain
Simple – adopting this new theory simplifies rather than complicates the domain
Fruitful – a theory should disclose new phenomena or new relationships among phenomena
I see elements of this in new products, and would adapt them to make a checklist for new products as follows:
Functional – it instructs us how to do a job at hand
Connectable – you can use the theory well with surrounding theories / products.3
Big enough – it’s a theory of a product not a feature
Simplifying – it makes the customer's world simpler not more complex
Valuable – it creates value for the customer by resolving current tensions as yet unaddressed or by enabling a new range of things (acts, products, services, etc) of obvious worth
How to Make Theory
How does one formulate a theory to begin with? The act of creation is hard to pin down, in a similar way that one cannot be told exactly how to craft a good song, perform well in theatre, or bring paint to canvas for a work of art. But I think I can give an overall pattern to look for and then a specific tip about how to make the most of creative insights.
I think great software products will have the following path. First, they are contrarian in a specific way. Second, their contrarian act (as manifested in the product) is insightful. It offers a perspective on “the job to be done” such that, if you agree with the contrarian view, the world is better or clearer in some meaningful way. Third, the world converges to this view via meaningful contact: it begins contrarian but ends as orthodox, and the transition happens naturally as the product’s perspective infects and converts the users. Good products win the argument with the world of customer expectations in which they began as the contrarian outsider. In product management we call this milestone “product market fit”.
The contrarian focus is directly connected to how products are a manifestation of theory. Emerging theories are, by their nature, contesting orthodox theories for their ability to best explain the world. These contests can be about the edges (i.e. refinement of earlier base theories) or at the core (a wholesale replacement). So too with products as manifestation of theory. And in both scientific theory and SaaS products the winner induces a kind of release of tension when encountered. I associate this feeling with the idea of parsimony, i.e. when someone gives me a contrarian (but superior) perspective it feels like a simplification of the world. In that act of simplification, tension I had been unknowingly holding is released.
Readers may now be wondering, what in the world does Jonah mean by a release of tension from great new products? What I am referring to here is how people in a domain are constantly exposed to anomalies in how things should work. These anomalies frustrate us, but often without a viable way to solve them we simply downplay them. Later, when a new approach to the domain is available that removes these anomalies we quickly allow the frustration at the past situation to be voiced. In effect, a tension was building and then released, and in doing so it made a return to the old situation quite negative. As Erik Hoel4 describes:
“A field is [ready for theory] when it is fundamentally in crisis, a crisis stemming from some deep and unexplained question that influences everything but is ignored.”
Which leads to the way that successful new products convert users to their contrarian theory. By example: you always hated taxis. They were expensive, their costs were hard to predict, they were hard to find, they were unreliable, they were opaque and impervious to customer satisfaction ratings. And yet we rarely spoke of these things until after on-demand ride hailing (Uber, and their copy-cats) emerged. And then we all collectively let ourselves see the orthodox option for all its flaws. On-demand ride hailing isn’t just better; it’s obviously better in a way that calls into question why we hadn’t thought of it previously. From origins that very often were outright illegal, nothing about on-demand ride hailing is contrarian today. It is simply the best way we know to do one-time mobility. This is an example of products as theories, and what happens when the theory wins. Contact with the product converted users to see the world from its theoretical view: and that created the consensus for a paradigm shift.
Attend to Anomalies
My last advice for product leaders seeking to create products that embody a new and better theory is to attend carefully to anomalies. Great product creators notice. Consider again the pattern I gave above: there needs to be an insight manifested in the product (i.e. available to the users not just the creators) that relieves tension because it addresses previously ignored anomalies. The best way to get there is for the product creating team to be sensitive to and documenting anomalies in the dominant products. In other words, if you are a good product visionary one of your habits would be to know what are the current best products and to be able to detect and introspect about the anomalies that exist with them. This is a rare skill. Going back to Uber: almost no one was detecting the anomalies mentioned and attending to them for a prolonged time within the focus of their inner world. Building this skill (or habit perhaps) seems possible. Ken Kocienda wrote about how an attention to experience and aesthetic was part of the core group at Apple under Steve Jobs, but also how this group was cultivating as much as selecting for these attributes in their team.
Footnotes
If the ASP model of software (such as via Citrix remote access) was a nightmare for users we must not have voiced that too much at the time, i.e. we suppressed it because nothing better existed. Until it did, and then we became rapidly aware and unwilling to accept those frictions in the user experience. This is an important pattern in how new products change the basis of measuring what a domain expects.
I am not a fan of his writing in general, but for those interested see his full thoughts here on how software as theory relates to product's team.
Strangely, this is something products often fail to ensure. My go-to example is the electrical plug situation. Everyone agrees the world would be better off with one kind of electrical plug (Brazilian Type N obviously). But there is no on-ramp to get ourselves there: it simply won’t happen without regulatory mandate and only the EU as the high place of workship for regulations would try to mandate an across-the-board wiring change. Yet, product teams often hand-waive adoption steps on the basis that the resulting solution (if widely adopted) would be amazing.
This quote comes from Erik Hoel’s book The World Behind the World, a look into the current state of concisouness research. I'm a fan of his Substack The Intrinsic Perspective as well. His article on aristocratic tutoring as a model for education was been influential on me.