Eureka!

A recent story in the New York Times about subway system mechanical failures ('$1 Billion Later, New York’s Subway Elevators Still Fail,' William Neuman, May 19,2008) reads like a software implementation project gone bad: insufficient training of service techs resulting in poor maintenance and frequent downtime; managers pushing for deliverables before they are ready for use, basic design flaws contributing to system instability. It reminded me of a question Scott Rosenberg posits in his fascinating study of software development, Dreaming in Code: “Why can’t we build software the way we build bridges?” (And yes, the book acknowledges that unfortunately we sometimes do build bridges the way we build software.)

With a career-long involvement in software development and implementation, I am fascinated by the question of why ‘Software is hard.’ I suspect that only software engineers think their discipline is uniquely difficult, but any software user (that’s everyone) probably does wonder why good software is so rare.

Creating innovative software usually starts with a Eureka! moment: a single person has a vision of creating something that is, in a word, transcendent. All of the complexity is stripped away as the pure, essential solution reveals itself to the visionary. Impelled by the epiphanic euphoria of the moment, the designer crafts a high-level design that retains as much of the purity of the Eureka vision, while acknowledging some of the messiness of actual data and user requirements. Through design and coding, more of the messiness of reality has to be dealt with: programmers have to bring both a creative mindset (any problem can be solved an infinite number of ways programmatically) and the discipline to create within logical structures.

The logic of even apparently simple software can be extremely dense. From a programmer’s perspective, the technical skills (programming language, operating system, database, etc.) differentiate competency far less than the cognitive skills (logical and critical thinking). Reading another person’s code gives you an intimate insight into how that person thinks (although not what that mind thinks about). This is true for one’s self as well. It’s not uncommon to revisit one’s own code written even recently and wonder at how much you knew then that you’ve now forgotten: as you re-enter the code, all of the variables have to be re-assigned in your own mind, as it were, as you immerse yourself into a mental construct that is unique to the time you spend embedded in a project.

I imagine this is no different in kind from other creative endeavors, such as architecture or orchestral composition. But we don’t expect architects or composers to be commonplace. A modest software company may easily have 30 – 50 people responsible for creating software – that is, actively engaged in the creative process. Just assembling that number of capable people is a huge undertaking. Managing the collaborative process is even more daunting. Translating that to other disciplines is unimaginable: can you imagine finding five Beethovens, much less 30, and then expecting the group of them to create Symphony No. 5? Yet, software is ubiquitous and we depend on it absolutely. Production levels must be high, since requirements change constantly and demand new solutions (software). We expect software to be continually new, and delight us with capabilities we had never imagined – as users we want to consume that Eureka moment and experience the thrill of a truly elegant and innovative solution.

With few exceptions, quality suffers as creative production rises. We lower our expectations. We’re willing to make trade-offs, and the software industry understands that. Finally delivered, the software has been re-imagined by programmers, test engineers, release managers, user input, and consultants who shape the final product. The inspired Archimedes shakes his head at what became of his vision.