Is software development a science or an art?

The software industry treats it as a science. It uses processes like MRDs, PRDs, and functional specs to convert customer needs into software that solves their problems. Various roles like product managers, engineering managers, project managers, architects, and programmers work together to drive the process like an efficient machine. Programmers are usually referred to as software engineers, but unlike mechanical, civil, and other kinds of engineers, software engineers don’t have any certification process or formal requirements. In fact, technically, software engineers aren’t engineers at all.

Many famous people in the field recognize programming as an art. Good code isn’t a commodity that’s just pumped out – it’s composed like a novel or music. Good software is created from good code using a good development process.

I’d argue that quality software is a melding of the two views – that software development is both a science and an art.

In his novel Zen and the Art of Motorcycle Maintenance Robert Pirsig explores the dualistic nature of beauty in technology.

Classical beauty, he explains, is the beauty of the way a technology works, the way all its parts work harmoniously together just as designed to achieve the desired effect. For a motorcycle, this could be how the valves, pistons, crank, and ignition system work together to efficiently convert fossil fuel into the rotational energy of the flywheel, which by way of the clutch, transmission, chain, and wheel is converted into forward kinetic energy of the motorcycle and its rider.

Romantic beauty, on the other hand, is the beauty that strikes the senses. For a motorcycle, this would be how it looks, sounds, and makes one feel as he rides it. While he wrote Zen and the Art of Motorcycle Maintenance in the 1970s, this classic/romantic paradigm provides an excellent ground for analyzing modern software development practices.

In our experience at Untangle, this classic/romantic split can explain many of the differences we see between open source and proprietary software. Proprietary software is typically designed in a top-down process where the customer drives the requirements. A product manager takes possible enhancements from the field and prioritizes them in a way that’s beneficial to the company’s future success – typically one that’s aligned with acquiring new customers. As romantic beauty strikes the senses, proprietary software is designed in a way that is optimized so that it’s romantically beautiful for the customer, while the development effort that enhances classical beauty is usually sacrificed as a lower priority because its immediate benefit to the customer is less visible.

Take Apple’s classic Mac OS, for example. Mac OS (up to OS9) was developed from 1984 to 2001. Mac OS was widely acclaimed to be much easier to use than competing operating systems. Over this 17-year period, Apple religiously focused on the user experience – continually making it better and simpler, but development effort towards classical beauty, the underlying implementation of the operating system, was a lower priority. By 2001, Mac OS9 was extremely easy to use but underneath was a Stone Age technology. Its lack of a command line made it inconvenient for power users, and its lack of true multitasking, virtual and protected memory, meant that by 2001 it was less competitive with its alternatives. Apple’s development process had led to a dead end – Mac OS was still romantically superior than most other options, but had become classically ugly to the point that Apple was forced to rewrite its entire architecture.

The problem was not that the engineering team undervalued the classical aspects of proprietary software. The problem was that as competitive pressures drove the company forward, the executive and product management team focused on the romantic improvements – those immediately evident to customers. Romantic improvements offer immediate benefits, whereas classical architectural changes are often harder to justify as a payout that’s not immediate or easily definable. For example, when Microsoft was pressured to deliver Windows Vista, it was forced to drop some of Vista’s most innovative pieces like WinFS (a relational database file system) and Palladium (a secure computing base). But you can be sure it didn’t drop all the user interface enhancements it saw in competitive software (like Apple’s).

Open source development methods, on the other hand, tend to focus mainly on classic beauty at the expense of romantic beauty. Many open source projects begin as a couple of hobbyists producing software for their own purposes. Because they’re their own customer, they don’t need documentation and don’t need a nice graphical user interface. Instead of focusing on enhancements that would improve the experience for other “normal” users, they tend to focus on improving the code and the implementation.

Take Linux, for example. Started in 1991 as the kernel of a free operating system, it continually focused on implementation. As opposed to the classic Mac OS, it had a command line, true multitasking, and virtual memory from the beginning. It has continued to innovate on its implementation, and the newest kernel has major innovations in scheduling, threading, networking, and hardware support. While the implementation excels in many aspects, basic things that would improve the user experience are left unattended. A basic pretty splash screen while the computer boots was not officially added until the latest version, something the classic Mac OS has had since day one. So while Linux is decades ahead in term of implementation and classic perspective, its decades behind from the romantic perspective and seemingly in no hurry to catch up.

In Zen and the Art of Motorcycle Maintenance, Pirsig argues that both classic and romantic beauty is crucial for quality (and is, in fact, subsets or children of quality). With a motorcycle, the combination is clearly important. Users expect engineering excellence (good horsepower and torque, excellent suspension, good gas mileage, good reliability), but users also expect motorcycles that inspire the senses. It should inspire when you see it and hear it and be fun to ride; otherwise there would be no point in owning one.

So while open source tends to create better classical solutions, proprietary groups tend to create better romantic solutions. Neither is wrong, but the customer expects both. Lately we’ve seen a group of successful combinations of the commercial open source process. In 2001, Apple replaced the classic Mac OS with Mac OS X. Mac OS X is based on an open source BSD kernel (an implementation of Unix). Because of this, Apple inherited all the classic advantages of Unix, like a command line and virtual memory, and maintained all the romantic beauty of the classic Mac OS. Likewise, Canonical, with its Ubuntu distribution, has taken the Debian GNU/Linux and built a distribution that caters to human beings. What results is an OS that has all the advantages of Debian GNU/Linux but is actually easy enough for normal people to use.

This combination of open source and commercial software is a growing trend and is at the heart of what we are doing at Untangle. Time and time again we’ve found when evaluating open source versus commercial offerings that open source shined. The reason for this is simple: we evaluate technology on its classical properties. When looking for a spam-detecting technology, we look at how well it detects spam, not how easy it was to use or how well documented or supported it is. Ease-of-use matters less because we intend to create a “whole product” offering by adding what’s important to the customer while keeping all the classical advantages we inherit by using open source.

By taking open source software and making it easy-to-use and providing support and documentation, a higher-quality offering is created that excels in both the classical and romantic sense. Over time, this will be a growing trend as software customers, like motorcycle customers, will expect both. As a result, those companies quietly harnessing open source in their proprietary product will begin to boast of it being an advantage as users begin to associate open source with high quality. Many companies will start to open source their code themselves. Closed source companies will get on the defensive, justifying their closed source methodologies and harassing their open source counterparts with FUD (fear, uncertainty, doubt), while continuing to struggle technologically.