Rewriting from ground up

You’ve got an existing, successful software product. You’ve hit the ceiling on extensibility and maintainability. Your project platform is inflexible, and your application is a software house of cards that can’t support another new feature.

You’ve seen the videos, the weblog posts and the hype, and you’ve decided you’re going to re-implement your product in Rails (or Java, or .NET, or Erlang, etc.).

Beware. This is a longer, harder, more failure-prone path than you expect.

Throughout my career in software development, I’ve been involved in Big Rewrite after Big Rewrite. I suspect it’s because I have an interest in learning eclectic computer languages, operating systems, and development environments. Not being just-a-Java-guy or just-a-Windows-guy has led to me becoming a serial rewriter. I’ve been on projects to replace C, COBOL, PHP, Visual Basic, Perl, PLSQL, VBX (don’t ask!) and all manner of architectural atrocities with the latest and greatest technology of the day.

In many cases, these Big Rewrite projects have resulted in unhappy customers, political battles, missed deadlines, and sometimes complete failure to deliver. In all cases, the projects were considerably harder than the projects’ initiators ever thought they would be.

This is not a technology problem. It’s not at all Rails-specific, but being in the limelight these days, Rails implementations are both very likely to happen and very risky right now.

Why So Hard?
So, why, in software rewrites, when you’re traversing exclusively familiar territory are the results so often unpredictable and negative?

Software as spec

"Make it do what it already does." That’s a tempting and simple way to view software requirements on a rewrite project. After all, the system already exists. The question of "what should it do when…" can presumably always be answered with: "what it already does".

There are two major problems with this assumption. The first, and most disruptive, is that the programmers don’t know what questions to ask. This is especially true if the programmers weren’t the original developers of the system (most often the case on a major technology shift), but even a programmer who did the original implementation of a product won’t remember every nook, cranny, and edge case. What’s worse, with the fragile safety net of an existing implementation, programmers can easily oversimplify the interface, and assume they know the capabilities of the system. If a combination of drop-down selections results in a whole new corner of the system, how are they to know without stumbling onto it (or performing an exhaustive and expensive test cycle)?

If the software you’ve built is complex enough that it needs to be rewritten, it’s probably also so complex that it’s not discoverable in this way. This means that domain experts are going to have to be heavily involved. It means that requirements are going to need to be communicated in much the same way they are on a green-field project. And it means that, unless it’s only used as a supplement, the existing system is more a liability to the rewrite than an asset.

Optimistic programmers might think I’ve missed something important here. If you’re rewriting a system, you’ve already got the code. The code can serve as the spec, right? Probably not.

Based on my own experiences and conversations with thousands of software developers around the planet, I unscientifically conclude that almost all production software is in such bad shape that it would be nearly useless as a guide to re-implementing itself. Now take this already bad picture, and extract only those products that are big, complex, and fragile enough to need a major rewrite, and the odds of success with this approach are significantly worse.

Existing code is good for discovering algorithms—not complex, multistep processes.

Invention or Implementation?

In his article, The C2I2 Hypothesis, programmer Zed Shaw criticizes the famous C3 project at Chrysler, which is known for being the birthplace of eXtreme Programming. He says that the project was an implementation—not an invention. An invention, according to Zed, is something new which needs creativity and high customer involvement, whereas an implementation is a project which participants (including programmers) have done before.

According to Zed:
If that’s the case, why was the customer involved all the time? They had a completely working specification in an already working system. Replacing it is more a matter of reverse engineering than gathering vision, customer feedback, use cases, stories, or any of the other crap the XP team used.

Here’s the problem: when does the label "Payroll System" become so broad that you don’t know if it’s an invention or an implementation? Could it be possible that at a huge company like Chrysler the payroll system was unlike any other payroll system that had ever existed? And, within the realm of this possibility, might it also be possible that Chrysler were redesigning the system, because through such changes as globalization and evolving international tax and labor laws, the system which was being replaced was no longer valid?

My point isn’t to say Zed is wrong. He makes some excellent points and may very well be right (though, knowing some of the guys on the C3 project, I’d guess they knew the difference between invention and implementation).

My point is that it’s not always clear cut which things are implementation and which are invention. Worse than being ambiguous, it’s often not clear that it is ambiguous. My experience says that most of the time, people doing the Big Rewrite will assume that they’re doing an implementation and will staff and estimate accordingly.

Most of the time, they’re wrong.

The wish list

Imagine going to the hospital for a kidney transplant, and before and during the surgery saying to the surgeon: “Oh, and while you’re already in there digging around, I’ve had some problems with my lungs that could use a little attention. And, yes, I’ve been overeating terribly—-could you do one of those stomach reduction things I hear about? And on that note, how about a little plastic surgery since we’ve got the knives out?”

This is effectively what happens on The Big Rewrite. An existing product, no matter how successful, always has a few warts. The rewrite is seen by many people as the perfect opportunity to shave off the warts. If we’re going to do it over again, we might as well do it right this time.

Under the veil of a rewrite, the assumption is that the personality and capabilities of the software aren’t changing. So, what might start as just a few little tweaks will usually turn into an unbridled reinvention, with none of the usual checks and balances that go into new product development. With potentially many stake-holders involved and an uncontrolled process, I’ve seen little tweaks end up increasing the total effort and feature-set of a Big Rewrite by as much as 100%.

The Big Bang

When you do a technology rewrite, you want things to be clean. That’s usually a major goal in a project like this. And at the beginning of a Big Rewrite, while you’re still wide-eyed and hopeful for your application’s ultra-elegant, scalable, maintainable future, you’re faced with a question: Should we deliver the Rewrite incrementally or all in one big release? Now, imagine your existing infrastructure is a home-grown Oracle Pro*C-based CGI framework with its own cookie-based authentication mechanism which relies on carnal knowledge of an aging mainframe ERP system. Incremental deliveries means making the new technology work within the dirty framework of the old system. One big release would mean we could just turn off the old system, turn on the new one, and keep our new efforts isolated and pristine.

In most cases, it’s the Big Bang approach that wins the argument.

Now picture yourself as a developer or project leader nine months into this project. The old system, still in production, has been patched and enhanced along side the new one as you’ve been developing it. You haven’t had time to keep up with each and every change that took place in the old system. As a result, on top of behavioral changes, you’ve got an ever-evolving database schema to port to the new platform. And the new system’s wish list has gotten so out of control, that there are major differences between the old and the new. To top it all off, working from the old system as a specification didn’t work, and you’re way behind schedule due to misunderstood requirements and rework.

The table has been set. The guests are on their third course. And now you have to come along and replace the table on which they’re eating without disturbing their meal.

On a big system with a lot of customers, data migration can be a huge problem. Not only do we have to keep track of what gets migrated when, but we have to actually perform the migration at some point. The Big Bang sounds like a lovely idea until you get to the actual event, and you realize it’s kind of like preparing for a world title boxing match when you know it’s the first and last time you’ll ever compete. The processes and software you have to create, the attention you have to pay before you can create an event like this is often as consuming, complex, and potentially disastrous as the system development effort itself.

But by making it a Big Bang release, you’ve maximized the chances that you’ll be behind schedule when you get to the end, and you’ve therefore maximized the chances that you won’t spend enough time preparing. This results in a bad time for both you and your customers.

Unfortunately, perhaps due to something intrinsic in human nature, this scenario is a cliche for Big Rewrite projects.

Justification and Lies

To add to the stress of the Big Bang comes another, mostly people-related issue. Almost all technology rewrites are driven by some technologist. Behind almost every technologist pushing for a Big Rewrite is a business person saying “But, why?” The question is valid. The product already works. It’s successful enough to even consider re-plumbing it, so we must have already gotten something right, no?

So, then come the justifications. They start with the real reasons the software is being rewritten (but usually censored to avoid the technologist looking like he or she screwed up big time on the initial development of the product). The system will be more maintainable. It will be easier to add features. “Oh yea? So we can do more features faster?” “Uh, yea.” “How much faster?” And so on.

As those discussions get heated and prove unsatisfactory, the list of promises gets longer. The system will be more scalable. System response time will improve for our customers. We will have greater uptime. And so on.

It’s rare, in fact, that a technology rewrite can deliver on all these fronts. A J2EE Web application may not prove in practice to provide higher availability than a mainframe application. Rails might be a more flexible and productive environment for a developer, but Rails apps slightly underperform equivalent PHP apps. So, you don’t sell Rails as something that will be faster than PHP. You sell it as something that is more flexible and maintainable, and will perform reasonably compared to a PHP application.

The piles of justification lead to piles of additional work and/or piles of mismatched expectations and disappointment after release.

Who's Tending the Store?

While we’re all in the back creating the next revision of a product, who’s tending to the day to day issues of the existing product? Typically, it’s the domain experts and the original implementers of the product.

Regardless of our intentions, day to day life and in-your-face time-sensitive issues can very easily steal all of the attention from a Big Rewrite. Screaming customers need their problems solved. Outages and serious bugs need to be fixed. Enhancements have to keep rolling in if your project takes as long as projects tend to take. Somebody has to do these things. Training new people is hard and doesn’t seem to make sense. If we’re getting rid of a system, why would we train someone how to maintain it?

So, the experts keep the old system running while the new system is being built. So, who builds the new system? Not the experts, that’s who. Usually, it’s people like me: technology experts. And while we’re banging away at the existing system’s UI, trying to figure out what needs to be coded, the domain experts are doing their jobs. Unfortunately, this means the domain experts aren’t watching the Big Rewrite very closely. Regardless of how good the team, if communication is impaired between the domain experts and the technology experts, things are going to move slowly, and wrong software is going to be created.

Many entrepreneurs idolize Steve Jobs. He’s such a perfectionist, they say. Nothing leaves the doors of [url=http://en.wikipedia.org/wiki/Infinite_Loop_(street)]1 Infinite Loop[/url] in Cupertino without a polish and finish that makes geeks everywhere drool. No compromise!

I like Apple for the opposite reason: they’re not afraid of getting a rudimentary 1.0 out into the world.

Quote:

“No wireless. Less space than a nomad. Lame.” — cmdrtaco, Slashdot.org, 2001, reviewing the first iPod

I remember my first 1G iPhone. Like a meal you have to wait for, or a line outside a club, the fact that I stood in line for hours made the first time I swiped to unlock the phone that much sweeter. It felt like I was on Star Trek and this was my magical tricorder… a tricorder that constantly dropped calls on AT&T’s network, had a headphone adapter that didn’t fit any of the hundreds of dollars of headphones I owned, ran no applications, had no copy and paste, and was as slow as molasses.

Now, the crazy thing about that release is when the original iPhone went public, flaws and all, you know that in a secret room somewhere on Apple’s campus they had a working prototype of the 3GS with a faster processor, better battery life, normal headphone jack… a perfect everything. Steve Jobs was probably already carrying around one in his pocket. How painful it must have been to have everyone criticizing them for all the flaws they had already fixed but couldn’t release yet because they were waiting for component prices to come down or for some bugs to be worked out of the app store.

Quote:

“$400 for an Mp3 Player! I’d call it the Cube 2.0 as it wont sell, and be killed off in a short time… and it’s not really functional. Uuhh Steve, can I have a PDA now?” — elitemacor, macrumors.com, 2001, responding to the original iPod announcement

Or, I wonder, are they really quite zen about the whole thing? There is a dark time in WordPress development history, a lost year. Version 2.0 was released on December 31st, 2005, and version 2.1 came out on January 22nd, 2007. Now just from the dates, you might imagine that perhaps we had some sort of rift in the open source community, that all the volunteers left or that perhaps WordPress just slowed down. In fact it was just the opposite, 2006 was a breakthrough year for WP in many ways: WP was downloaded 1.5 million times that year, and we were starting to get some high-profile blogs switching over. The growing prominence had attracted scores of new developers to the project and we were committing new functionality and fixes faster than we ever had before.

What killed us was “one more thing.” We could have easily done three major releases that year if we had drawn a line in the sand, said “finished,” and shipped the darn thing. The problem is that the longer it’s been since your last release the more pressure and anticipation there is, so you’re more likely to try to slip in just one more thing or a fix that will make a feature really shine. For some projects, this literally goes on forever.

Quote:

“hey – heres an idea Apple – rather than enter the world of gimmicks and toys, why dont you spend a little more time sorting out your pathetically expensive and crap server line up? or are you really aiming to become a glorified consumer gimmicks firm?” — Pants, macrumors.com, 2001

I imagine prior to the launch of the iPod, or the iPhone, there were teams saying the same thing: the copy + paste guys are *so close* to being ready and we know Walt Mossberg is going to ding us for this so let’s just not ship to the manufacturers in China for just a few more weeks… The Apple teams were probably embarrassed. But if you’re not embarrassed when you ship your first version you waited too long.

A beautiful thing about Apple is how quickly they obsolete their own products. I imagine this also makes the discipline of getting things out there easier. Like I mentioned before, the longer it’s been since the last release the more pressure there is, but if you know that if your bit of code doesn’t make this version but there’s the +0.1 coming out in 6 weeks, then it’s not that bad. It’s like flights from San Francisco to LA, if you miss one you know there’s another one an hour later so it’s not a big deal. Amazon has done a fantastic job of this with the Kindle as well, with a new model every year.

Usage is like oxygen for ideas. You can never fully anticipate how an audience is going to react to something you’ve created until it’s out there. That means every moment you’re working on something without it being in the public it’s actually dying, deprived of the oxygen of the real world. It’s even worse because development doesn’t happen in a vacuum — if you have a halfway decent idea, you can be sure that there are two or three teams somewhere in the world that independently came up with it and are working on the same thing, or something you haven’t even imagined that disrupts the market you’re working in. (Think of all the podcasting companies — including Ev Williams’ Odeo — before iTunes built podcasting functionality in.)

By shipping early and often you have the unique competitive advantage of hearing from real people what they think of your work, which in best case helps you anticipate market direction, and in worst case gives you a few people rooting for you that you can email when your team pivots to a new idea. Nothing can recreate the crucible of real usage.

You think your business is different, that you’re only going to have one shot at press and everything needs to be perfect for when Techcrunch brings the world to your door. But if you only have one shot at getting an audience, you’re doing it wrong.

After the debacle of the 2.0 -> 2.1 lost year of 2006 the WordPress community adopted a fairly aggressive schedule of putting a major release out 3 times a year, and we stuck to it fairly well although in 2009-2010 we’ve slacked a bit, falling into the “one more thing” mentality again. But more fundamentally it’s still shrink-wrap software, which means that updates burden its users in some way so we have to spread them out.

That’s why I love working on web services and pretty much everything Automattic focuses on is a service. On WordPress.com we deploy code to production twenty or thirty times a day and anyone in the company can do it. We measure the deploy time to hundreds of servers and if it gets too slow (more than 30-60 seconds) we figure out a new way to optimize it. In that short rapid iteration environment the most important thing isn’t necessarily how perfect code is when you send it out, but how quickly you can revert if you need to so the cost of a mistake is really low, under a minute of brokenness. Someone can go from idea to working code to production and more importantly real users in just a few minutes and I can’t imagine any better form of testing.

Quote:

“Real artists ship.” — Steve Jobs, 1983

A version 1.0 of this essay appeared in the book Do More Faster. I should also note that Automattic is always hiring.