Welcome to the new Gallery website!
Gallery has a new website! It's been many months in the making and has taken a great deal of time from many people. We chose Drupal as our content management system and have transitioned all the data from our old website over to a shiny, new extremely fast server. The road to this new site has been long and rocky at times. Read more to hear the story of what we did and how we did it.
In the beginning...
The Gallery website has been around in one incarnation or another since about 2000. In the early days, it was just couple of HTML pages on an old server in a closet behind a DSL line. As the project started to grow and more people got involved, it was a drag to keep updating the HTML by hand so in December of 2001 we cast around and found PostNuke. A full-on content management system was far superior to simple HTML pages in that it enabled us to have many site administrators with varying degress of permissions. This lightened the load on our main developers and freed them up to spend more time writing Gallery code.
As the project grew, the website became more and more popular. Eventually we had to move it off the DSL line and onto a colocated server better suited to handle the site's resource requirements. As we made improvements we found that every time we reduced the latency on the website we received significantly more traffic, so optimizing the site became a major goal for us. By August of 2005 the site handled roughly 2.7M page views from 515K unique visitors. This is not an exceptionally large amount of traffic, but for a site where the majority of the traffic is interactive forums it meant that we had to continually tune the site to keep the latency down. In 2001 Postnuke was the best CMS available but as we started having to tweak aspects of it to do get what we needed we essentially created a fork of Postnuke that was very challenging to maintain.
One of the main problems with Postnuke was that much of our data was stored in the embedded best-of-breed phpBB forum software. When we originally started using Postnuke we were using phpBB1.4, which came with the app by default. But we rapidly hit limits in the early version (and had to regularly apply security patches) so we took the hit, installed phpBB2 using the PNphpBB2 module and ported all the forum data forward using one-off custom migration scripts. But even though phpBB2 was a significant improvement over phpBB1.4, we still had the problem that the forums data was essentially in a separate application that only looked like it was part of the same site. There was no way to have a forum post appear on the front page of the site, which meant that any comments on news stories added would wind up in a separate system from forum posts. Search integration was weak. Transient session ids would appear and disappear in the urls as you moved from Postnuke to phpBB2. The mods we applied to phpBB2 were very difficult to maintain through an upgrade.
General inertia kept us using Postnuke for several years during which time it had a harder and harder time coping with the load. We forked off a separate database solely for forum searches to keep them from locking up the main database. We routinely applied security patches and fixed little issues as they cropped up. Eventually we decided that with the next generation of Gallery we should put out a whole new website so we started looking around for alternatives.
When it came time to choose a new CMS, we wanted to make sure that we chose one that scale to handle our needs, was under active development, had a large and active community and was easy to customize. We wanted to make sure that if we did have to make changes, they could be mostly localized into the theme code so that we could avoid tweaking the framework code and winding up with another forked CMS. It had to be flexible enough that we could add in our own apps like our paid support system, many of which are only visible to a subset of our users or the development team. We gathered input from as many sources as we can and eventually narrowed our choices down to two four solutions: Xaraya, Postnuke, Mambo and Drupal.
Xaraya looked promising, but they had not hit their 1.0 release yet and we didn't want to be stuck in a bind if they were not ready when we needed it. The latest version of Postnuke promised the easiest upgrade path but among other reasons, we wanted to get away from having embedded applications like phpBB2 for our main content. We wanted very tight integration.
This left Mambo and Drupal and it was tough to decide between them. They both seemed to offer similar capabilities and features. Both had active development communities. Both had good scalability stories. Neither had an embedded Gallery2 story at the time, but we were reasonably confident that one would emerge over the months that it would take us to port the site. Both had achieved a level of stability and had a reasonable security story. As with all modern CMS, both were modular and had flexible theming systems. Mambo seemed to have a better forums system, but Drupal's seemed capable of handling our needs and Drupal's node-based forums gave us greater flexibility. It was Drupal's adoption by a few high profile sites that demonstrated that it really could fulfill our needs. That coupled with the fact that we personally knew several Drupal developers who could help us with the migration made us choose Drupal in the end.
By far, the biggest challenge in porting was to move over all of our data. Because the Gallery website contains a great deal of information in the forums and has been around for a long time, there are many internal pages and external sites that link deep into the site. We wanted to be sure that we not only preserved all that data but to save all the links also. It would have been very easy to sacrifice these internal and external links or even start with a blank slate, but it would have meant that we'd be severing ties with the history of the project. There are still useful links out in the wild that point to our old phpBB1_4 installation so we wanted to preserve our history in one place, so we aimed to keep as much as we could.
We didn't have time to do this ourselves so we hired Kitt Hodsden on a contract basis to do the data migration for us. Her goal was to figure out how to move all of our data over to the new site in a way that would sacrifice as little as possible. At first glance this did not seem like such a difficult task but the details of the migration were extremely tricky and demanding. In order for us to have a useful beta cycle, we required that she automate the entire process so that at any given time we could run the script and it would blast the beta site and recreate a brand new one by importing all of the data from Postnuke again. Our data is diverse, our links are many, Postnuke and Drupal are dissimilar -- this was a complex task and it took Kitt many months of effort to get it to a point where it was 80% migrated. This also included writing several custom modules for us and migrating over the data for them as well. A secondary goal of this project was to fund a way for Kitt to create a migration path from Postnuke to Drupal that she could then leverage for her consulting firm. There were many, many niggling details to resolve but in the end we accomplished a push-button migration script that let us get a beta up to find and resolve all the remaining issues before going live.
We hit some snags
Partway through the process of porting to Drupal we started reorganizing our documentation. In order to make sure that we were using the right tools, we started experimenting with MediaWiki as our repository on a second server. This turned out to be a far better tool than the wiki implementation in Postnuke or Drupal and it did not take us long to collectively decide to migrate our data over there. This is when we discovered one of the downsides to Drupal -- it does not have a good MediaWiki integration story. By that point Kitt was halfway through porting to Drupal and we were committed. But this left us in the unfortunate bind of having two loosely linked websites without a clean integration.
During this process, James Walker got the Drupal/Gallery2 integration module far enough that we could use it almost out of the box on the new site. With a little work we added the features we needed (like having the random image block in the sidebar) and sent those changes back to James for inclusion in Drupal's CVS repository.
The Drupal forums module was not quite as good as we had hoped. Drupal follows a post-and-comments model which makes some of the nice forum moderation options like splitting certain posts off of a forum topic into a new topic very difficult. The default threaded comment presentation style did not mesh very nicely with our forum ideal. For a site that is heavily forum based, this was not promising. However, some investigation of the contributed Drupal modules turned up code that could be massaged into doing what we needed. It was not ideal, but provided enough functionality for us to get by.
Making it look good!
As Kitt started getting close to a point where she had a working data migration, we started focusing on creating a new theme for the site. We wanted a light and welcoming theme that would provide our users with intuitive site navigation. To make this happen, we hired Trae McCombs to design mockups and create a prototype of a Drupal theme for us. He created the overall look of the site around Mike Harding's winning submission in the 2003 Gallery Logo contest. Trae then translated that into a PHPTemplate based Drupal theme based on Lincoln's Revenge. With some design help from Chad Kieffer, we put the finishing touches on it to create a theme that gives us a warm fuzzy feeling when we look at it.
You may notice that the orange feature box on the front page looks very similar to the one found on the Drupal.org theme designed by Steven Wittens. That's because it's actually a placeholder for a new feature box. During the theme design process, Bharat copied the Drupal.org feature box as a proof of concept to demonstrate the kind of design that he'd like to see. It was never the plan to actually go live with it like this, but with the time pressure and the last minute confusion of releasing the new website and Gallery 2 on subsequent days, this item fell off the todo list and we went live with it the way it was.
The feature box is a very important part of the website and clearly we did not want to lose it. But since we very clearly violated Steven's copyright we needed to replace it with a more permanent solution. Luckily for us, he very generously offered to perform contract work for us to replace the current feature box with something new and unique for Gallery.
We achieved our goal.. Now what?
Our goal was to launch the new site before we release Gallery 2.0 so that we kick off the new product with a whole new brand. It took a tremendous amount of effort from several people but we're very proud that we've made it happen. There are still some outstanding issues we need to resolve, like figuring out how to integrate our MediaWiki data into Drupal in a sensible fashion and exploring other ways to improve our forums without forking Drupal.
Our journey with Drupal is just starting. Now that we've got a shiny new website we're going to take good care of it so that it will last us for many years to come! We hope that you're as happy with it as we are!