-When I first started at CC, Squid was in use on most of the servers. For some reason it was also necessary to have watchdog scripts run periodically to check for problems with Squid and to restart it if necessary. That seemed less than ideal, and I won't even get into the Squid configuration file, but suffice it to say that it can be frightening even for technical people. We still have watchdog scripts, but now they are more of a safety net than a necessity. When looking to replace Squid,
Varnish seemed like the logical choice, even though documentation was fairly lacking at the time. Thankfully, Varnish installs very easily and pretty much works out-of-the-box for a routine setup, so the dearth of documentation was acceptable.

After starting to use Varnish it became clear that we were hitting resource limits on the machines, all 32 bit at the time. We managed to upgrade one of the servers to 64 bit remotely via this method:
http://labs.creativecommons.org/2008/07/15/32-to-64bit-remotely/.
Once running on a 64 bit machine, Varnish was much happier, and so were we. We subsequently upgraded all of our servers to 64 bit, and we haven't had problems since.

There were a few bumps in the road that were annoyances, but not show-stoppers. Some of them are outlined in a post I made at the CC Labs blog: http://labs.creativecommons.org/2008/04/03/varnish-cache-at-cc/. Some of those may have been bugs in the version we were using at the time (v1.1.2), and some of them may have been related to misconfigurations on our part. Whatever the case, we are now happily running version 2.0.1 on all of our servers and have not witnessed any issues at all.

As others have noted, we were extremely pleased to see how well Varnish handles spikes in traffic. Without Varnish, our machines would be brought to their knees when theses spikes occur (we have tested this). So as far as I'm concerned, I don't have plans to provision another server without putting Varnish in front of Apache, even if the caching isn't important.

Creative Commons is a non-profit. We do not have unlimited resources to provision the latest, greatest and most powerful hardware. In this sense, Varnish has surely been a cost-saving measure, allowing us to serve quite a lot of data and connections on more moderately powered, economical hardware. I now consider Varnish to be an integral part of the Creative Commons setup.

Varnish's support of multiple backends has been of great use to us, and has taken the place of numerous mod_proxy and mod_rewrite rules. It cut out a few, now unnecessary, middlemen, and besides having simplified our configuration, I don't doubt that bypassing mod_proxy and mod_rewrite has also improved performance. One other interesting use of remote backends is as a very convenient way to seamlessly move a busy site to a new server without having to worry about DNS convergence and precise timing.

I thank the Varnish developers, and also Linpro for supporting the project.

Nathan Kinkade
Web Engineer
Creative Commons

Creative Commons is a nonprofit organization that works to increase the amount of creativity (cultural, educational, and scientific content) in “the commons” — the body of work that is available to the public for free and legal sharing, use, repurposing, and remixing.

Creative Commons provides free, easy-to-use legal tools that give everyone from individual creators to large companies and institutions a simple, standardized way to grant copyright permissions to their creative work. CC licenses let people easily change their copyright terms from the default of “all rights reserved” to “some rights reserved.”

Creative Commons licenses are not an alternative to copyright. They work alongside copyright, so you can modify your copyright terms to best suit your needs. We’ve collaborated with intellectual property experts all around the world to ensure that our licenses work globally.