The organization
The Roskilde Festival in Roskilde, Denmark, is the second largest music event in Europe and entertains more than 100.000 guests over 4 days (+4 day warm-up) annually. The festival is powered by approximately 25.000 volunteers that work around or during the festival. 40 volunteers work around the year to develop and operate 10+ specifically designed web-based information management systems used by the festival.

The challenges of volunteer developers
As with many open-source projects, the group of volunteer developers is very dynamic, since people can leave or join rather spontaneously. Therefore, a key element is to provide new developers a simple entry into the development process, and ensure that systems stay stable even if developers drop out.
Contrary to many open-source projects, developers are not end-users, so there is no obvious itch to scratch [1] until new developers have experimented with the systems and technologies. Also, once new developers have made contributions to the systems, they need to see it used by end-users in production soon after to not loose interest in the project.

Quality assurance by continuous integration
To provide sandboxes for experimenting and short ‘time-to-production’, we have built a versatile quality assurance framework to allow new developers to dive right into changing code and try out new things, without the risk of breaking any of the running systems. Through a series of verification steps, both manual and automated, all system contributions are continuously integrated into the production mainline in a lean and reliable way.

The development process
Our setup builds around the philosophy that “developers need flexibility, operators need stability”.
To achieve this we use a chain of environments. As contributions flow through these environments – from the developers hand into production mainline – the developers flexibility becomes limited as focus is turned to stability of a release. The chain consists of the following 4 environments that are focused around per-system subversion repositories.

  • Development environments: VirtualBox images with replicates of the production environment that can be used locally by developers to try out new things. The developers commit contributions from the images into subversion projects.
  • Preview environment (alpha-test): Central servers with exports of the subversion projects that allow developers and change managers to check out subversion revisions to preview individual changes and run unittests
  • QA environment (beta-test): Central servers with exports of the subversion projects that allow beta-testers to verify releases through integration tests and acceptance tests, and and release managers to test release procedures. QA environments are separated from preview environments because they are also used for end-user education, which requires them to be more stable and less frequently updated.
  • Production environment: Central servers that host production versions of the systems, which are used by end-users.

The test framework
The test part of the continuous integration framework consists of the following tools:

Hudson automatically triggers basic testing in the preview environment on every commit to Subversion. Commits are considered atomic and only include one change each, which is validated manually – also in the preview environment. Atomicity enables easy rollback of changes and preserves traceability.
Integration tests and end-user acceptance tests are recorded using Selenium IDE and collected in Bromine. Before the release of a new version, Hudson triggers Bromine to execute the entire test-suite of Selenium tests from the previous release as a regression test on the new system deployed in a QA environment. The tests are executed either on dedicated internal Selenium clients, or in the SauceLabs cloud via Bromine’s integration with Sauce OnDemand. Bromine produces result reports which provide an overview of the success rate of the tests. The result reports are automatically integrated into Hudson, which makes the test result overview fast and clean. Futhermore, end-user testers make exploratory testing in the QA environment to validate the released bug-fixes and features.
When the new release is approved, the Bromine test suite is updated to include new Selenium tests for the new bug fixes and features.

After release, the wheel turns once again with planning of a new release, where everyone – new as old – can participate, with small or big changes, thanks to the integrated quality assurance framework.

About the author:
Anders Nickelsen(twitter.com/anickelsen) is the leader of the QA-team in the IT-development group at Roskilde Festival.

[1] The Cathedral and the Bazaar, Eric S. Raymond, O’Reilly Media, ISBN: 1-56592-724-9