First published: 04th September 2009
The Apache Software Foundation has released a report that describes how attackers broke into various systems, ending with the Foundation downing most of their production servers on August 28th while recovery took place.
The attackers gained access to a server owned by the ApacheCon conference production company and use the SSH key of the backup account to gain access to a staging server and introduce a CGI script to production webservers that was used to obtain remote shells. Fortunately, largely because of defence-in-depth strategies, Apache Software Foundation code repositories, downloads, and users were not put at risk by the intrusion.
The report is instructive in that it provides a detailed picture of how an attacker can eploit weaknesses, and also for Apache's unflinching analysis of their strengths and weaknesses. High points were the use of ZFS snapshots that allowed speedy restore of a known-good state, redundant services in two locations and diversity in the server operating systems that made it difficult for the attackers to escalate privileges on multiple machines. Low points were mis-management of SSH keys, an rsync setup that allowed undetected introduction of files to production servers, CGI scripting enabled on hosts where it was not needed, and keeping log files on the initially-compromised server that allowed the attackers to destroy the evidence of their entry method.
Apache's openness and transparency extend to more than their source code.