Set it and Forget it – The Unfortunate Common Administrative Approach and Its Impact on Security
Introduction
Take a moment to think of what prompts you to run an apt-get upgrade (or similar command for your operating system of choice) on systems you are responsible for. A betting administrator would, without hesitation, throw money down on the fact that systems administrators rarely, if ever, perform upgrades unless prompted to by a critical CVE announcement. Or, worse yet, as the victim of an all too late reaction to a successfully exploited SQL Injection or other web vulnerability.
Indeed, such is the case for an overwhelming majority of public-facing systems: administrators set things up once, and neglect to maintain them thereafter. Although, nowadays it is perhaps a little unfair to label these offenders as “administrators.” The line between an admin, a DevOps engineer, full-stack, scrum, cloud, et cetera ad nauseum, is becoming increasingly blurred.
Even the professional responsibilities sometimes no longer cross boundaries – an administrator may be responsible only for provisioning the base server itself, while a development team is in charge of the actual web applications running on the server (not uncommon today in agile development models designed such that admins will “step out of the way” of developers). But the ownership of blame for lack of updates is of little importance. What matters is the lack of updates themselves, as this has been a problem since the invention of the ability to update software some decades ago.
Even Though Old Software and Web Applications Might Be Vulnerable, They Are Still Widely Used
The Open Web Application Security Project (OWASP) – a non-profit organization founded in 2001 to act as a central educational and informational organization on web application security and related vulnerabilities – has kept a semi-annual list of what it considers the top ten most common and severe web application attack vectors. Consistently near the top of that list almost every year are what OWASP categorizes as Security Misconfigurations, including out of date software.
Indeed, if the list were focused on all internet-connected devices and not just web applications, there is little to no doubt that out of date software would rank as a root cause for most all successful attacks. Out of date software are perhaps not as often exploited on web application systems as SQL injections or cross-site scripting, per OWASP’s observations, but there is indeed an incredible amount of them left unpatched, perhaps since the day they were first setup and are heavily at risk.
Several hundred million websites running today are being served on critically insecure servers. At least 175 million websites are currently served from Windows Server 2003 – over a decade old and as of July 2015 completely devoid of all support including security patches. For months after HEARTBLEED was announced – perhaps the most infamous security vulnerability in the past decade – hundreds of thousands of websites remained unpatched. The Kaminsky DNS vulnerability has taken the better part of a decade to fix globally.
Why Administrators Still Run Old Code, Unpatched Systems and Web Applications?
Usually this gets chalked up to laziness and forgetfulness – putting it off until either something terrible happens or something breaks. And that is undoubtedly the reason for many systems left unpatched. However, often the reasons are far more technical and less lackadaisical.
By Updating the Software You Might Break It
Microsoft has so often introduced Windows Update “recommended” patches that can break various things, that most major companies run their own WSUS upgrade repository to test patches before letting them be installed on production systems – often resulting in critical updates not being applied for weeks or months after their release. (Of course, we would be remiss to not mention those who are jaded by potential threat ofunwanted forced changes and thus choose to manually update… eventually.)
Start Afresh with Every Version
On the other side of the world: As part of its core principles on release cycles and upgrading, the Linux Mint distribution, for example, strongly emphasizes a “fresh start” approach to major release upgrades – an approach that, while clean, deters most admins and general users alike due to the time and labor cost.
Trust Automated Software Updates Won’t Break Your Server
This is in direct contradiction with Ubuntu’s policy, which suggests putting faith in the automated background upgrade systems to not break anything. But this approach has in fact been known to break things quite often. Packages introduce breaking changes all the time in minor semantic versions that may make it through poorly implemented apt-get upgrade cron jobs, either by intent, or, more often, by mistake that makes it past tests. Automated -y “yes to all” flags or postinst repair scripts do not always generate the best outcome. And admins know this, sometimes influencing their choice to forego automated upgrades in favor of stability and the watchful audit of manual updates… eventually.
Running Outdated Software is Wrong. Then What’s Right?
Obviously, update the thing, yes, but how? How do I apt-get upgrade seems self-explanatory, but there are many variables in play here, including not least of which the fact that those tasked with the duty of managing servers have many of them to tend to, numbering perhaps in the dozens. And up til now, we have mentioned only one of many facets to a web-facing application.
What about databases? Or the web application itself, maybe a web forum, a blogging software or the custom-built business specific web application? If the constraints change on these, how do I keep things updated and synchronized like firewall rules or authentication secrets?
Automation Can Help You Solve Many Security Issues
One of the best solutions for this problem is automation – specifically, automated configuration management utilities like Puppet, Chef, and SaltStack (the latter being well suited toward larger and more complex deployments). With SaltStack, for example, there exist configuration template collections known asformulas covering many common web services like Nginx, MySQL, and even web applications themselves such as WordPress and Drupal.
Automation utilities like these grant you the ability to synchronize your systems and deploy configurations and updates far more efficiently than by hand. Additionally, they allow you spin up duplicate environments for testing with little more effort than copying and pasting a few lines of YAML and Jinja. But wait, there’s more!
Keeping your servers synchronized and updated is great, but not enough. If an attacker manages to exploit a 0-day vulnerability, or exploits a known vulnerability that does not have a patch released yet no amount of updating will help you.
[adsense size='1']
The Importance of Monitoring and Logging
Instead, here is where you need to begin implementing monitoring and logging solutions. There is a whole myriad of utilities one can use – OSSEC, as one example, provides an incredibly powerful array of utilities that perform many actions at one: host intrusion detection, adaptive firewalling, alerting, log management, and more.
To keep with the focus of free enterprise-grade utilities, however, there are some better solutions that, while seemingly very complex, provide far more functionality and options than others. First, on the monitoring side, we have Check_MK – an extremely robust and scalable distributed monitoring solution built around the Open Monitoring Distribution management utility and other existing technologies like Nagios and PNP4Nagios. The beauty of applications like Check_MK is that you can deploy a simple check script on almost any type of operating system, and it automatically inventories everything for you and even assembles most reporting and graphs out of the box. While this by default mostly focuses on reliability of services, this is still just as important as the security aspect. More on this in a moment.
Next, on the logging side, we have all kinds of solutions available. The aforementioned OSSEC can handle this (and does so remarkably well), but there is something better. There is a common combination of services known as an ELK Stack (‘ELK’ being an abbreviation for Elasticsearch, LogStash, and Kibana respectively) that can handle hundreds of millions of log lines with ease. But couple this with services like the Suricata Intrusion Detection/Prevention System and the Scirius rule manager for Suricata, and you have a very powerful bundle known as SELKS (which can not only chug logs using LogStash, but can also monitor network and device security with the other combined components). This bridges the gap between logging and monitoring and adds adaptive security to the mix, too. What could be better?
So, Is That All I Need for Secure Web Servers and Web Applications?
Of course not. The web application itself – the system generating your web pages, such as WordPress, vBulletin, a Node.js app, a Ruby on Rails project, a C web application running behind GWAN, or whatever else – is often targeted more than the services that run it. Although it is crucial that the services in the background are up-to-date and hardened, the web application is the most public-facing element and requires as much, if not more attention. After all, it is not MySQL itself that is to blame for SQL Injections consistently being the number one stop in OWASP’s Top Ten list; that fault lies with insecure web applications.
[adsense size='1']
Automating Web Application Updates
Web application updates can be something also automated using a configuration manager. SaltStack provides excellent functionality when it comes to managing installations, such as either working with package managers (like apt or yum), installing packages by hand (gdebi is really handy for installing .deb packages manually, by the way), or by simply placing and maintaining the integrity of files in a certain location (such as the contents of a .tar.gz). But even this is not entirely necessary sometimes. WordPress, for example, can handle its own automatic updates.
However, this is of course not so true for many applications. A good administrator should keep updated on news about the web applications they support, such as posted vulnerabilities (sometimes designated as CVEs) or new releases. This sort of news alert may not always be something that can be automated (like via mailing lists or an RSS feed aggregator), which is likely why it so often goes ignored. Running a public-facing web application takes some responsibility, and some of that indeed includes proactively staying on top of new announcements.
That’s a Lot of Things!
Indeed, it is. Which is probably why these sort of deployments are seemingly rare and absent from all but the most security-conscious organizations. There is no denying that the three main tools mentioned above – SaltStack, Check_MK, and SELKS alone – would require perhaps at least two to four weeks of solid practice and experimentation to really begin to understand their true potential and configuration abilities. But nothing good worth having comes easy. A strong security posture requires a constant battle which takes time, patience, and much work – a likely reason why the most minimal effort seems to be the most common applied to web-facing services.
There are many technical guides on how to set some of these things up that are simple and easy to follow (assuming, of course, you already have a basic understanding of setting up Linux servers). This guide, for example, walks you through how to get a SELKS stack installed on a Debian 8 “Jessie” 64-bit server. So roll up your sleeves and start digging into automation, monitoring, and logging utilities, and secure your web applications! Do it before it is too late… now, not eventually.
Gloss