Slide deck on the security aspects of using Open Source Software. Focused on the Apache HTTP Server project, this deck discusses general topics like what Open Source software is, what the prevailing myths surrounding it are and how the open development process works to ensure the result is secure.
1. Security and Open Source Software
Sander
Temmesander@temme.
net
@keysinthecloud
2. Your Presenter
Member, Apache Software Foundation
Contributor, Apache HTTP Server
Sales Engineer & Consultant
Open Source Integration Expert
3. Agenda
Open Source Software
Security Process
Security Implications
Development Model
4. Three Questions
How does open source respond when security
problems occur?
How does the open source development
process affect software quality?
Is open source software more susceptible to
security problems?
12. Open Source Security Myths
Given enough eyeballs, all bugs are shallow
Open Source is Communist!
13. Open Source Security Myths
Given enough eyeballs, all bugs are shallow
Open Source is Communist!
Bad guys have the code, too!
14. Open Source Security Myths
Given enough eyeballs, all bugs are shallow
Open Source is Communist!
Bad guys have the code, too!
Open Source is more secure than Closed Source
15. Attack Goals
2% Defacement/Planting Malware
6%
4% Information Leakage/Stealing
4% 28% Sensitive Data
Disinformation
11% Monetary Loss
Downtime
Link Spam
19% Phishing
26%
Other
Source: The Web Hacking Incidents Database, 2009 Report
19. The httpdProject
#1 Web Server
Non-profit Foundation
Contributors
Oracle, IBM, Novell, VMWare, Red Hat, Google
Many individual contributors
http://httpd.apache.org
Many packagers and distributors
http://people.apache.org/~coar/mlists.html
20. Apache Security
Very few vulnerabilities reported
No critical vulnerabilities in 2.2.x
Upgrade to any new release
announce-subscribe@httpd.apache.org
Default installation locked down
But it doesn’t do a whole lot
http://httpd.apache.org/security/vulnerabilities-oval.xml
http://www.apache.org/security/
21. Apache Security Process
Report security problems to
security@apache.org
Real vulnerabilities are assigned CVE number
Vulnerabilities are classified, fixed
New httpd version released
http://httpd.apache.org/security_report.html
http://cve.mitre.org/
http://httpd.apache.org/security/impact_levels.html
announce@apache.org
http://www.apache.org/security/committers.html
26. Database Privileges
Bugzilla: GRANT SELECT, INSERT, UPDATE, DELETE, INDEX, ALTER, CREATE, LOCK
TABLES, CREATE TEMPORARY TABLES, DROP, REFERENCES ON bugs.* TO bugs@localhost
IDENTIFIED BY '$db_pass';
Wordpress: GRANT ALL PRIVILEGES ON databasename.* TO
"wordpressusername"@"hostname” IDENTIFIED BY "password";
Joomla 1.5: GRANT ALL PRIVILEGES ON Joomla.* TO
nobody@localhost IDENTIFIED BY 'password';
Drupal:
SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX,
ALTER, CREATE TEMPORARY TABLES, LOCK TABLES
Gallery 2:mysql gallery2 -uroot -e"GRANT ALL
ON gallery2.* TO username@localhost
IDENTIFIED BY 'password'”;
27. Getting it Right: Bugzilla
GRANT
SELECT, INSERT, UPDATE, DELETE, INDEX, ALTER, CR
EATE, LOCK TABLES, CREATE TEMPORARY
TABLES, DROP, REFERENCES ON bugs.* TO
bugs@localhost IDENTIFIED BY '$db_pass';
Install script
Creates database
Executed as root
Application privileges
Limited
Only as needed
This is not always practical
30. Support
Often community based
You can be part of it
Visible to the world
Don’t post confidential information!
Support contracts available
From third party companies
users@httpd.apache.org
33. Mailing Lists
All communication by e-mail
Several lists
announce@<project>.apache.org
users@<project>.apache.org
dev@<project>.apache.org
cvs@<project>.apache.org
34. Code Changes: Transparency
Source history available
Every modification posted
Instant code review
Etiquette
35. Bus Factor
Development Community
Project Survival
Closed Source Equivalent
Vendor out of business
Product end-of-life
36. Tips for Open Source Users
Get on announce mailinglist
Investigate community
Get involved
37. Conclusion
Open Source responds proactively to security
issues
Open Development encourages clean and
secure code
Security Issues are universal and not specific to
Open or Closed Source Software
I have been involved with the Apache community since 2000.I mainly contribute to the project most people associate with the name Apache: the Apache HTTP Server. I also help out on the Infrastructure committee, which runs the apache.org sites.In my professional life, I am a Sales Engineer for Thales E-Security, a very small division of a very large company. We sell cryptographic hardware, and in practice I am the go-to guy for open source software integrations with our products. I am going to talk mainly from my experience with the Apache development community
So, what will we talk about today? First I will discuss Open Source software, its various forms and incarnations and where we typically find it. Then we will talk about the security process followed by open source communities. We will discuss some security implications that come with using software, and relate those to the open source approach, and finally talk about the open source development model and practices. Meanwhile, we will be looking for the answers to the following three questions about Security:
Read out: (Build) How does the Open Source Development Community respond when security problems occur? (Build) How does the Open Source development process affect software quality? And (Build) Is Open Source Software more susceptible to security problems than closed source software?First, a show of hands please:Whoin the audience runs software deployments? Who knowingly uses Open Source?
Here is a quick sketch of the landscape. First, the familiar Closed Source software: you pay money, you buy a license. The license typically restricts the rights you have with regards to the number of systems on which you can install and use the software, and whether you can give copies to someone else. Failure to obey these restrictions may land you in jail. (Build) Some very familiar names in this category, and thousands more. Second, Open Source Software is developed in an open process and published as source code, or with the source code readily available. Many open source projects make their software available at no charge, though that is not its defining characteristic. The license typically grants significant liberties regarding redistribution. The larger open source projects are backed by independent, non-profit foundations like (Build) the Apache Software Foundation, Debian Linux, the FreeBSD operating system, Mozilla (makers of Firefox), the Python programming language and the Free Software Foundation who oversee, among others, the Gnu Compiler Collection (gcc) and the Emacs text editor.A relatively new trend is for companies to publish their software under an open source license, and generate revenue from services and support. (Build) Sometimes the open source software is more readily available than others. Some treat the open source version as trialware, and have significant proprietary improvements for the “Enterprise” version they sell. Finally, a lot of companies include open source software in their own offerings. (Build) BEA WebLogic contains Struts, an open source tag library from the Apache Software Foundation. IBM HTTP Server is Apache. Apple uses the open source gcc compiler to build all their code, and includes numerous open source components on both their regular and Server operating system. Autodesk’s Maya uses the Open Source Python programming language Cisco’s Webex suggests use of the open source web scripting language PHP in its URLs, and NetApp’s operating system is based on FreeBSD. You may already be using Open Source without even knowing it!
There is a number of things Open Source is decidedly not. Just because it’s free, doesn’t mean it’s Open Source. Open Source software also never comes with any bait-and-switch that requires payment upon actual use. What you download is what you get. Sometimes companies that have a product they no longer want to maintain, and decide to release it as open source. With a grand publicity gesture, even. However, just having the source out there doesn’t mean that someone is going to jump in and maintain that. Companies who make such a move frequently don’t succeed in attracting a development community, and we call such efforts “abandonware”. Finally, Open Source Software is not in the Public Domain. Copyright remains with the developer, and there is a license agreement that covers what you can, and cannot, do. Public Domain entities are not under Copyright.
So who are these people who develop software to give it away for free?Frequently, they are users who run into bugs or missing features. They can fix the bugs or add the features, and contribute the result back.Consultants, make a living implementing open source for their customersVendors who incorporate open source software in their closed source offeringsFinally, hobbyists who develop code for personal edification. Anecdote: trans-Atlantic Pilot.
First, Open Source development is a great resume builder. Having contributed to Open Source projects shows that you can work with a team, and gives you beautiful, sparkling clean code samples to which you can point without getting in trouble with your former employer.Many contributors came to Open Source development because, as I mentioned before, they started as users and found ways to contribute.Or, they worked with Open Source Software as part of their employment.
While some of us may use desktop open source offerings like Firefox or OpenOffice, and we may run Linux on our desktops and laptops, the vast majority of open source software deployments is found in the server room. We use open source operating systems like Linux and FreeBSD, and open source application environments like Apache, Tomcat, PHP or Ruby on Rails. This means that many of these deployments are in the line of fire, accepting connections from the Internet and hence are under direct attack from worms, bots and malware. So let’s look at a pie chart of attack methods.
So, let’s talk about some myths surrounding open source software, and look for a balanced view. First, in the mid-nineties a gentleman named Eric Raymond wrote a paper titled “The Cathedral and the Bazaar”. In it, he compared the open source development model against traditional, closed source development and one of the statements he made was (Build) “given enough eyeballs, all bugs are shallow.” This is a very interesting statement, and stands at the base of the reasons why Open Source works. However, it’s easy to misinterpret. Does it mean that you have to be a programmer in order to use Open Source, and that you have to be able to fix bugs? Not necessarily, but there is nothing standing in your way if you are, and want to contribute. Does it mean that there is an army of altruistic programmers waiting in the wings to fix my problems for me? That’s not the way it works. What Eric meant to say was that given a large and diverse enough development and user community, it is likely that someone else will encounter your problem and fix it to serve their own purposes. And they are likely to contribute the fix back to the open source project for several reasons: to pay it forward; to get recognition; or simply because the open source codebase is the best place for that fix to live. This is not always the way it works, but surprisingly often, it is.
Next one: “Open Source is Communist!” OK, a statement used for effect. As discussed above, people use open source software to serve their own purposes, which may coincide with yours. Of course some people contribute for altruistic reasons, and there is nothing wrong with that. However, most of the time contributing back is a business decision. For instance, a decade ago IBM realized that they make money running people’s IT infrastructure, and not by selling web servers. So instead of developing and maintaining their own web server implementation, they decided to have their developers work on Apache, and contribute the results back to the newly incorporated Apache Software Foundation. They were actually instrumental in starting the foundation. Of course there is a faction in the open source movement that proclaims that All Software Must Be Free, and that making money selling software is somehow uncouth. One wonders how they eat, and how they pay for their tie-dyed shirts and sandals, and the answer is usually that someone, somehow supports their activities.
OK, now we get into the Security realm. If the bad guys have access to the source code, won’t it be easier for them to find flaws and exploit them? I have three arguments against this one: I think the past decade or so has shown us that the bad guys don’t need the source code in order to find and export flaws… has anyone here heard of Patch Tuesday? Given that there are more white hats than black hats in the world, we should make it easier, not harder to find flaws. That way, it becomes more likely that flaws are found by the good guys and fixedThe fact that you have the source code available enables you to quickly incorporate critical patches as they appear, and won’t even have to wait for a release to come out. A vibrant and diverse user and development community should also be able to turn around fixes pretty quickly. We’ll talk about what that looks like in a little bit. And finally:
Let’s talk about the notion that one technology is more secure than another. You sometimes hear this. “Oh, you should be using Linux, it’s more secure!” “Oh, you’re on Windows, good luck with that.” Unfortunately, purely by itself no technology is a silver bullet for security. Flaws are found and fixed in both open and closed source software. Whatever technology you are using, you need to manage your deployments. Microsoft has made great progress in systems security, but they are faced with an enormous legacy. So far, no technology has appeared that you can just drop into your data center and forget about.
OK, let’s get into the bad guys’ heads. This pretty pie chart is taken from the 2009 report of the Web Hacking Incident Database. It suggests that close to 75% of attacks are done to steal data, to deface the website or to plant malware on it. Information Theft is particularly attractive to the malfeasants, because the results can be used for financial gain. Data can be used or sold for profit, where defacements only bring bragging rights.
So how did the miscreants get in? This pie chart again comes from the Web Hacking Incidents Database 2009 Report. The problem with this report is, that they only found 44 incidents with enough data to include in the report. In Third place, Content Spoofing. For instance, a malicious JavaScript is snuck into a web forum or comments page and wreaks havoc on the browser of the next user to load that page.In Second place, we have Insufficient Authentication: sensitive resources like an Administrative Interface to the app were not protected by authentication.In First Place–drum roll–retaining the top position since the prior edition of the report, we have Good Old SQL Injection. Let’s see what that looks like with a practical example:
Read out last two frames, with fingerphone.
Next, we’ll do a Case Study on how an Open Source development project may cope with security vulnerabilities. We’ll talk about the Apache HTTP Server project.
Just to put the project and its community in perspective:The Apache HTTP Server has the largest percentage of servers on the Internet. Justshort of 80 people have commit rights. Between 15 and 20 people contributing at this time. However, 865 subscribe to the Developer mailing list. However, over 2,000 are on the User mailing list, and the Announcementlist has 16385 subscribers. 54 people sit on the Project Management Committee, which approves releases and recruits new committers.Every contributor regarded as individual. A self-employed consultant has the same voice as a VP of the largest software firm in the world.
By itself, Apache has a very good Security track record. No critical vulnerabilities have to date been reported against the latest shipping version of the server, and by “critical” we mean vulnerabilities that would allow an attacker to take control of the server host through the Apache server program. Of course I recommend that you stay current, and upgrade when new Apache releases are announced. On the slide is a subscription e-mail address for the announcement mailinglist, which only carries announcements of new Apache HTTP Server releases and is very low traffic.A default installation of Apache is locked down tightly, but it doesn’t do a whole lot.In the rare occasion when a security issue is found, there is a Process in place to fix it.
Say, someone finds a vulnerability in the Apache web server. They are encouraged to let the developers know first, even before tweeting about it. There is a foundation-wide mail address that is monitored by volunteers, and dispatches security reports to the affected projects. Real vulnerabilities are asigned an entry in the Common Vulnerabilities and Exposures database, or CVE, maintained by defense contractor Mitre corp. As a report is received by the development team, its impact is analyzed and an impact level assigned. The higher the impact level, the quicker it will be fixed: a low level vulnerability that does not appear in the default configuration of the server might get pushed off to the next regular release. However, for higher impact vulnerabilities the team can turn around a release within days.
As I said, a default installation of Apache is very secure. However, most websites today run a lot more software than just Apache. Here is a picture of what the entire software stack might look like:
The top layer, the Application, is what makes your web application unique. The software on the lower layers on the stack is used by many other sites, and, if kept up-to-date, likely to be in good shape security wise. The Application tier is likely to contain custom code, unique to your application, exercised in no other places and for no other purpose. This is where the vulnerabilities occur.
Transition Slide!
You’re right, there are still no notes here.
For instance, these are literal instructions taken from the installation manual of a number of high-profile open source applications. For instance, theJoomla content management system installation manual instructs you to give its user all possible privileges on the database that backs the system. This means that the web app could drop the database tables if an attacker were to make it try to do so. Let’s look at an open source application that got this right.
The Bugzilla issue tracking system comes with an install script that you run, as user root, when you set up the system. It adjusts the file system privileges on every file in the application, and sets up the database. The user as which the application actually connects to the database only has the privileges it needs.
Another issue at hand is Provenance. If you purchase a box of software, you can reasonably assume the store sells you the real thing. But what if you download a program from the Internet? And how can you make sure that the developers were actually authorized to make that code available? What if they stole it from their employer? Or their employer’s customer? Can you be sure that the software doesn’t infringe someone’s patents?At Apache, we have several defense mechanisms against this. All Apache releases are digitally signed by the person who made the release. This assures the integrity of the release archive, and through the PGP grass-roots web of trust, the identity of the signer. Secondly, every individual who contributes code to any Apache project signs an Individual Committer License Agreement, a one page contract that assures the contributor only submits code that his theirs to contribute. In turn, the Apache Software Foundation takes responsibility for the submitted code. Corporations can sign a similar contract to cover their submissions. These contracts also contain a clause in which the contributor grants a license to the Foundation and to any users of the contributed code for any patents held by the contributor that might cover that code.
This has builds.What recourse do you have when the software fails? Open Source licenses contain a no-warranty clause: the software is delivered as-is, and as a friend of mine likes to say, if it breaks you get to keep the pieces. However, it’s not much better on the closed source side. Closed source end-user license agreements typically also contain a clause that renders the vendor not liable should the software fail.
Talk about support mailinglist, ask good questions and get good responses. Support contracts with Service Level Agreements etc. available from vendors, sometimes more evident than others
Transition Slide!
Over-arching theme is Transparency
What if one of your development team members gets hit by a bus?If there is a viable development community, security holes get fixed. This is an advantage of open source over closed source
Things to consider if you are considering open source software: announce: low volume, essential announcementscommunity: viability, tumbleweeds, if no community, can you fix it yourself or are you out of depthGet involved with the support community, be a good citizen; don’t demand answers; if no one responds that likely means no one has found or fixed this issue yet. If someone asks a question for which you know the answer, do speak up.