Study shows open-source code more bug-free than proprietary
February 24, 2012 —
Open-source software and proprietary software contain similar numbers of bugs per thousand lines of code. Software test tool company Coverity released its annual Coverity Scan: Open Source Integrity Report, in which it compares the codebases of open-source projects against those proprietary projects.
The results of the scan showed that in 37 million lines of open-source code across 45 projects, there was an average of .45 bugs per thousand lines of code. Proprietary code, which was sampled anonymously from Coverity's customer base, included over 300 million lines of code across 41 projects, where it was found that there were .64 bugs per thousand lines. Coverity claimed the average for all software is 1 bug per thousand lines of code.
It should be noted that there was a significantly larger set of code to be had in the proprietary sample. There, the average project size was 7.5 million lines of code. In Coverity's open-source sample, the average project size was only 832,000 lines.
In the open-source scan, the most common bugs were bugs that raise potential security issues, such as null-pointer dereferences and uninitialized variables. More overt issues, such as buffer overflows and insecure data handling were less common.
Coverity's report also dissected the Linux kernel, PHP and PostgreSQL. Each one had exemplary ratios, especially when it came to high-risk bugs. But PHP and PostgreSQL both had the best ratios in the Scan report: PHP 5.3 with .20, and PostgreSQL 9.1 with .21 bugs per thousand lines.
Zack Samocha, the Coverity Scan Project’s director, said that larger projects are at a bit of a disadvantage if you're judging them entirely based on their defects per thousand lines of code. “The reason you have higher [defect] density in larger projects is that usually those are older, bigger applications that have been out in the market for a while. They just have more defects,” he said.
“The other thing is the amount of people involved in that project is higher, and usually when we have so many people working on the large project, people are a bit more careful about barging in and fixing them.”
Samocha said that while the anonymous horde of proprietary software scanned for the 2011 report was larger in terms of lines of code, the actual projects were quite similar. He said there were “examples of browsers, operating systems, the top of the line you'd expect for the proprietary code, and we compared those to the most active open-source projects.”
For next year, Samocha said he hopes to have Coverity's cloud platform integrated with the popular software repositories GitHub and SourceForge. He said this is already in the works at Coverity, and that it will open up the opportunity for other open-source projects to use Coverity's defect-scanning tools on their projects with a few mouse clicks.
Related Search Term(s): bugs, Coverity, open source