GrammaTech adds real world benchmarks to SWAMP

Software development and quality managers that are looking to measure the benefit of static analysis now have a platform to do just that. GrammaTech, under contract for the Department of Homeland Security (DHS) Science and Technology Directorate (S&T), has created independent real-world benchmarks that are now available in the Software Assurance Marketplace (SWAMP).

Several different synthetic benchmarks exist that can be used to measure how well static analysis tools perform in detecting bugs. However, many of these have limitations with the code paths typically being too simple. GrammaTech created BugInjector, a tool that can inject Common Weakness Enumeration (CWE) based bug patterns into existing code bases, thus delivering real-world benchmarks.

The BugInjector tool is available directly from GrammaTech to inject bugs into private code bases for training purposes. Additionally, four different real-world code bases (nginx, grep, sqlite, lighttpd) have been injected with bugs and are available through the SWAMP, enabling users to easily benchmark how well their static analysis tools are able to find these bugs in realistic code paths.

“There is an urgent need for benchmarks, such as those from GrammaTech, to allow software developers to evaluate static analysis tools in a comprehensive and real-world setting,” says Barton Miller, Professor of Computer Sciences at the University of Wisconsin – Madison and Chief Scientist of SWAMP. “Also, developers of static analysis tools now have the ability to enhance their tools or benchmark new static analysis technologies with realistic test cases. Integrating these benchmarks into the SWAMP platform increases their effectiveness and availability.”

“GrammaTech CodeSonar® has always focused on highest recall,” says Paul Anderson, VP of Engineering at GrammaTech, Inc. “Many tools claim that they can catch a particular CWE, but there has never been a way to measure how well tools perform if this CWE is hidden deep inside a code path. BugInjector provides an automated way to objectively measure static analysis tool recall; interested parties can now evaluate CodeSonar®’s market leading recall against other tools easily.”

The SWAMP’s static analysis capabilities are available for use in the cloud or on-premise at no cost. Interested parties can sign up to use the SWAMP and find the BugInjector test cases on the Resources page under Packages. After selecting a package and version containing a CWE of interest, users can run an assessment of the chosen “bug injected” software using one or more software assurance tools. GrammaTech CodeSonar® is one of the commercial tools that is integrated into the SWAMP, along with many other open source static analysis tools. Users can also download BugInjector test cases to run against tools they are developing.

About the SWAMP

The Software Assurance Marketplace (SWAMP) is a joint effort of four research institutions – The Morgridge Institute for Research, Indiana University, the University of Illinois at Urbana-Champaign, and the University of Wisconsin–Madison – to advance the capabilities and to increase the adoption of software assurance technologies through open continuous assurance capabilities and a shared facility. The SWAMP project is funded by the Department of Homeland Security Science & Technology Directorate. Services include access to high throughput computing capacity, over 30 software assurance tools, and a library of more than 500 open-source code samples with known vulnerabilities to help developers improve the quality of their static and dynamic testing tools.