the value of static analysis

I attended a practitioner’s report yesterday that described experiences in integrating a number of static analysis tools into eBay’s development process. In essence, it was a short presentation of the paper, “Understanding the Value of Program Analysis Tools.” The valuation methodology was simple: did the cost of the tools find enough defects to at least be comparable in cost to an equivalent team of manual testers?

One thing that’s important to note is that comparing the cost per defect found by static analysis against the cost per defect found by a manual tester is actually comparing different things. This comparison seems like a reasonable starting point, however, if we assume that a defect found by a developer using static analysis would have later been found by manual testers.

The presenter mentioned a few challenges in using static analysis tools: they require customization per domain and tracking their effectiveness is actually a bit tricky. Customization per domain is necessary because different types of defects may be high or low priority depending on the domain that you’re working in. True positives (defects identified by static analysis tools that are actually defects) are difficult to track because once a developer fixes the defect, it can’t be related to the tool unless it’s reported to some kind of centralized database.

I asked the presenter how difficult it was to integrate these tools into the development process of a large number of teams. I was interested in this because it would have been impossibly difficult to get so many teams to buy-in to use static analysis in the environments I’ve worked in. She said that the developers were quick to buy-in, management was convinced once they saw a few positive numbers, but the testers were a little reluctant about it. I can completely understand this: the valuation methodology makes it seem that static analysis tools could replace functional testing.

While I’m on the subject of static analysis, I’ll talk about a demo that I attended yesterday on automatically generating test cases based on constraint solving and counter example. A guy demoed Agitar Software’s TheoryExplorer and showed how it has been used effectively to find issues with J2SE and Jakarta Commons Collections. While this isn’t really anything new, it’s reasonably well integrated with Eclipse and seems relatively easy to use. He was also using the new @Theory and @DataPoint annotations in JUnit 4.4, which I wasn’t aware of but I’ll start using right away.