I wrote the other day about some Testing Cramps. One of the comments in regards to the post was why didn’t I just write the tests and contribute them back? The thing is I have done so for some projects, but I also try to write my own tests as well for the code I’m a commiter on, and encourage others to do so when they are adding or fixing code in their projects as well. Usually if I see a bug that has been fixed, that has not tests, I’ll leave a comment on the bug.
The image at the top of this page shows the current code coverage percentages for the various XSL Tools related plugins/bundles as of today. What does this code coverage report say to me?
- The PsychoPath processor has decent code coverage. It’s close to the 80% mark which I was striving for. (Side Note: it also has 8000 plus tests as well).
- Most of the core plugins in XSL Tools have decent code coverage from the current test suite. While not quite where I want it to be, it is better than other portions.
- The Launching and Debugging framework portion is one of the weakest areas that we are testing. It can mean that the code is overly complicated, was not written with tests in mind, and probably never had tests to begin with. It also probably has hidden bugs that need to be tracked down.
Note: Recently while writing a unit test for one bug, I ended up finding and fixing a couple more that were in the same general area. Writing tests can reveal bugs that are hiding. Bugs that will eventually bite your adopters and keep you away from that new feature you would rather be writing than fixing a bug.
With this said, just looking at the overall project level percentage isn’t enough. It can hide areas where there still needs to be coverage. For example, look at the xsl.core project at the package level.
What we see here is that the reason it’s only at 63% coverage is because there are some packages that have very weak code coverage. Again these are places where there are no unit tests that are executing that portion of the code. If we were to drill down farther we could see at a Class and even Method level which pieces are actually getting executed.
So even if you have a fairly high percentage on a project you still may not be getting all the coverage you think.
One thing I have learned is that there is usually a pretty good relationship to the number of bugs that are reported against a particular area, and the amount of code coverage that is there. Typically the lower the percentage, the more likely there are bugs. Another thing is that you may also just have dead code that is no longer being executed.
Running coverage reports occasionally is a good thing, it will reveal what your test suite is actually executing, compared to what you think your test suite is executing. It reveals where your coverage is good and where you need to improve the coverage.