Over the last few months we have been attempting to implement some 'better' best practice methodologies within our SDLC. Some of the things that we have already been addressing are:
- Simple Coding Standards and Rules
- Nant Build Scripts
- Continuous Integration with the help of Cruise control
Although I feel this is a good start, one of the more critical best practice aspect that has not been properly addressed is Unit and Regression testing. At the beginning of the previous project we had good intentions but because we got extremely battered for time, the Unit tests were all but abandond. However, the time has come to fully welcome on board Unit and Regression tests, then try and finally get some numbers in the NCover stats.
I remember walking past the ParaSoft stand in the JavaOne pavillion back in 2004 where they where promoting JTest. They now have .TEST for Visual Studio 2003/2005. Last week I was able to load up a trial version of .TEST for VS2003 to have a bit of a look. Initially I found the integration appealing, an example of this is code that violates a rule is underlined, documentation and an example on how to fix it is provided with a simple right-click.
Rules validation Keypionts
- Integrates directly into Visual Studio, underlines violations.
- Create custom rules configurations (some of the rules seem a little silly, they mean well, but aren't always practical)
- Each rule documented with examples of the violation and how to fix it.
- Based on the best practice standards on MSDN.
The rule validator does a pretty reasonable job, although be warned checking a 100,000+ line application can be a little bit too slow. However once this is done the first time, you can check files individually. It appears that all this rule validation data is stored inxml files which makes loading up the solution a little slow aswell.
But what about FxCop?
Yes, what about FxCop?! FxCop is fast but it generates a massive list of violations based on the IL, not on the source. FxCop certainly does the job, but if you want something a little more user-friendly and 'right-there' in visual studio then the .TEST rule validation is worth a look.
Unit test generation Keypoints
- Automatically creates and adds files to a testing project in the solution
- Based on NUnit
- Inbuilt coverage statistics
- Has own custom extensions (nunit.framework + dottest.framework)
Although I didn't really go as in depth into the test generation as I would have liked (which was the point of looking into .TEST), one of the reasons for not is that the generation is really slow. I'm sure that .TEST is crunching/parsing/analysing/generating lots and lots of stuff, but bottom line is...it's almost too slow. However after the test cases are (eventually) generated they are easily customisable to make sure they're doing the right thing. .TEST kindly marks all new tests it generates with a 'Not Verified' flag so you don't get confused. Generally speaking the generated code is relitively clean, easy to read and modify, but be warned it can spew out some really large files.
I'd say that already I've started a love-hate relationship with .TEST. There are also some more extensive reports that .TEST generates. These are broken down by team member with a small todo list of tasks (about 50) for team members to complete each day. This is a good little personalisation that helps prevent becoming over-whelmed if the list is 1000s of items long.
The Unit test generation is probably the slowest feature (even on a per-file bases) but it makes setting up the tests very simple to do. I think on the most part I prefer black-box testing which .TEST handles with no dramas.
.TEST seems like a relatively good little package which is enjoyable to use. However the things I have been reading about Visual Studio Team Foundation Server makes me think that it may have all these bases covered plus many more.
Another product that I have also been able to try is NTest by Incenteus. Basically has the same Unit Test generation features as .TEST but it works off the IL code. For some reason I just don't think that NTest is as intelligent when it comes to generating relevant code, but as a cheaper alternative I guess it might be worth while.