even more FitNesse GoodNesse
running all your suites on a regular basis
We ran into a little hitch not too long ago with our fitnesse tests. We
are now up to 4 different suites of fitnesse tests, one for each of 4 different
solutions. The solution have dependencies among themsleves. We had been doing
work on one solution, doing our TDD red-green-refactor and our fitnesse
equivalent. Moving along nicely. And then cam a day and we had to do some work
on a different solution that we hadn't made any changes to in a few weeks. Made
some simple changes, checked them in, and (as described in a few
previous
posts) our CC.net build kicked off, followed by the fitnesse tests for that project.
What surprised us was that the fitnesse tests broke. In a strange place. Turns
out we had an integration breakage. Changes to project B had caused the fitnesse
tests in project A to fail, and we hadn't noticed since those tests hadn't been
run in a couple of weeks. So after we got the tests passing, we resolved to set
up a system to keep us from going that long with breakages we didn't know about.I set up a new project in my ccnet.config file like this:We set up a new build file to run all 4 fitnesse suites - the parts that we call the 'regression' suite, which we expect
to always pass. As you can see in the trigger block, it runs every night at 11:30, whether there have been any code changes or not.
The build file is where I think the interesting bits are.
Since properties in nAnt are mutable, it makes it pretty easy to do paramaterized targets. The strangest bit about this build file is how it adds up the number of faiures.
Fitnesse TestRunner is nice enough to return an integer that indicates the number of failures and exceptions. When we exec the TestRunner, we set failonerror
to false, and collect the result into a property. As mentioned before, we use the java tools to format the XML output into html. Then, we do a little nAnt
expression voodoo to add up all the failures, so that failing one suite doesn't prevent the other suites from running. It is only at the very end
of the 'fit' target that we fail with a message if there were any failures in any of the suites. Since we already have the fitnesse-details.xsl stylesheet in place,
we can then see what the problems were right in our CC.Net dashboard.Pretty cool.
<project name="nightly_fitnesse"> <webURL>http://buildserver/ccnet/Controller.aspx?_action_ViewProjectReport=true&server=buildserver&project=nightly_fitnesse</webURL> <triggers> <scheduleTrigger time="23:30" buildCondition="ForceBuild" /> </triggers> <sourcecontrol type="svn"> <executable>c:program filessubversionbinsvn.exe</executable> <trunkUrl>svn://codeserver/fitnesse/trunk/</trunkUrl> <tagBaseUrl>svn://codeserver/fitnesse/tags/</tagBaseUrl> <workingDirectory>d:workfitnesse-trunk</workingDirectory> <tagOnSuccess>false</tagOnSuccess> <autoGetSource>true</autoGetSource> <username>cruisecontrol.net</username> <password>ccnetpassword</password> </sourcecontrol> <tasks> <nant> <executable>D:workdependencies-trunkexternalnantNAnt.exe</executable> <baseDirectory>d:workfitnesse-trunk</baseDirectory> <buildFile>AustinNightlyFitnesse.build</buildFile> <targetList> <target>fit</target> </targetList> <!-- this build timeout is fairly long, since these tests can get long. --> <buildTimeoutSeconds>1500</buildTimeoutSeconds> </nant> </tasks> <publishers> <merge> <files> <file>D:workfitnesse-trunkresults*.xml</file> </files> </merge> <xmllogger /> </publishers> </project>
<?xml version="1.0" encoding="utf-8" ?> <project name="AustinNightlyFitnesse" default="fit" xmlns="http://nant.sf.net/release/0.85-rc3/nant.xsd"> <property name="fit.server" value="localhost" /> <property name="fit.port" value="8080" /> <property name="results.dir" value="results" /> <target name="fit" description="run fit tests"> <property name="FitRegressionFailureSummary" value="0" /> <delete dir="${results.dir}" failonerror="false" /> <mkdir dir="${results.dir}" /> <property name="fit.suite" value="GuideEngine.SuiteRegressionTests" /> <property name="fitlibs.dir" value="c:fitnessedotnetGuideEngine" /> <call target="runFitSuite" /> <property name="fit.suite" value="LookupWeb.SuiteRegressionTests" /> <property name="fitlibs.dir" value="c:fitnessedotnetLookupWeb" /> <call target="runFitSuite" /> <property name="fit.suite" value="FireFly.SuiteRegressionTests" /> <property name="fitlibs.dir" value="c:fitnessedotnetFireFly" /> <call target="runFitSuite" /> <property name="fit.suite" value="ShareDoc.SuiteRegressionTests" /> <property name="fitlibs.dir" value="c:fitnessedotnetShareDoc" /> <call target="runFitSuite" /> <fail message="${FitRegressionFailureSummary} Fitnesse Regression test(s) failed." if="${int::parse(property::get-value('FitRegressionFailureSummary')) != 0}" /> </target> <target name="runFitSuite" > <!-- same basic steps, but these must all pass, or the build fails --> <property name="fit.testrunner" value="${fitlibs.dir}TestRunner.exe" /> <echo message="running fit regression tests at http://${fit.server}:${fit.port}/${fit.suite} using ${fit.testrunner}" /> <exec program="${fit.testrunner}" commandline="-results ${results.dir}Fit-${fit.suite}.html ${fit.server} ${fit.port} ${fit.suite}" failonerror="false" resultproperty="FitRegressionResult" /> <!-- now transform those results from html to xml, and then fail if there were any errors. Fitnesse helpfully returns that as a number. --> <exec program="java.exe" commandline="-cp binfitnessefitnesse.jar fitnesse.runner.FormattingOption ${results.dir}Fit-${fit.suite}.html xml ${results.dir}Fit-${fit.suite}.xml ${fit.server} ${fit.port} ${fit.suite}" failonerror="false"/> <property name="FitRegressionFailureSummary" value="${int::parse(property::get-value('FitRegressionFailureSummary')) + int::parse(property::get-value('FitRegressionResult'))}" /> <echo message="${FitRegressionResult} Fitnesse Regression test(s) in ${fit.suite} failed." /> <echo message="So far, a total of ${FitRegressionFailureSummary} Fitnesse Regression test(s) failed." /> </target> </project>