even more FitNesse GoodNesse

Fri, 14 Apr 2006 02:40:29 GMT

We ran into a little hitch not too long ago with our fitnesse tests. We are now up to 4 different suites of fitnesse tests, one for each of 4 different solutions. The solution have dependencies among themsleves. We had been doing work on one solution, doing our TDD red-green-refactor and our fitnesse equivalent. Moving along nicely. And then cam a day and we had to do some work on a different solution that we hadn't made any changes to in a few weeks. Made some simple changes, checked them in, and (as described in a few previous posts) our CC.net build kicked off, followed by the fitnesse tests for that project. What surprised us was that the fitnesse tests broke. In a strange place. Turns out we had an integration breakage. Changes to project B had caused the fitnesse tests in project A to fail, and we hadn't noticed since those tests hadn't been run in a couple of weeks. So after we got the tests passing, we resolved to set up a system to keep us from going that long with breakages we didn't know about.

I set up a new project in my ccnet.config file like this:

<project name="nightly_fitnesse">
      <webURL>http://buildserver/ccnet/Controller.aspx?_action_ViewProjectReport=true&amp;server=buildserver&amp;project=nightly_fitnesse</webURL>      <triggers>
        <scheduleTrigger time="23:30" buildCondition="ForceBuild" />
      </triggers>
      <sourcecontrol type="svn">
        <executable>c:program filessubversionbinsvn.exe</executable>
        <trunkUrl>svn://codeserver/fitnesse/trunk/</trunkUrl>
        <tagBaseUrl>svn://codeserver/fitnesse/tags/</tagBaseUrl>
        <workingDirectory>d:workfitnesse-trunk</workingDirectory>
        <tagOnSuccess>false</tagOnSuccess>
        <autoGetSource>true</autoGetSource>
        <username>cruisecontrol.net</username>
        <password>ccnetpassword</password>
      </sourcecontrol>
      <tasks>
        <nant>
          <executable>D:workdependencies-trunkexternalnantNAnt.exe</executable>
          <baseDirectory>d:workfitnesse-trunk</baseDirectory>
          <buildFile>AustinNightlyFitnesse.build</buildFile>
          <targetList>
            <target>fit</target>
          </targetList>
          <!-- this build timeout is fairly long, since these tests can get long.  -->
          <buildTimeoutSeconds>1500</buildTimeoutSeconds>
        </nant>
      </tasks>
      <publishers>
      <merge>
          <files>
             <file>D:workfitnesse-trunkresults*.xml</file>
          </files>
      </merge>
        <xmllogger />
      </publishers>
    </project>

We set up a new build file to run all 4 fitnesse suites - the parts that we call the 'regression' suite, which we expect to always pass. As you can see in the trigger block, it runs every night at 11:30, whether there have been any code changes or not. The build file is where I think the interesting bits are.

<?xml version="1.0" encoding="utf-8" ?>
  <project name="AustinNightlyFitnesse" default="fit" xmlns="http://nant.sf.net/release/0.85-rc3/nant.xsd">      <property name="fit.server" value="localhost" />
      <property name="fit.port" value="8080" />
      <property name="results.dir" value="results" />
  
      <target name="fit"	description="run fit tests">
        <property name="FitRegressionFailureSummary" value="0" />
        <delete dir="${results.dir}" failonerror="false" />
        <mkdir dir="${results.dir}" />
        <property name="fit.suite" value="GuideEngine.SuiteRegressionTests" />
        <property name="fitlibs.dir" value="c:fitnessedotnetGuideEngine" />
        <call target="runFitSuite" />
        <property name="fit.suite" value="LookupWeb.SuiteRegressionTests" />
        <property name="fitlibs.dir" value="c:fitnessedotnetLookupWeb" />
        <call target="runFitSuite" />
        <property name="fit.suite" value="FireFly.SuiteRegressionTests" />
        <property name="fitlibs.dir" value="c:fitnessedotnetFireFly" />
        <call target="runFitSuite" />
        <property name="fit.suite" value="ShareDoc.SuiteRegressionTests" />
        <property name="fitlibs.dir" value="c:fitnessedotnetShareDoc" />
        <call target="runFitSuite" />
        <fail message="${FitRegressionFailureSummary} Fitnesse Regression test(s) failed."
          if="${int::parse(property::get-value('FitRegressionFailureSummary')) != 0}" />
      </target>
      <target name="runFitSuite" >
        <!-- same basic steps, but these must all pass, or the build fails -->
        <property name="fit.testrunner" value="${fitlibs.dir}TestRunner.exe" />
        <echo message="running fit regression tests at http://${fit.server}:${fit.port}/${fit.suite} using ${fit.testrunner}" />
        <exec program="${fit.testrunner}"
              commandline="-results ${results.dir}Fit-${fit.suite}.html ${fit.server} ${fit.port} ${fit.suite}" 
              failonerror="false" resultproperty="FitRegressionResult" />
        <!-- now transform those results from html to xml, and then fail if there were any errors. Fitnesse helpfully returns that as a number. -->
        <exec program="java.exe" 
              commandline="-cp binfitnessefitnesse.jar fitnesse.runner.FormattingOption ${results.dir}Fit-${fit.suite}.html xml ${results.dir}Fit-${fit.suite}.xml ${fit.server} ${fit.port} ${fit.suite}" 
              failonerror="false"/>
        <property name="FitRegressionFailureSummary" 
                            value="${int::parse(property::get-value('FitRegressionFailureSummary')) + int::parse(property::get-value('FitRegressionResult'))}" />
        <echo message="${FitRegressionResult} Fitnesse Regression test(s) in ${fit.suite} failed." />
        <echo message="So far, a total of ${FitRegressionFailureSummary} Fitnesse Regression test(s) failed." />
      </target>
  </project>

Since properties in nAnt are mutable, it makes it pretty easy to do paramaterized targets. The strangest bit about this build file is how it adds up the number of faiures. Fitnesse TestRunner is nice enough to return an integer that indicates the number of failures and exceptions. When we exec the TestRunner, we set failonerror to false, and collect the result into a property. As mentioned before, we use the java tools to format the XML output into html. Then, we do a little nAnt expression voodoo to add up all the failures, so that failing one suite doesn't prevent the other suites from running. It is only at the very end of the 'fit' target that we fail with a message if there were any failures in any of the suites. Since we already have the fitnesse-details.xsl stylesheet in place, we can then see what the problems were right in our CC.Net dashboard.

Pretty cool.

Something Pithy - testing tag

even more FitNesse GoodNesse