(This step-by-step is assuming you have Java installed on the machine you are using.)
First we'll need to download Hudson here and start our Hudson instance: (just grab the latest stable release)
java -jar hudson.war
(see Hudson site for other deployment options.)
Next, grab the CI::Reporter gem we'll need for test results xml formatting :
gem install -y ci_reporter
OR for JRuby
jruby -S gem install -y ci_reporter
UPDATE: Make sure you insert the following lines in your Rakefile:
Hit your running Hudson instance: http://localhost:8080/ and configure a new build by clicking the "New Job" link. Enter the name of your build and select the "Build a free-style software project" (click OK.)
Configure your SCM data, then scroll down to the "Build" section and select either "Execute Windows batch command" (windows) or "Execute shell" (*nix). This is where we'll call our 'rake test' task using the CI::Reporter gem and specifying a spot for the report XMLs to live:
OR for JRuby
UPDATE: When using Subversion, it will check things out into a sub directory of your "workspace" (found ~<user running hudson>/.hudson/jobs/<your job name>/workspace) If you are checking out 'trunk' and don't specify a different name for the directory, your shell execution settings should look like this:
(be sure to know how your SCM works before applying the following)
This will setup the CI::Reporter and place the XMLs in the "results" directory for us.
We are almost done, now under the "Post-build Actions" section, check the "Publish JUnit test result report" box and enter the following in the "Test report XMLs" field:
Now click Save and go test it out... Select your newly created job from the Dashboard.
Click the "Build Now" link... done. After a few builds you should see a graph on the job's main page that looks like this:
If you have a failure, you can drill down by clicking the "Test Result" link on an individual build's page.
Clicking on an individual failed test's name will show you the output from that failure
That's all it takes, pretty simple and works like a charm. Now you can go back and play with the other settings Hudson has, triggers, scheduling, job artifacts, notifications... you could even have a job that prepares a build by generating all your static content so your server doesn't have to take the initial hit after startup. (web projects)
If interested, there is a way to create plugins for Hudson.