5.3.2013

Monitoring Tests - When This Runs Over

Martin Fowler made a reference to Monitoring Tests in his review of Gap Inc.'s SCMS PO System - testing. I really like the idea of formalizing monitoring as a type of testing. Monitoring involves an assertion like any other test. Why not automate the validation of that assertion as the system is running?

Premature optimization is wasteful

I prefer to avoid premature optimization and fussing over performance of systems. In my experience systems run magnitudes faster than necessary 99% of the time. Sometimes I have a section of code that I don't expect to take more than a specified amount of time and I use that assumption to design simplified, more reliable code. If that assumption is violated, it would be nice to know about it.

Reacting to disaster

I often find there is a threshold above my assumption that can be crossed and still not cause any problems. At that point though, it might be a good idea to start talking about optimization. In the worst case these assumptions are absent from code, in the best case they may be in a comment:

// Processing settlements shouldn't take more than 10 minutes, it could be an issue if it gets close to 30 minutes because we try to email statements an hour after processing starts.
ProcessSettlements();

Even in this case, going over an hour probably isn't a big deal. There's probably defensive coding in the statement module to abort or defer sending statements if settlements aren't complete. But who wants to wait an hour to find out there might be a problem?

Being Proactive - When This Runs Over

Why not step it up a notch and turn the comment into an executable assertion?

using (WhenThis.RunsOver(Settings.Default.SettlementsWarningDuration).Then(SendSettlementsWarning))
{
    ProcessSettlements();
}

private void SendSettlementsWarning(TimeMonitor monitor)
{
    // put whatever code to send the warning here, this just shows how you can use the constraints of the monitor to format the message
    var message = new
        {
            Subject = "Settlements Are Taking Longer Than Expected",
            Body = "Process started at " + monitor.Started + " and is expected to complete in " + monitor.NotifyAfter + " but it's " + DateTime.Now " and they are not yet complete.",
            //Recipients
        };
    // message.Send();
}

The duration is configurable so I can adjust it to avoid being pestered, but I can find out well before an hour if I have a problem!

Here's the full example with some tests.

Monitoring Tests defined

I would define Monitoring Tests as any test that validates a system as it runs and provides automated feedback to those that maintain the system.

Using AAA syntax, the components of these tests are:

  • Arrange: this already exists, it's the system itself!
    • ie: inputs to the ProcessSettlements method
  • Act: this is the scope we want to validate, which also already exists!
    • ie: the ProcessSettlements method
  • Assert: this is the only missing piece, it requires identifying missing assumptions and/or assumptions in comments and making them executable
    • ie: the consumer of the ProcessSettements method has constraints for how long it should take because it has to run statements later

Summary

There is immense value in making assumptions in a system executable and providing automated feedback. Time tracking, exception management, logging and performance counters are ways we already use this. Monitoring Tests are about having your system tell you about problems as soon as possible, preferably before they occur! Dip your toe in the water, find a framework to help, use a service online.

The value of formalizing monitoring as a type of test is that we begin to think explicitly in terms of the AAA syntax and methodically engineering what of this we automate and what we don't. Like other tests, we can evolve these to meet ever changing monitoring needs.





comments powered by Disqus