I stumbled upon this article this morning: Why your users hate Agile development (and what you can do about it).
The article is a great starting point for at least a dozen conversations. Here are some of my ideas to mitigate the risks the article introduced:
In my last post I introduced some benefits of using reactive programming techniques. This post will start us down the journey of testing so that subsequent topics can easily be expressed and verified.
As previously mentioned, when working with APIs it's helpful to introduce an anti-corruption layer. This layer allows you to build strong separation between an external system and your own. Think of this layer as a condom, practice safe development! This layer is a great place to transform and filter interactions to constrain how you use the external system and to inject a layer that can be very helpful for testing.
FuturesQuoteClient publishes FuturesQuotes via an event. First, we'll build a small anti-corruption layer with transformed events. Then we can do the same with observables. The anti-corruption layer will transform a FuturesQuote to QuoteWithContract and exclude quotes with invalid contracts.
The FuturesQuoteClient for reference:
public class FuturesQuoteClient
{
public delegate void QuoteHandler(object sender, FuturesQuote quote);
public event QuoteHandler Quotes;
...
}
Most APIs that use events won't have the ability to stub them for testing. Also, in c#, only the defining class of an event can raise it. Therefore, to separate the real client we'll need a wrapper around it. This is like a wrapper before our anti-corruption wrapper :)
/// <summary>
/// An interface to abstract the quote source.
/// </summary>
public interface IFuturesQuoteClient
{
event FuturesQuoteClient.QuoteHandler Quotes;
}
This allows us to abstract an interface which we can stub during testing. Here's an implementation of the wrapper for the real client:
/// <summary>
/// Implementation of real wrapper around quote source.
/// </summary>
public class FuturesQuoteClientWrapper : IFuturesQuoteClient
{
public FuturesQuoteClientWrapper(FuturesQuoteClient client)
{
client.Quotes += (sender, quote) =>
{
var handler = Quotes;
if (handler != null) handler(this, quote);
};
}
public event FuturesQuoteClient.QuoteHandler Quotes;
}
The wrapper takes a FuturesQuoteClient and forwards quotes to the IFuturesQuoteClient.Quotes event interface. I'm not going to test this wrapper, I only included it to show the extra dimension of complexity required. This wrapper would require an integration test with the real quote service.
Now that we can isolate the anti-corruption layer from the real client, we can begin to build the functionality through tests.
First, we want to transform FuturesQuote to QuoteWithContract:
[Test]
public void OnFuturesQuote_AValidContractOnAFuturesQuote_TriggersQuoteWithContract()
{
var validFuturesQuote = new FuturesQuote
{
Symbol = "CZ2013"
};
var futuresQuoteClient = MockRepository.GenerateStub<IFuturesQuoteClient>();
var quotesWithContractClient = new AntiCorruptionLayerEventClient(futuresQuoteClient);
var quotes = new List<NotifyOnBarrierEventsReactive.QuoteWithContract>();
quotesWithContractClient.Quotes += (sender, quote) => quotes.Add(quote);
futuresQuoteClient.Raise(c => c.Quotes += null, null, validFuturesQuote);
quotes.Should().HaveCount(1);
var quoteWithContract = quotes.Single();
quoteWithContract.Quote.ShouldBeEquivalentTo(validFuturesQuote);
}
FuturesQuote. IFuturesQuoteClient instead of creating this class manually. AntiCorruptionLayerEventClient is the class that will implement the anti-corruption layer, it doesn't exist yet. It requires a client of type IFuturesQuoteClient. AntiCorruptionLayerEventClient.Quotes and put the quotes into a list.IFuturesQuoteClient stub with the validFuturesQuote event argument.QuoteWithContract arrives for the original validFuturesQuote.First we need our transformed event:
public class AntiCorruptionLayerEventClient
{
public delegate void QuoteWithContractHandler(object sender, NotifyOnBarrierEventsReactive.QuoteWithContract args);
public event QuoteWithContractHandler Quotes;
protected virtual void OnQuotes(NotifyOnBarrierEventsReactive.QuoteWithContract quote)
{
var handler = Quotes;
if (handler != null) handler(this, quote);
}
...
}
That's a lot of typing, events aren't fun!
To get the test to pass we have to transform and forward the quotes:
public class AntiCorruptionLayerEventClient
{
...
public AntiCorruptionLayerEventClient(IFuturesQuoteClient client)
{
client.Quotes += TransformToQuoteWithContract;
// neglects handling unsubscribing from the event and cascading that up the chain :)
}
private void TransformToQuoteWithContract(object sender, FuturesQuote quote)
{
OnQuotes(new NotifyOnBarrierEventsReactive.QuoteWithContract(quote));
}
}
The constructor takes IFuturesQuoteClient, subscribes to FuturesQuote events and transforms them to QuoteWithContract. Finally, it raises the transformed event.
IFuturesQuoteClient abstraction.The above example demonstrates testing the composition of the anti-corruption layer. Testing the actual transform is better left to a unit test where the infrastructure of the anti-corruption layer doesn't get in the way of more detailed testing. Here's one possible test:
[Test]
public void MapFromFuturesQuote()
{
var quote = new FuturesQuote
{
Symbol = "CZ2013"
};
var commodityContract = NotifyOnBarrierEventsReactive.QuoteWithContract.ContractFromQuote(quote);
commodityContract.ContractMonth.Should().Be(12);
commodityContract.ContractYear.Should().Be(2013);
commodityContract.ProductCode.Should().Be("C");
}
Notice how simple this test is! Regardless if we use events or observables these tests will look the same.
Now let's look at the same scenario with an anti-corruption layer built on observables. To isolate the client we still have to apply a wrapper, fortunately the reactive extensions provides this for us (as shown in the last post):
private static IObservable<FuturesQuote> FuturesQuotesSource(FuturesQuoteClient client)
{
return Observable
.FromEvent<FuturesQuoteClient.QuoteHandler, FuturesQuote>(h => client.Quotes += h, h => client.Quotes -= h);
}
Again, I'm not testing the wrapper to isolate our anti-corruption layer, that would be done in an integration test. However, since we're leveraging a built in wrapper, we don't have to worry about as many test cases as we would with our implementation of FuturesQuoteClientWrapper.
Here's the equivalent test to verify quotes are transformed:
[Test]
public void OnFuturesQuote_WithAValidContract_StreamsQuoteWithContract()
{
var validFuturesQuote = new FuturesQuote
{
Symbol = "CZ2013"
};
var scheduler = new TestScheduler();
var futuresQuotes = scheduler.CreateColdObservable(ReactiveTest.OnNext(0, validFuturesQuote));
var quotesWithContractClient = new AntiCorruptionLayerObservableClient(futuresQuotes);
var quotesWithContracts = scheduler.Start(() => quotesWithContractClient.Quotes);
quotesWithContracts.Messages.Should().HaveCount(1);
var quoteWithContract = quotesWithContracts.Messages.Single().Value.Value;
quoteWithContract.Quote.ShouldBeEquivalentTo(validFuturesQuote);
}
FuturesQuote.TestSchedulerTestScheduler is optimized to control and monitor the testing process.AntiCorruptionLayerObservableClient requires an IObservable<FuturesQuote>TestScheduler.CreateColdObservable creates this observable for testing, there's more to learn about this API but for now those details aren't relevant.validFuturesQuote message in the test observable.scheduler to Start which causes it to dispatch our validFuturesQuote messageStart method takes an IObservable<T> and returns an ITestableObserver<T> where T is our transformed QuoteWithContractSelect transforms but will be invaluable later on.validFuturesQuoteTestScheduler is designed to help with testing situations where time is involved. There's a bit of overhead to test Select operations where timing is irrelevant0 above) when setting up the observable scenario (scheduler.CreateColdObservable(ReactiveTest.OnNext(0, validFuturesQuote)))Messages.Single().Value.Value)There's one constructor to create the AntiCorruptionLayerObservableClient from the FuturesQuoteClient, and one to create this from IObservable<FuturesQuote>. These are equivalent to FuturesQuoteClientWrapper and AntiCorruptionLayerEventClient.ctor(IFuturesQuoteClient) respectively:
public class AntiCorruptionLayerObservableClient
{
...
public AntiCorruptionLayerObservableClient(FuturesQuoteClient client)
: this(FuturesQuotesSource(client))
{
}
public AntiCorruptionLayerObservableClient(IObservable<FuturesQuote> quotes)
{
Quotes = quotes
.Select(q => new NotifyOnBarrierEventsReactive.QuoteWithContract(q));
}
public IObservable<NotifyOnBarrierEventsReactive.QuoteWithContract> Quotes { get; private set; }
}
The only real code involved is in transforming the source observable, one call to Select!
In the anti-corruption layer event sample I didn't go into excluding quotes with invalid contracts. It's trivial and I think the reactive sample is much more expressive:
[Test]
public void OnFuturesQuote_WithAnInvalidContract_StreamsNothing()
{
var invalidFuturesQuote = new FuturesQuote
{
Symbol = "invalidcontract"
};
var scheduler = new TestScheduler();
var futuresQuotes = scheduler.CreateColdObservable(ReactiveTest.OnNext(0, invalidFuturesQuote));
var quotesWithContractClient = new AntiCorruptionLayerObservableClient(futuresQuotes);
var quotesWithContracts = scheduler.Start(() => quotesWithContractClient.Quotes);
quotesWithContracts.Messages.Should().BeEmpty();
}
To get this test to pass we simply add a filter:
public AntiCorruptionLayerObservableClient(IObservable<FuturesQuote> quotes)
{
Quotes = quotes
.Select(q => new NotifyOnBarrierEventsReactive.QuoteWithContract(q))
.Where(NotifyOnBarrierEventsReactive.IsValidContract);
}
Implementing an anti-corruption layer around information streams with reactive programming offers several benefits over imperative approaches. I'll admit the benefits aren't as exciting without the time dependent aspects. However, when working with time it will become patently obvious that there's no real comparison.
Reactive programming offers many advantages over traditional imperative techniques when handling information streams like commodity market data. In this and several follow up posts I want to demonstrate some of the advantages I've found using these techniques. I'll be demonstrating this with c# and the Reactive Extensions framework.
What is reactive programming? From Wikipedia:
"reactive programming is a programming paradigm oriented around data flows and the propagation of change." en.wikipedia.org/wiki/Reactive_programming
Spreadsheet calculations are a great example of reactive techniques. If you are unfamiliar and want to know more look at the Wikipedia article or follow along in this series as I demonstrate some of the advantages.
I'm going to use barrier options to demonstrate the value of reactive programming. From Wikipedia:
a barrier option is an exotic derivative typically an option on the underlying asset whose price breaching the pre-set barrier level either springs the option into existence or extinguishes an already existing option. en.wikipedia.org/wiki/Barrier_option
If you aren't familiar with finance, an option is easy to think of as an insurance policy. One purpose is to protect a future purchase from major price changes. Imagine if you could buy insurance to cover the cost of gas if it rose above an extreme level, say $8/gallon.
To protect against major price swings you may need to pay a hefty premium. One way to reduce the premium is to purchase a barrier option. One way a barrier can work is to cancel the insurance policy. Say you are worried about gas rising above $8 a gallon but you doubt it will rise about $12. You might be willing to pay a reduced premium in exchange for the small chance that gas will breach $12 a gallon causing the policy to be canceled. Barriers in this case reduce the seller's risk and hence reduce premium.
If a breach occurs you might want to find out ASAP to mitigate the risk. This is the scenario I will use to demonstrate reactive techniques.
First I'll start with an imperative approach that is pretty common to come across.
Note: Most of the following code can be found in a more complete fashion here https://github.com/g0t4/blog-samples/tree/master/reactive/src.
Many APIs that provide quotes in c# have been around a while and therefore use an event based interface:
var client = new FuturesQuoteClient();
client.Quotes += OnQuote;
Simply attach an OnQuote handler to process quotes.
In this example I'm going to assume there is a "cache" of options that gets updated periodically. The details of this are irrelevant.
_ActiveBarrierOptionsByContract = GetActiveBarrierOptionsByContract();
Here is an example of the OnQuote handler. I tried to leverage extracted methods and LINQ to show even in an imperative fashion, the code can be concise and clean (at least in my opinion):
private void OnQuote(object sender, FuturesQuote quote)
{
var contract = ContractFromQuote(quote);
if (IsValidContract(contract))
{
return;
}
IEnumerable<CommodityBarrierOption> activeBarrierOptionsForContract;
if (!_ActiveBarrierOptionsByContract.TryGetValue(contract, out activeBarrierOptionsForContract))
{
return;
}
activeBarrierOptionsForContract
.Where(option => BarrierIsBreached(option, quote))
.Where(NoticeNotAlreadySent)
.ToList()
.ForEach(option => NotifyTheHumans(option, quote));
}
When the quote arrives, we parse the contract with ContractFromQuote. If the contract isn't valid we ignore the quote. When you interface with an external API there are going to be things you want to transform. This is part of your "Anti-Corruption layer" if you are familiar with domain driven design.
Without going into the details of options on futures, suffice to say the contract describes what type of gas (say regular versus premium) and when you will be buying it (a month). I'm really simplifying this as the location would matter too.
If _ActiveBarrierOptionsByContract doesn't contain the contract, we ignore the quote. We only care about quotes for contracts that we insured.
Next, we check each option to see if the BarrierIsBreached and if NoticeNotAlreadySent. If these conditions are met we'll NotifyTheHumans.
This example ignores any performance considerations. I'll address those in subsequent posts.
Now let's refactor to incorporate reactive patterns. Starting imperative means the reactive solution will be somewhat "tainted" but will suffice to move us in the reactive direction.
The imperative solution makes it difficult to decouple transforms and filters from handlers. Instead of receiving all FuturesQuotes, we only want to know about quotes with valid contracts. We could create new events and wire up conditions to trigger them. However, leveraging the reactive framework makes this possible with much less typing and in a much more expressive fashion.
First we have to wrap the FuturesQuote event to produce an observable. Think of this as a stream of FuturesQuotes:
private static IObservable<FuturesQuote> FuturesQuotesSource(FuturesQuoteClient client)
{
return Observable
.FromEvent<FuturesQuoteClient.QuoteHanlder, FuturesQuote>(h => client.Quotes += h, h => client.Quotes -= h);
}
This creates the observable:
var client = new FuturesQuoteClient();
FuturesQuotesSource(client)
We now have a stream of FuturesQuotes.
The Reactive Extensions provides LINQ operators that work on observables. In the example above we parse the contract into a local variable, instead we could create a new type QuoteWithContract and store both the quote and the parsed contract:
public class QuoteWithContract
{
public FuturesQuote Quote { get; set; }
public CommodityContract Contract { get; set; }
public QuoteWithContract(FuturesQuote quote)
{
Quote = quote;
Contract = ContractFromQuote(quote);
}
...
}
Now we can apply the Select LINQ operator to transform our stream from FuturesQuote to QuoteWithContract:
var client = new FuturesQuoteClient();
FuturesQuotesSource(client)
.Select(q => new QuoteWithContract(q))
The other thing we want to ensure is that the contract was valid, we can do this with the Where LINQ operator:
FuturesQuotesSource(client)
.Select(q => new QuoteWithContract(q))
.Where(IsValidContract)
private static bool IsValidContract(QuoteWithContract quote)
{
return quote.Contract != null;
}
I like extracting method conditions to increase readability of the stream composition. We now have a stream of only quotes with a valid contract.
At this point we could Subscribe to the quotes stream and execute our imperative code:
private void OnQuote(QuoteWithContract quote)
{
IEnumerable<CommodityBarrierOption> activeBarrierOptionsForContract;
if (!_ActiveBarrierOptionsByContract.TryGetValue(quote.Contract, out activeBarrierOptionsForContract))
{
return;
}
activeBarrierOptionsForContract
.Where(option => BarrierIsBreached(option, quote.Quote))
.Where(NoticeNotAlreadySent)
.ToList()
.ForEach(option => NotifyTheHumans(option, quote.Quote));
}
But why not keep with the concept of a stream and re-frame our solution as producing a stream of barrier breach notices! To do this I've taken the above code and extracted this method that can take a quote and create the resulting breach notices:
private IEnumerable<BarrierBreachNotice> BarrierBreachNotices(QuoteWithContract quote)
{
IEnumerable<CommodityBarrierOption> options;
if (!_ActiveBarrierOptionsByContract.TryGetValue(quote.Contract, out options))
{
return Enumerable.Empty<BarrierBreachNotice>();
}
return options
.Where(option => BarrierIsBreached(option, quote.Quote))
.Select(option => new BarrierBreachNotice(option, quote));
}
public class BarrierBreachNotice
{
public BarrierBreachNotice(CommodityBarrierOption option, QuoteWithContract quote)
{
Option = option;
Quote = quote;
}
public CommodityBarrierOption Option { get; set; }
public QuoteWithContract Quote { get; set; }
}
Now we can transform a stream of quotes, along with our active barrier options "cache", into a stream of BarrierBreachNotice. We'll use the SelectMany operator as each quote could generate more than one notice:
FuturesQuotesSource(client)
.Select(q => new QuoteWithContract(q))
.Where(IsValidContract)
.SelectMany(BarrierBreachNotices)
And we want to avoid duplicate notices:
FuturesQuotesSource(client)
.Select(q => new QuoteWithContract(q))
.Where(IsValidContract)
.SelectMany(BarrierBreachNotices)
.Where(NoticeNotAlreadySent)
Our original goal was to notify the humans:
FuturesQuotesSource(client)
.Select(q => new QuoteWithContract(q))
.Where(IsValidContract)
.SelectMany(BarrierBreachNotices)
.Where(NoticeNotAlreadySent)
.Subscribe(NotifyTheHumans);
private void NotifyTheHumans(BarrierBreachNotice notice)
{
...
}
This subscription is much less work, no awkward += on an event and object sender overhead, a simple call to Subscribe!
Notice how easy it is to continue the idea of a stream. Transforms and filters are very easy to use to partition event handling into an ordered, composable pipeline. This is one of the extremely powerful aspects of reactive programming abstractions.
Next I want to discuss testing, further thinking in streams, performance and other considerations and how reactive programming techniques apply.
This is pretty common in a software development project:
We want BizTalk, .net, ruby, Nodejs, SQL Server, Sharepoint, Oracle...
This is pretty common too:
We want a table of customers.
We want an export of invoices.
We want a list of orders.
We want the ability to apply discounts.
We want a report of inventory.
This is very rare:
We want to provide more value to our customers.
We want more customers and to do that we may need to increase the frequency of doing X.
We want to increase customer satisfaction.
We want happier employees.
We want to expand Y and to do that we may need to automate Z.
When I hear this, I know I can make a difference.
This is my opinion:
Deliverables (ie: features, technologies, software) should be a means to an end.
Why?
If you want to maximize business value, the end should be a business objective. If you want to maximize features, technologies and software then let those be your ends.
I'd prefer to minimize features, technologies and software. They aren't cheap to maintain!
In February I decided to set aside full time development to take an academic journey with some new ideas in software development. It's been a few months and I've consumed some amazing content. Learning is only half of the equation, now I'm thirsty for some feedback! I've decided to pursue two new ventures: value based consulting and developer coaching.
I'm very interested in what happens when value is more central to the development process. In the past I've worked with very talented people to create highly valuable solutions. But the value was often dependent on the customer asking for the right features. At times we would build amazing support for a requested feature only to find lack luster usage. See my article on why I think this happens: Why objectives and value are important in software development.
I believe the conversation has to shift to diagnosing value first, then prescribing solutions. The first step I will make is to incorporate the ideas of value based pricing. To hold myself accountable to delivering value, not just software. That of course is easier said than done. It will require time to learn and re-engineer the processes I use as a developer. I'm very excited about the possibilities!
In addition to my passion for innovating in software development, I have a desire to share what I do with others. I want to transform my experiences developing, speaking and training into coaching opportunities. Coaching encompasses a more objective approach to helping a team achieve results. It will be tailored to the goals of each development team and the software they are building.
To start this process I've partnered with JetBrains to offer my experience with several of their products, especially ReSharper. I look forward to partnering with other companies and offering coaching for a variety of topics.
I've started Full City Tech Co. to bring these ideas to life. Everyone knows I'm a coffee aficionado. Full City is a stage in the roasting process when the beans enter the medium to dark range, in my opinion when the coffee starts to taste good! Likewise, the practices I'm investing in are really rounding out my expertise as a consultant.
I'll keep posting more about my efforts here and on the Full City Tech Co. site. If you're interested in what I'm doing or know someone that might be, feel free to get in touch with me.
I want to thank everyone I've worked with at Phoenix Web Group, School District 145, friends and family who have helped me grow and supported me in everything I've done. I look forward to what I can learn and what I can share in return!