Satalyst Brilliance on 28 Jul 2015

Azure Continuous Integration Monitoring Part 3

Satalyst Innovation

Azure powered multi-stage CI monitoring tools

This is the second in a series of 3 posts covering something fun we’ve been messing around with here at Satalyst. Part 1 includes a little about what we’re trying to do, part 2 describes the infrastructure we built on Azure to do all the heavy lifting, and part 3 shows the finished shiny and wraps things up.

In the previous posts, I wrote about the new CD pipeline monitoring infrastructure that the Satalyst folks have been building in their spare time over the last few months with the ultimate aim of building a better build light.

So far we’ve used the infrastructure I described in the previous two posts for two purposes. During the March 2015 FedEx day, Khoi wrote a web dashboard that pulls data from the DocumentDB to let you inspect the historical results of all the builds being monitored. At the June FedEx day, I built the physical LED board and wrote the software to run on the Pi that subscribes to the Service Bus Topic and controls the lights.

And here’s the finished product!

Anthony's FedEx project

You can see a Raspberry Pi 2 on the right hand side hooked up to a simple set of 10 LEDs with current limiting resistors. This can handle a 5-stage pipeline, and currently shows a complete 4-stage pipeline in a healthy condition. Stage 1 is at the top, stage 5 (not used in the above picture) at the bottom.

As you can probably tell, the red and green LEDs are paired together, a fancier version might use an RGB LED so you only need a single LED per pipeline stage, but hey, you’ve gotta start somewhere!

As each stage is building, both the red and green LED for that stage blink to indicate that the stage is in progress. It turns out that a 500ms duty cycle for the blinking is quite pleasant – fast enough to tell it’s doing stuff, slow enough to not impart a sense of urgency.

When the first stage starts, it will blank out all the other stages, to make it clear that the entire pipeline has to be considered together. I took a short video of the first stage blinking, but it’s kind of boring, we’ve all seen LEDs blinking… Just picture a red and green LED blinking on off on off on off for two minutes…

Once a stage is successful, it goes green, and the next stage starts blinking. And so on until all the stages are done. If a stage is unsuccessful, it goes red, and the pipeline stops.

Given that the notifications that drive the lights contain the status of the entire pipeline, it would be really easy to build some other device that does something different, like set off a confetti cannon when the whole pipeline passes!

So what’s left to do? Well the software on the Pi is pretty simple, and totally not professional quality, so it needs some clean up and redesign. There’s a bunch of additional features it needs too, like the ability to set it to only display the status of a specific pipeline (it currently just sets the lights to the status given in the last message it saw), the ability to remotely change which build it’s displaying the status for, to solder it up neatly on a protoboard and put it in a jiffy box and stick it to a wall, etc.

It’s worth remembering that the Pi, and even the web dashboard, just tie into the Azure infrastructure we’ve put in place, so the possibilities to consume the events and historical data are pretty endless. All that’s needed is a network connection and the ability to read from either the DocumentDB or Service Bus Topic.

Adding in worker tools other than TeamCity and Octopus Deploy would also be easy, as long as the tool can emit notifications to an XMPP room, and someone writes the appropriate notification parser so we can make sense of the notification.

Now to try and sneak that confetti cannon into the budget 🙂

Confetti canon