A Duty to Alert - PagerDuty at Compose

Compose's Tim Yocum recently wrote a guest blog at Pagerduty which we reproduce here for our readers who don't read that blog.

3

Compose users know that we take care of their data and their databases with our intelligent infrastructure and smart people ready to respond to alerts 24/7 and take care of any problems. That's part of the Compose promise. But we think there's a lesson to be learned by our users from how we handle those alerts.

Growing your own problems

A typical Compose customer has two major parts to their stack, their data storage and their application. Now while the data storage alerts are handled by us, we find there's often a good chance that there's no mechanism for generating and handling alerts in the application. It's not by design; early on, your own customers will be letting you know that your system isn't working and you'll have organically scaled to handle those complaints, translate them into issues with your application, triage the problems and resolve the issue.

But as you scale up your systems and company, that organic scaling puts a load on your people and that will affect response times and the quality of triage. Your application's architecture will become more complex with more moving parts and more subsystems. It is at this point we often find we get calls about database problems which turn out to be further up the customer's stack, up in the application.

Difficult fixes

There may already be instrumentation or monitoring in some of your system components so that you get a number of alerts from your systems. By adding more instrumentation you can cover the blind spots. Other systems can also provide performance metrics and regular checks. By adding all these to the equation you can provide as much alert visibility and sensitivity as possible.

But then you realize that what you've created is more alerts for your people handle. There'll be a lot of noise because you'll typically find that any single failure has a ripple effect in creating alerts in different systems. Some of the failures will be like smoke alarms, ringing loudly with no obvious cause, while others will be only generate one alert yet have a massive impact. On top of all of that these alerts may get to the people monitoring the system in a range of different routes. You may be tempted to build your own alert management system... but thats another component to monitor and you'll be spending more time engineering system monitoring than growing your business.

Getting help in

The team at Compose had experienced all this before and we went and addressed the problem early on. Our answer for our storage infrastructure is PagerDuty. PagerDuty is a service which lets us plug in all our monitoring tools into one reliable system. It's unlikely anyone could use all PagerDuty's integrations; there's 75 of them. They cover monitoring systems for infrastructure like Icinga, Nagios and Opsview, web availability like xpect.io and NodePing, application performance such as New Relic and AppDynamics, single sign on and API management. If you are building your own monitoring and alerting, you can plug in your own code into the system with an API, webhooks and email routes into the PagerDuty system.

PagerDuty pulls in all the information from your configured systems, then applies rules to the data to make sure that there's no "alert spam" where repeating alerts or side-effect alerts drown out the real issues. Then it takes those reduced alerts and contacts by email, phone or SMS the people who are on duty - using a PagerDuty managed on-call scheduler.

The contact engine is a good example of engineering you couldn't reproduce in-house - PagerDuty has redundant phone, SMS and email providers and constantly tests them to ensure they are not just working but working well all the time. Unless your business is delivering alerts, you wouldn't have time or resources to meaningfully create that kind of alert delivery system.

PagerDuty at Compose

At Compose, our business is delivering databases to you which is why we use PagerDuty. Our older Nagios and newer Sensu server monitoring systems both report into PagerDuty to report on the overall state of the servers. We then use our own Compose plugin to monitor our production systems for high lock and stepdown events turning them into alerts too. We have a premium support offering and we use PagerDuty to ensure rapid responses to it's 911 contact points. Our 911 support email are picked up by PagerDuty's email hooks while the 911 calls pass through Twilio into PagerDuty turning those calls into alerts too.

With the alerts collected, collated and deduplicated at PagerDuty, we then use its rotation management to handle two simultaneous overlapping rotations of support staff. The lead rotation is the primary contact and the second is a backup contact. The scheduling overlaps and where there's jobs which are best done by two people, we can bring the primary and secondary in on the job.

We then add to that mix the ops team to act as an extra backup. Finally, ensure that scheduled maintenance doesn't unnecessarily alert the people on call - no one likes being woken up or disturbed at dinner only to find out everything is fine. We have scripts we invoke through hubot that take hosts and services down and ensure that alerts from those systems are picked up in Nagios and Sensu and not forwarded on to PagerDuty.

Be Alerted

"PagerDuty has become an indispensible part of Compose operations. We used to rely on manually checking multiple systems and using a calendar to work out the on-call rotation" says Ben Wyrosdick, one of the Founders of Compose. "Now, we are more effective, not just at getting alerts in a timely fashion, but also internally it helps up spread the load of delivering high quality support. We wouldn't be without it now."

Whatever your business is, you want to focus on that. If you use Compose then you're already using us to take the task of database management off your to-do list. Using PagerDuty to monitor your applications will let you focus on your business even more. You can even use the Compose integration to get alerts from your Compose hosted databases as part of your complete alert management system.