Compose, Syslog-NG and your logs

If you want your logs delivered to a cloud service, you'll be pleased to know that we have a new add-on which lets you do just that. The syslog-ng add-on is available for Elasticsearch, Redis, PostgreSQL and RethinkDB currently and will let you send your logs to the Loggly or Papertrail services.

In other add-on news you can get our New Relic add-on which is now available on more databases. Redis, PostgreSQL, Elasticsearch and MongoDB+ (our currently in-beta next generation MongoDB deployment system) can now all make use of the New Relic add-on. If you're only interested in alerts and not logs, then New Relic is most likely the option for you.

The New Relic add-on costs $4.50 per month per deployment, and the new syslog-ng add-on currently costs $9 per month per deployment.

Why logging services?

If you want a deeper understanding of what is going on with your systems, you can't beat reading the logs. Logs are a rich source of information about system and application activity, but then we work in a world where we have many systems, on premises or hosted, and getting around all of them to dig through the logs is not an efficient use of anyone's time.

Enter the cloud logging services. There are services that, if you configure your systems to also send a copy of their logs to them, will aggregate all the logs you send through so you can search and correlate and track down whatever is on your mind. They are great at bringing together logs from disparate systems and getting them into an analyzable form and giving you the tools to do just that.

Enabling Syslog-ng

Do you have a cloud logging service account? No? Get yourself an account with Loggly or Papertrail. Both do free services – Loggly does a trial while Papertrail does a free but constrained account. If you're exploring the potential of log services, the Papertrail offering is ideal. We'll be using that as our example here. Go to Papertrailapp.com where you'll see this screen:

Enter your email and a new password and you'll create a free account. You'll be welcomed with a tour guide page:

We can make use of the quick start and just click Add your first system. Papertrail will then display instructions on how to configure various systems to log to it. But we don't need that information to connect Compose's syslog-ng; we just need the host and port at the top of the page:

Make a note of the host and port. If you already have a cloud logging platform, get the host and port number for your service. If you are using Loggly, you'll also need your Loggly UID.

To set up Syslog-ng logging on your database go to the Compose dashboard at app.compose.io and select the deployment you want to configure logging on. That can, as we mentioned, be any Elasticsearch, Redis, PostgreSQL or RethinkDB deployment. From the bottom of the side-bar, select the Add-ons view.

For our listed databases, there'll currently be two panels on this view, one for setting up New Relic monitoring and a second one for configuring Syslog-ng. It's the second panel we're interested in:

Select the service you'll be using in the drop down and enter the hostname, port and, if using Loggly, UID number into the available fields. Before you press Add Syslog-NG bear in mind the warning that when you add the Syslog-NG capsule, there will be a rolling restart of the database to enable the logging to be installed. Make sure your application will not be affected by this restart, either by scheduling away from busy periods or by using database calls which can handle rolling restarts.

When you are ready, click Add Syslog-NG and a new capsule will be provisioned to forward all logs to your selected provider.

If you return to your log service provider, you should now be seeing all logs for that database coming through. Depending on the service, you'll be able to do different things with those logs. For example, as well as interactive real-time search and filtering, Papertrail, lets you create and save searches on one or more of the incoming logs and use those searches to create alerts. Alerts can, in turn, generate email, Slack/HipChat or Campfire messages, pass data to services like Boundary for correlation, PagerDuty for notification or GeckoBoard, Stathat or Librato for graphing and forecasting. And if that isn't enough, it can call on webhooks too to trigger even more events.

Your logging service can take logs from all sorts of services too, not just Compose, so you can use it to get a more holistic view of your complete cloud and on-premises stack if you wish. At Compose, we like to enable our customers to get exactly the amount of monitoring they want and need and the Syslog-NG add-on is that latest addition to that tradition.