« Snappy and "ImportError: No module named 'click.repository'" Canonical »

Verifying post-deploy connections with conn-check

1v$ date
1v$ Wed Nov 19 12:14:38 GMT 2014

The Problem

Deployments of a service have a number of different network dependencies that require verification:

  • Connections between services (e.g. app <-> postgresql, are ports unblocked at the firewall(s)? If talking to a data centre instance do we have a route?)
  • External services (e.g. webservices such as S3)
  • Verification that the services on the other end are real (you're actually talking to MongoDB or rabbitmq via AMQP, not just another TCP service running on those ports)
  • Verification of authentication

Although many of these can be solved with smoke tests, it's not always immediately obvious that there is a problem, or what the problem is.

Our solution

conn-check is a tool that started life inside Ubuntu One to verify connections between services and to S3 etc. post-deploy.
During a mini-sprint in Uruguay a few months ago we separated conn-check into it's own own package and open sourced it.
Since then we've been improving it and using it to verify deploys on our services (such as login.ubuntu.com).

conn-check takes a simple YAML config defining a list of checks to perform (udp, tcp, tls, http, amqp, postgres, mongodb, redis, memcache), and performs those checks asynchronously using Twisted's thread pool, and outputs the results in a Nagios check standard output, so conn-check can be run regulary as a Nagios check to continually verify network status between services (and alert on change, e.g. out of band firewall changes).

Automatically generating configs

We have also released a separate package called conn-check-configs which provides tools for automatically generating conn-check YAML configs from a source such as a Django settings module or Juju environment status (we're currently using the Django settings export, with Juju env export being tested in a branch).

Getting conn-check

You can get conn-check by:

  • Installing from PyPI:
    pip install conn-check
    (You can sudo this to get it system installed, but personally I'd put it in a virtualenv or use pipsi.)
  • Installing with apt-get from my PPA:
    sudo add-apt-repository ppa:wesmason/conn-check
    sudo apt-get update
    sudo apt-get install conn-check
  • If you use Juju to manage your infrastructure/deployments, then you can use our charm to deploy conn-check for your service (and even add Nagios checks automatically via the nrpe relation).

blog comments powered by Disqus