The Problem

Deployments of a service have a number of different network dependencies that require verification:

• Connections between services (e.g. app <-> postgresql, are ports unblocked at the firewall(s)? If talking to a data centre instance do we have a route?)
• External services (e.g. webservices such as S3)
• Verification that the services on the other end are real (you're actually talking to MongoDB or rabbitmq via AMQP, not just another TCP service running on those ports)
• Verification of authentication

Although many of these can be solved with smoke tests, it's not always immediately obvious that there is a problem, or what the problem is.

Our solution

conn-check is a tool that started life inside Ubuntu One to verify connections between services and to S3 etc. post-deploy.
During a mini-sprint in Uruguay a few months ago we separated conn-check into it's own own package and open sourced it.
Since then we've been improving it and using it to verify deploys on our services (such as login.ubuntu.com).

conn-check takes a simple YAML config defining a list of checks to perform (udp, tcp, tls, http, amqp, postgres, mongodb, redis, memcache), and performs those checks asynchronously using Twisted's thread pool, and outputs the results in a Nagios check standard output, so conn-check can be run regulary as a Nagios check to continually verify network status between services (and alert on change, e.g. out of band firewall changes).

Automatically generating configs

We have also released a separate package called conn-check-configs which provides tools for automatically generating conn-check YAML configs from a source such as a Django settings module or Juju environment status (we're currently using the Django settings export, with Juju env export being tested in a branch).

Getting conn-check

You can get conn-check by:

• Installing from PyPI:

  pip install conn-check

  (You can sudo this to get it system installed, but personally I'd put it in a virtualenv or use pipsi.)
• Installing with apt-get from my PPA:
  

  sudo add-apt-repository ppa:wesmason/conn-check
  sudo apt-get update
  sudo apt-get install conn-check
  

• If you use Juju to manage your infrastructure/deployments, then you can use our charm to deploy conn-check for your service (and even add Nagios checks automatically via the nrpe relation).