Verifying post-deploy connections with conn-check
1v$ Wed Nov 19 12:14:38 GMT 2014
Deployments of a service have a number of different network dependencies that require verification:
- Connections between services (e.g. app <-> postgresql, are ports unblocked at the firewall(s)? If talking to a data centre instance do we have a route?)
- External services (e.g. webservices such as S3)
- Verification that the services on the other end are real (you're actually talking to MongoDB or rabbitmq via AMQP, not just another TCP service running on those ports)
- Verification of authentication
Although many of these can be solved with smoke tests, it's not always immediately obvious that there is a problem, or what the problem is.
conn-check is a tool that started life inside Ubuntu One to verify connections between services and to S3 etc. post-deploy.
During a mini-sprint in Uruguay a few months ago we separated conn-check into it's own own package and open sourced it.
Since then we've been improving it and using it to verify deploys on our services (such as login.ubuntu.com).
conn-check takes a simple YAML config defining a list of checks to perform (udp, tcp, tls, http, amqp, postgres, mongodb, redis, memcache), and performs those checks asynchronously using Twisted's thread pool, and outputs the results in a Nagios check standard output, so conn-check can be run regulary as a Nagios check to continually verify network status between services (and alert on change, e.g. out of band firewall changes).
Automatically generating configs
We have also released a separate package called conn-check-configs which provides tools for automatically generating conn-check YAML configs from a source such as a Django settings module or Juju environment status (we're currently using the Django settings export, with Juju env export being tested in a branch).
You can get conn-check by:
- Installing from PyPI:
pip install conn-check(You can sudo this to get it system installed, but personally I'd put it in a virtualenv or use pipsi.)
- Installing with apt-get from my PPA:
sudo add-apt-repository ppa:wesmason/conn-check sudo apt-get update sudo apt-get install conn-check
- If you use Juju to manage your infrastructure/deployments, then you can use our charm to deploy conn-check for your service (and even add Nagios checks automatically via the nrpe relation).