Snappy and "ImportError: No module named 'click.repository'"

1v$ date
1v$ Mon Dec 15 23:26:08 GMT 2014

If you're playing around with Snappy and Ubuntu Core, the new transaction based packaging platform in town, and receive the following error:

ImportError: No module named 'click.repository'

It's most likely because you also have the ubuntu-sdk PPA installed, and a new version of python3-click has been pushed up that doesn't include the repository submodule that Snappy uses. I found unravelling the two project's dependencies to be non-trivial at the moment, so here's an alternative quick fix:

  1. sudo apt purge -y python3-click
  2. Create a new file at /etc/apt/preferences.d/snappy-ppa-1500, with these contents (telling apt that we prefer duplicate packages from the snappy-dev PPA).
  3. sudo apt update
  4. sudo apt install -y --force-yes snappy-tools

For the time being this should select the packages we need from the snappy-dev PPA, and giving us the right python3-click, complete with repository submodule.

The other alternative is to confine your Snappy experiments to an LXC or VM; or completely purge both PPAs and all the packages from them and reinstall just Snappy if you don't care about Ubuntu Touch or regular click package dev right now.
I'm sure the conflict will be fixed soon though. :-)

Verifying post-deploy connections with conn-check

1v$ date
1v$ Wed Nov 19 12:14:38 GMT 2014

The Problem

Deployments of a service have a number of different network dependencies that require verification:

  • Connections between services (e.g. app <-> postgresql, are ports unblocked at the firewall(s)? If talking to a data centre instance do we have a route?)
  • External services (e.g. webservices such as S3)
  • Verification that the services on the other end are real (you're actually talking to MongoDB or rabbitmq via AMQP, not just another TCP service running on those ports)
  • Verification of authentication

Although many of these can be solved with smoke tests, it's not always immediately obvious that there is a problem, or what the problem is.

Our solution

conn-check is a tool that started life inside Ubuntu One to verify connections between services and to S3 etc. post-deploy.
During a mini-sprint in Uruguay a few months ago we separated conn-check into it's own own package and open sourced it.
Since then we've been improving it and using it to verify deploys on our services (such as login.ubuntu.com).

conn-check takes a simple YAML config defining a list of checks to perform (udp, tcp, tls, http, amqp, postgres, mongodb, redis, memcache), and performs those checks asynchronously using Twisted's thread pool, and outputs the results in a Nagios check standard output, so conn-check can be run regulary as a Nagios check to continually verify network status between services (and alert on change, e.g. out of band firewall changes).

Automatically generating configs

We have also released a separate package called conn-check-configs which provides tools for automatically generating conn-check YAML configs from a source such as a Django settings module or Juju environment status (we're currently using the Django settings export, with Juju env export being tested in a branch).

Getting conn-check

You can get conn-check by:

  • Installing from PyPI:
    pip install conn-check
    (You can sudo this to get it system installed, but personally I'd put it in a virtualenv or use pipsi.)
  • Installing with apt-get from my PPA:
    sudo add-apt-repository ppa:wesmason/conn-check
    sudo apt-get update
    sudo apt-get install conn-check
    
  • If you use Juju to manage your infrastructure/deployments, then you can use our charm to deploy conn-check for your service (and even add Nagios checks automatically via the nrpe relation).

Canonical

1v$ date
1v$ Sat Nov 8 0:47:50 GMT 2014

Just thought I'd mention that I switched jobs way back in April and I now work for Canonical (yes, that one, the producers of Ubuntu Linux), in Online Services, and couldn't be happier.

Expect to hear soon about some of the things I've been working on.

Speaking schedule

1v$ date
1v$ Wed Aug 21 10:55:26 GMT 2013

May speaking

1v$ date
1v$ Wed May 1 15:21:03 GMT 2013

Sockii

1v$ date
1v$ Wed May 1 15:01:12 GMT 2013

Just a note to say that yesterday I released Sockii, a smart HTTP and WebSocket aggregator, and blogged about it over on the Server Density blog.

PHP Weekly

1v$ date
1v$ Wed Nov 28 7:33:30 GMT 2012

From the 6th of December I'm going to be trying a little experiment. I'm going to run a weekly newsletter filled with links to articles, videos, project and dev news from the world of PHP called PHP Weekly (yes much in the style of Peter Cooper and Gareth Rushgrove).

If you fancy following along head on over and subscribe, or follow me on twitter where I will also post links to freshly archived mailshots.

hippybot - Hipchat bot written in Python

1v$ date
1v$ Tue Sep 13 15:17:08 GMT 2011

I recently had a reason to try out Hipchat, being a bit of a "chat enthusiast" (I was an IRC network admin for about 7 years and have a been developer on a few web chat and IM systems) I found Hipchat to be one of the nicest (that I've used) of the recent stream of "IRC for the web" products out there (for example 37 Signal's Campfire).

Normally this would have been the end of my quick perusal, until the next time I had reason to the use the service, but I noticed they had not only an API, but also Jabber/XMPP exposure for creating bots. Now my interest was piqued.
Looking at the existing third-party bots out there I found you had your choice of languages…as long as that choice was Ruby or Javascript. ;-)

Now I'm not averse to developing in either, but they're not my preferred languages by far (okay Javascript has a soft spot in my heart but I still haven't made the jump to server-side JS dev).

So with that I present hippybot
(Hipchat Python Bot, geddit?)
Not to be confused with Hippy, a PHP library for the Hipchat API.

I wrote hippybot using the excellent python-jabberbot, a library that utilises the equally brilliant xmpppy to allow the creation of bots in an easy to understand and pythonic way (e.g. inherit from base class, set some connection strings, override event methods if need be, use decorators to extend commands the bot responds to).

How is this different from a regular Jabber bot? I've written some Hipchat-specific customisations into the bot:

  • The connection config assumes Hipchat's XMPP server, and various details such as the username and channel names have Hipchat specific [pre/suf]fixes added to them automatically.
  • The bot "understands" at-sign mentions (e.g. those popularised by Twitter and supported by Hipchat for directing messages to specific people within rooms, as well as triggering notifications), and commands added by plugins can leverage this.

I've added a simple plugin API to the bot, which consists of either creating a function in a module with a decorator, or a class in a module with as many decorated methods (each a different command to trigger the bot) as you like, and the ability to create aliases for commands and define whether a command is triggered by an at-sign mention or not (e.g. @hippybot do something!).

In the next release I plan to add a simple wrapper to Hipchat's API (already in master), better at-sign mention handling, and some more core plugins.

Feel free to check it out and for an example of the sorts of things you can do with a bot like this just check out some of the things Github's hubot does.

Webcomic hosting survey

1v$ date
1v$ Fri Jun 17 14:59:17 GMT 2011

I very rarely ask people to spread links about or push my wares, but in this instance I would greatly appreciate sharing "this link":http://goo.gl/S9n7P to anyone you might know who creates/publishes comics of any form (web comics, digital comic books etc.) online.

Morgan Robert Mason

1v$ date
1v$ Fri Apr 29 0:06:13 GMT 2011

Morgan Robert Mason, born April 26th 12.07am, 10 lbs exactly, mother and baby well (but exhausted).