Using supervisord to Manage You Daemons

Say you have some daemons running. You''d like to restart them automatically if they fail, grab logs from them and in general manage them - supervisord will do it all for you.

One of my (super cool) clients needed also to start/stop daemons when configuration changes. The solution was to have a script that updates supervisord.conf every time we have a configuration change and then selectively start/stop on the daemons that have change (by default if your run supervisorctl update, it will restart all the daemons).

For this example, I''ll assume that the daemon processes are python -m SimpleHTTPServer and I have a list of port I''d like to listen on. This list of ports might change.

Example Usage

    $ ./updated.py 8000 8001 8002

    $ supervisorctl status

    httpd-8000                       RUNNING   pid 31768, uptime 0:00:04

    httpd-8001                       RUNNING   pid 31767, uptime 0:00:04

    httpd-8002                       RUNNING   pid 31766, uptime 0:00:04

    $ ./updated.py 8000 8004 8002 # Remove 8001, add 8004

    httpd-8001: disappeared

    httpd-8004: available

    httpd-8001: stopped

    httpd-8001: removed process group

    httpd-8004: added process group

    $ supervisorctl status

    httpd-8000                       RUNNING   pid 31768, uptime 0:00:12

    httpd-8002                       RUNNING   pid 31766, uptime 0:00:12

    httpd-8004                       RUNNING   pid 31785, uptime 0:00:02

    $

    

python -m

python -m lets you run modules as scripts. If your module is just one .py file it'll be executed (which usually means code under if __name__ == '__main__'). If your module is a directory, Python will look for __main__.py (next to __init__.py) and will run it.

One of Python's mottoes is "batteries included", and this goes for python -m as well. Here are some (all?) of the gems hidden in the standard library. Sadly not all of them have help, but I poked around in the source code to see the usage.

json.tool

This is by far the one I use most, it'll indent nicely an JSON input in the standard output and very helpful in combination with curl.

$ curl -sL http://j.mp/1IuxaLD

[{"x":1,"y":2},{"x":3,"y":4},{"x":5,"y":6}]

$ curl -sL http://j.mp/1IuxaLD | python -m json.tool

[

    {

        "x": 1,

        "y": 2

    },

    {

        "x": 3,

        "y": 4

    },

    {

        "x": 5,

        "y": 6

    }

]

zipfile

zipfile will let you view, extract and create zip files - very much like the zip and unzip. Here's the help:

$ python -m zipfile -h

Usage:

    zipfile.py -l zipfile.zip        # Show listing of a zipfile

    zipfile.py -t zipfile.zip        # Test if a zipfile is valid

    zipfile.py -e zipfile.zip target # Extract zipfile into target dir

    zipfile.py -c zipfile.zip src ... # Create zipfile from sources

gzip

Like zipfile, let's you compress and decompress .gz files, like gzip/gunzip. By default it'll compress a file but with -d will decompress.

python -m gzip wordlist.txt  # Will create wordlist.txt.gz

python -m gzip -d wordlist.txt.gz  # Will extract to wordlist.txt

filecmp

Compare two directories.

$ python -m filecmp /tmp/a /tmp/b

diff /tmp/a /tmp/b

Only in /tmp/a : ['1']

Only in /tmp/b : ['2']

Identical files : ['4']

Differing files : ['3']

Encode/Decode

Several modules lets you encode/decode in various formats:

• base64
• uu
• encodings.rot_13
• binhex
• mimify
• quopri

For example

$ echo 'secertpassword' | python -m encodings.rot_13

frpregcnffjbeq

Servers

There are several servers that you can run, the ones I know are SimpleHTTPServer, CGIHTTPServer and smtpd (mail). If you quickly want to serve some files from a directory on your machine, just run:

python -m SimpleHTTPServer

Clients

Modules that provide simple clients to various protocols are:

• ftplib
• poplib
• nntplib
• smtplib (on localhost only)
• telnetlib

For example if you want to view Star Wars in text mode, do

$ python -m telnetlib towel.blinkenlights.nl

System Info

You can use platform to get some platform information (very much line uname -a) and locale to get locale information. Use mimetype to get the mime type of a file:

$ python -m mimetypes doc.html

type: text/html encoding: None

Python Utilties

• compileall will compile all Python files to .pyc
• dis will show bytecode for a file
• pdb will start the Python debugger on a given file (see here)
• pydoc will show documentation on a module/class/function
• site will print some site information (sys.path, USER_BASE ...)
• sysconfig will show many system related information (such as exec_prefix)
• tabnanny will tell you of you mix tabs and spaces (like starting python with -t or -tt)
• tokenize will print list of tokens in Python file

I mostly use pdb and pydoc, for example:

$ python -m pydoc os.remove

Help on built-in function remove in os:

os.remove = remove(...)

    remove(path)

    

    Remove a file (same as unlink(path)).

Profiling

There are several profiles and timers you can use from the command line:

• cProfile - Show profile information
• profile (use cProfile :)
• timeit - Time how long things run
• pstats - Print output of profiles
• trace - Show tracing information on run

Example:

$ python -m cProfile match.py

         28537 function calls (27503 primitive calls) in 0.057 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)

        1    0.000    0.000    0.000    0.000 :1()

        1    0.000    0.000    0.000    0.000 :1(ArgInfo)

        1    0.000    0.000    0.000    0.000 :1(ArgSpec)

   ...

$ pyton -m timeit 'import math; math.factorial(100)'

100000 loops, best of 3: 12.9 usec per loop

timeit has good help from the command line.

IDLE

You can start IDLE by running python -m idlelib.idle

ensurepip

Python 2.7.9 and 3.x comes with an easy way to install pip. Run python -m ensurepip and pypi is at your service.


That's about it ... What are you favorite python -m tools? Which ones did I miss?

EDIT: The good folks at comp.lang.python reminded me a few I forgot:

unittest

python -m unittest discover will run unittest in discovery mode. Just drop a new Python file starting with test and it'll be picked up next time you run the tests. You can also specify a specific test to run with python -m unittest test_file.py TestClass.test_method.

calendar

python -m calendar will show calendar of the current year. You can also run python -m calendar YEAR to display a specific year and python -m calendar YEAR MONTH to display a specific month.

Easter Eggs

python -m this will display the Python Zen

python -m antigravity will open XKCD comic web page (which my company is named after).

Quick View of matplotlib Styles

matploblit 1.4.2 added support for styles. I created a notebook to show how the ones bundled in look, you can view it here.

Calculate Distance in .kmz File

Here's a little script that calculates the distance of a path (series of GPS points) in a .kmz file (which I generate with My Tracks). It combines several elements - Zip file, XML (with namespaces), zip, sum and of course a bit of math.

The Versatile date Command

*nix systems comes with a date command line utility. Many people use it to view the current time, however there''s much more that date can do.

For example we run a daily job to process yesterday data. The job is a Python script that get the date as a paramter in YYYYMMDD format. This translates to one cron line:

@daily /path/to/job.py $(date --date=yesterday +%Y%m%d)

Here are some things you can do with date:


As a bonus, there is also a cal command which displays calender. Note that years are 4 digits so cal 4 14 will diaplay April for the year 14, not 2014.

cal has a handy -w switch that shows the work week as well.'

Generate QR Code Using Google Charts API

Here's a small utility to generate QR code image using Google Charts API.

Note the hand crafted Python 2/3 support, for more advanced stuff you might want to have a look at six. However for this script I wanted to stay without external dependencies.

Common Errors

"Experience is the name every one gives to their mistakes."

    - Oscar Wilde

Students are sometimes amazed by how I can tell the cause of exception without looking at the code. This is the result of making many errors in Python myself and also from observing many students doing mistakes. Here's a short list I've compiled of most likely cause of errors:

NameError

• You forgot to import a module
• You made a typo

AttributeError: 'NoneType' object has no attribute ...

• You forgot a return in your function

AttributeError

• Typo on dot lookup (obj.foo)
• Object is different type from what you think (str vs int)
• Object does not implement a dunder method (e.g. __len__)

There's also the 3'rd party didyoumean module, which might be interesting for beginners. It changes the default stack trace to add a hint about what might be the problem.

A Streaming Chart using Flask and flot

I was teaching a course on "Python Analytics" (pandas, scikit-learn, matplotlib ...) and was asked to provide an example of streaming chart - ones that updates periodically. I''ve showed a couple of examples one by generating image using matplotlib and another with bokeh-server. After a couple of days I remembered another way - using flot to render the chart. Here''s a small example using Flask as the web server (the code works both on Python 2 and 3).


Few comments:

• Don''t use debug=True in production :)
• For simplicity everything is in one file. However for larger application you might want to take the HTML template(s) out
• Data is in memory, a restart will wipe it out. If you need data persistence - pick a database (shelve, sqlite3 ...)
• A big shoutout to Continuum Analytics - Anaconda (and conda) has made my life so much easier teaching this workshop.

Resolve SSH Host Name

~/.ssh/config let''s you give meanigful names to hosts/ips. But sometimes you want the reverse lookup - what''s the ip of web1? Here''s a little Python script that does that.

EDIT: EAFP > LBYL

Archlinux Install Steps (on VirtualBox)

My favorite Linux distro for using under VirtualBox is Archlinux with XFCE window manager. It''s light, fast and has all the latest shiny new toys (just the way I like it :).

I found myself setting up VMs to try things out and wrote down the steps I use, this is a trimmed down version of the Installation part in the Archlinux beginners guide.

Be a Better Developer by Coding in Four Different Types of Lauguages

I like programming languages and find out that every time I learn a new language it improves my coding in the other ones as well. I learn new ways of doing things, different ways of thinking and it's great.

I usually tell new developers they need to write a (small) project in at least four types of languages - mainstream (procedural/OO), functional, logic based and assembly. Each of these types will give you a different way of solving problems and enrich your programming experience by and order of magnitude.

Here are my recommendations for each category.

Main Stream

By "main stream" I mean procedural/OO languages. There are tons of these and its up to what you're working with currently (though it might be a good excuse to learn a new language). I'm a Python expert, but pick anything - Go, JavaScript, Ruby, C, C++, Java, C# ...

(Yeah - I know they differ a lot. But thinking in most of them is probably the same. The main difference will probably be dynamic vs static typing).

Functional

Many choices here as well. Personally I like the Lisp family of languages, mostly Clojure and Scheme but you can check out a Common Lisp implementation (I think SBCL leads the pack currently), Haskell, ML and others.

Logic Based

If you haven't done logic programming - it'll blow your mind! It's a totally different way of thinking. Prolog is the main language, one free implementation is SWI but there are others as well.

Assembly

Learning assembly will give you a better understanding on how computers work and what are the abstractions other programming languages do for you. I recommend picking one that targets the machine you're working on.

Add That Trailing Comma

Lets say you wrote this simple code and at first things were going well.

Then after a while a friend came in and did a little fix.

But things started falling apart, after a lot of digging in - you find this.

Python will join two string together in this case, not what you wanted. Always leave a trailing comma.

Couple more things:

• As Dave Cheney pointed out, using this practice has the nice effect that one line change shows as one line change in the diff since you don''t have to add a comma to the previous line.
• Go probably learned from Python and made trailing comma mandatory.
• You can see more Python "gotchas" here.

draft2gist - Publish draftin.com Documents to gist

I've started playing with draftin.com, so far very nice.

draftin.com lets you publish documents to several sites, and if the site you want to publish to is not on the list - there are WebHooks.

I've written a small AppEngine service that is a WebHook for publishing draftin.com documents to gist. Feel free to use it and let me know if you find any errors.

Below is the server code, rest of the files are here.

HTTP Proxy Stripping Headers (go)

At one of my clients, we wanted to write an HTTP proxy that strips some of the headers sent from the target (backend). With the help of golang-nuts mailing list - this turned out to be pretty simple.

timeat on pypi

timeat is now on pypi. There's some extra code to get current time from NTP server. Should work both on Python 2.x and 3.x

timeat in Go

Wrote timeat, which shows time at a specific location, in Go as well. (For comparison the Python version is here). To install either go get bitbucket.org/tebeka/timeat or download the executables.

Working in a multi-timezone team, this script comes handy from time to time (as well as worldtimebuddy :).

Generating Byte Arrays for Assets in Go using xxd

Go's net/http server is pretty fast, but sometimes you want to get faster. One way to do that is to create a binary array in memory for static files (assets). Here''s how I generate the byte arrays automatically with xxd.

Note that this makes the build go slower as you have more and bigger assets.If this is a problem, take a look at nrsc ;)

Since the go toolchain does not support custom steps currently, I''m using make.

Makefile

httpd.go

Hook to Update Tag List when Changing git Branch

I use ctags with Vim to move around. At work we use git feature branches, which means code changes when you switch a branch. Here''s a very simple post-checkout hook to update the tag list whenever you switch branches.

Use dict to Speed Up Your Code

In Python, dictionary access is very fast. You can use that to get some speedup in your code  by replacing if/else with dictionary get.

In [1]: fn1 = lambda x: 1 if x == 't' else 0

In [2]: fn2 = {'t': 1, 'f': 0}.get

In [3]: %timeit fn1('y')  # Check "True" branch
10000000 loops, best of 3: 124 ns per loop

In [4]: %timeit fn2('y')
10000000 loops, best of 3: 79.6 ns per loop

In [5]: %timeit fn1('f')  # Check "False" branch
10000000 loops, best of 3: 125 ns per loop

In [6]: %timeit fn2('f')
10000000 loops, best of 3: 81.3 ns per loop

About 30% speedup - not bad :)

HTTPDir - Small OSX Utility to Serve a Directory over HTTP

HTTPDir is a small utility that lets you serve content of a directory over HTTP.

This is handy when you develop static sites that has reference to external resources. It is also aimed to people who are not comfortable with the command line.

HTTPDir is a simple Python script that uses Tkinter. It is packed in a format that OSX recognizes as an application. See the code here (look under HTTPDir.app/Contents/MacOS).