PythonWise

If it won't be simple, it simply won't be. [source code] by Miki Tebeka, CEO, 353Solutions

Tuesday, February 24, 2015

Adding Server SSL Certificate on Linux

Here's a small script to add a server SSL certificate on Linux. You can export the certificate from your browser. Inspired by this stackoverflow post.

Tuesday, February 03, 2015

Logging from Celery to logstash and a structured log (JSON)

I love Celery, and we're using it at one of my customers. One thing we wanted to have is centralized logging, since you can have multiple workers on multiple machines. We looked at several solutions and the winner came out to be logstash + kibana (AKA ELK stack).

Here's some code to log to logstash with Celery current task information (if available) and also to a structured log (every line is a JSON object) for backup in case of network issues.

Tuesday, January 27, 2015

Using supervisord to Manage You Daemons

Say you have some daemons running. You'd like to restart them automatically if they fail, grab logs from them and in general manage them - supervisord will do it all for you.

One of my (super cool) clients needed also to start/stop daemons when configuration changes. The solution was to have a script that updates supervisord.conf every time we have a configuration change and then selectively start/stop on the daemons that have change (by default if your run supervisorctl update, it will restart all the daemons).

For this example, I'll assume that the daemon processes are python -m SimpleHTTPServer and I have a list of port I'd like to listen on. This list of ports might change.

Example Usage

$ ./updated.py 8000 8001 8002
$ supervisorctl status
httpd-8000                       RUNNING   pid 31768, uptime 0:00:04
httpd-8001                       RUNNING   pid 31767, uptime 0:00:04
httpd-8002                       RUNNING   pid 31766, uptime 0:00:04
$ ./updated.py 8000 8004 8002 # Remove 8001, add 8004
httpd-8001: disappeared
httpd-8004: available
httpd-8001: stopped
httpd-8001: removed process group
httpd-8004: added process group
$ supervisorctl status
httpd-8000                       RUNNING   pid 31768, uptime 0:00:12
httpd-8002                       RUNNING   pid 31766, uptime 0:00:12
httpd-8004                       RUNNING   pid 31785, uptime 0:00:02
$

Friday, January 09, 2015

python -m

python -m lets you run modules as scripts. If your module is just one .py file it'll be executed (which usually means code under if __name__ == '__main__'). If your module is a directory, Python will look for __main__.py (next to __init__.py) and will run it.

One of Python's mottoes is "batteries included", and this goes for python -m as well. Here are some (all?) of the gems hidden in the standard library. Sadly not all of them have help, but I poked around in the source code to see the usage.

json.tool

This is by far the one I use most, it'll indent nicely an JSON input in the standard output and very helpful in combination with curl.
$ curl -sL http://j.mp/1IuxaLD
[{"x":1,"y":2},{"x":3,"y":4},{"x":5,"y":6}]
$ curl -sL http://j.mp/1IuxaLD | python -m json.tool
[
    {
        "x": 1,
        "y": 2
    },
    {
        "x": 3,
        "y": 4
    },
    {
        "x": 5,
        "y": 6
    }
]

zipfile

zipfile will let you view, extract and create zip files - very much like the zip and unzip. Here's the help:
$ python -m zipfile -h
Usage:
    zipfile.py -l zipfile.zip        # Show listing of a zipfile
    zipfile.py -t zipfile.zip        # Test if a zipfile is valid
    zipfile.py -e zipfile.zip target # Extract zipfile into target dir
    zipfile.py -c zipfile.zip src ... # Create zipfile from sources

gzip

Like zipfile, let's you compress and decompress .gz files, like gzip/gunzip. By default it'll compress a file but with -d will decompress.
python -m gzip wordlist.txt  # Will create wordlist.txt.gz
python -m gzip -d wordlist.txt.gz  # Will extract to wordlist.txt

filecmp

Compare two directories.
$ python -m filecmp /tmp/a /tmp/b
diff /tmp/a /tmp/b
Only in /tmp/a : ['1']
Only in /tmp/b : ['2']
Identical files : ['4']
Differing files : ['3']

Encode/Decode

Several modules lets you encode/decode in various formats:
  • base64
  • uu
  • encodings.rot_13
  • binhex
  • mimify
  • quopri
For example
$ echo 'secertpassword' | python -m encodings.rot_13
frpregcnffjbeq

Servers

There are several servers that you can run, the ones I know are SimpleHTTPServer, CGIHTTPServer and smtpd (mail). If you quickly want to serve some files from a directory on your machine, just run:
python -m SimpleHTTPServer

Clients

Modules that provide simple clients to various protocols are:
  • ftplib
  • poplib
  • nttpserver
  • smtplib (on localhost only)
  • telnetlib
For example if you want to view Star Wars in text mode, do
$ python -m telnetlib towel.blinkenlights.nl

System Info

You can use platform to get some platform information (very much line uname -a) and locale to get locale information. Use mimetype to get the mime type of a file:
$ python -m mimetypes doc.html
type: text/html encoding: None

Python Utilties

  • compileall will compile all Python files to .pyc
  • dis will show bytecode for a file
  • pdb will start the Python debugger on a given file (see here)
  • pydoc will show documentation on a module/class/function
  • site will print some site information (sys.path, USER_BASE ...)
  • sysconfig will show many system related information (such as exec_prefix)
  • tabnanny will tell you of you mix tabs and spaces (like starting python with -t or -tt)
  • tokenize will print list of tokens in Python file
I mostly use pdb and pydoc, for example:
$ python -m pydoc os.remove
Help on built-in function remove in os:

os.remove = remove(...)
    remove(path)
    
    Remove a file (same as unlink(path)).

Profiling

There are several profiles and timers you can use from the command line:
  • cProfile - Show profile information
  • profile (use cProfile :)
  • timeit - Time how long things run
  • pstats - Print output of profiles
  • trace - Show tracing information on run
Example:
$ python -m cProfile match.py
         28537 function calls (27503 primitive calls) in 0.057 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 :1()
        1    0.000    0.000    0.000    0.000 :1(ArgInfo)
        1    0.000    0.000    0.000    0.000 :1(ArgSpec)
   ...

$ pyton -m timeit 'import math; math.factorial(100)'
100000 loops, best of 3: 12.9 usec per loop

timeit has good help from the command line.

IDLE

You can start IDLE by running python -m idlelib.idle

ensurepip

Python 2.7.9 and 3.x comes with an easy way to install pip. Run python -m ensurepip and pypi is at your service.


That's about it ... What are you favorite python -m tools? Which ones did I miss?

EDIT: The good folks at comp.lang.python reminded me a few I forgot:

unittest

python -m unittest discover will run unittest in discovery mode. Just drop a new Python file starting with test and it'll be picked up next time you run the tests. You can also specify a specific test to run with python -m unittest test_file.py TestClass.test_method.

calendar

python -m calendar will show calendar of the current year. You can also run python -m calendar YEAR to display a specific year and python -m calendar YEAR MONTH to display a specific month.

Easter Eggs

python -m this will display the Python Zen
python -m antigravity will open XKCD comic web page (which my company is named after).

Saturday, December 20, 2014

Quick View of matplotlib Styles

matploblit 1.4.2 added support for styles. I created a notebook to show how the ones bundled in look, you can view it here.

Wednesday, December 17, 2014

Calculate Distance in .kmz File

Here's a little script that calculates the distance of a path (series of GPS points) in a .kmz file (which I generate with My Tracks). It combines several elements - Zip file, XML (with namespaces), zip, sum and of course a bit of math.

Sunday, December 07, 2014

The Versatile date Command

*nix systems comes with a date command line utility. Many people use it to view the current time, however there's much more that date can do.

For example we run a daily job to process yesterday data. The job is a Python script that get the date as a paramter in YYYYMMDD format. This translates to one cron line:

@daily /path/to/job.py $(date --date=yesterday +%Y%m%d)

Here are some things you can do with date:


As a bonus, there is also a cal command which displays calender. Note that years are 4 digits so cal 4 14 will diaplay April for the year 14, not 2014.

cal has a handy -w switch that shows the work week as well.

Blog Archive