"svkpull" and "svkpush"

I've been using svk lately. I mirror our main Subversion development branch and then create my own local branch. When my changes are ready I use svk smerge to integrated my changed back to the main branch.

Since I do this a lot of time, I've create two scripts: svkpull and svkpush that updates my local branch from the main branch and push my changes to the main branch.

svkpull

#!/bin/bash

# Get changes from mirrord subversion repository

# Miki Tebeka 

case $# in
0 ) up=1; project=`basename $PWD`;;
1 ) up=0; project=$1;;
* ) echo "usage: `basename $0` [PROJECT_NAME]"; exit 1;;
esac

echo "Project is $project"

svk info //$project > /dev/null
if [ $? -ne 0 ]; then
echo "error: bad project - $project"
exit 1
fi

svk sync //$project/mirror
if [ $? -ne 0 ]; then
exit $?
fi

svk smerge --incremental -l //$project/mirror //$project/local
if [ $? -ne 0 ]; then
exit $?
fi

if [ $up == 1 ]; then
svk up
extra=""
else
extra="Don't forget to run 'svk up'"
fi

echo
echo "Done. $extra"

svkpush

#!/bin/bash

# Push changes to mirrord subversion repository

# Miki Tebeka 

USAGE="usage: `basename $0` [-m comment] [PROJECT]"
COMMENT=""

while getopts "m:h" opt
do
case $opt in
    m ) COMMENT=$OPTARG;;
    h ) echo $USAGE; exit;;
    * ) echo "error: unrecognized option -$opt"; exit 1;;
esac
done

shift $(($OPTIND - 1))

case $# in
0 ) up=1; project=`basename $PWD`;;
1 ) up=0; project=$1;;
* ) echo $USAGE; exit 1;;
esac

echo "Project is $project"

svk info //$project > /dev/null
if [ $? -ne 0 ]; then
echo "error: bad project - $project"
exit 1
fi

svk sync //$project/mirror
if [ $? -ne 0 ]; then
exit $?
fi

if [ "$COMMENT" == "" ]; then
svk smerge //$project/local //$project/mirror
else
svk smerge -m "$COMMENT" //$project/local //$project/mirror
fi

if [ $? -ne 0 ]; then
exit $?
fi

echo
echo "Done."

Notes:

• If project is X the mirror is in //X/mirror and local branch is in //X/local
  
• I use svk smerge --incremental -l only when getting changes
  

DRY

The DRY (do not repeat yourself) principle is a hard to achieve, but worth the effort.
(This rule is also known as SPOT - Single Point Of Truth).

One good solution I find when some piece of information is very configurable and need to be shard with many different programs is to write a little script that prints the required value.

For example, say we need to decide what is the root directory for something. It can be either the debug version of a standard one.

The script rootdir.py will look something like:

from os import environ
from os.path import join
from getpass import getuser

if "DEBUG" in environ:
   # Every user has his/her own test root directory
   ROOT_DIR = join("/tmp", "testing", getuser())
else:
   ROOT_DIR = "/usr/local/cool"

if __name__ == "__main__":
   print ROOT_DIR

If you're using Python, you can just write from root import ROOT_DIR, however if you're using bash you can write ROOTDIR=`./root.py` and then use $ROOTDIR

Minimizing IDLE Popup Time

Well... I'm back :)

IDLE 1.2 (that comes with Python 2.5) has a nice feature that when you write something like os. and wait a bit, IDLE will pop a menu with all the attributes os has (A.K.A intellisense).

However the default time you have to wait before this helpful menu pops is 2 seconds. I find this a bit too long.

The solution is to create a file call config-extensions.cfg in a directory called .idlerc that is in your home directory (this is where IDLE stores all of its configuration files). Write the following lines into this file:
And you're done! Next time you write os. the menu will pop up almost immediately.

Q. How do I know where is my home directory?
A. OK, you windows user, just type the following in IDLE:

(Yeah, yeah. I'm on a Mac now).

Going Offline For a While

I will be offline for about a month or two...

Have fun in the meanwhile, and do come back.

putpath

Since I am (sadly) on Windows platform, I use cygwin a lot.

One of my most used scripts is putpath which puts the windows path of the first argument to the windows clipboard (if called without any arguments it uses the current directory).

It uses the putclip program that comes with cygwin.

#!/usr/bin/env python
'''Put a path in clipboard'''
__author__ = "Miki Tebeka "

from os import system, popen
from os.path import exists
from optparse import OptionParser

# Command line parsing
p = OptionParser("usage: %prog [options] [PATH]")
p.add_option("-u", help="unix path format", dest="unix",
        default=0, action="store_true")
p.add_option("-c", help="C string format", dest="cstr",
        default=0, action="store_true")

opts, args = p.parse_args()
if len(args) not in (0, 1):
    p.error("wrong number of arguments") # Will exit

if len(args) == 0:
    path = "."
else:
    path = args[0]

if not exists(path):
    raise SystemExit("error: %s - no such file or directory" % path)

# Output format (unix/dos)
if opts.unix:
    format = "u"
else:
    format = "w"

path = popen("cygpath -%sa '%s'" % (format, path)).read().strip()
if opts.cstr and (not opts.unix):
    path = path.replace("\\", "\\\\")
popen("putclip", "w").write("%s" % path)

"today"

When releasing a new version, I like to send an email that starts with:

  Stupid version 0.1.2 is out, grab it from 
  ftp://stupid.com/pub/stupid-0.1.2.tar.bz2 before SOMETHING is over!

Where SOMETHING is an event happening at the time of the post. To find what events happen today, I use the following script (called "today"):

#!/usr/bin/env python
'''Launch wikipedia page for today'''

from time import strftime
import webbrowser

day = strftime("%d")
# Change 06 -> 6
if day.startswith("0"):
   day = day[1]

url = "http://en.wikipedia.org/wiki/" + strftime("%B") + "_" + day
webbrowser.open(url)

unpack

I always have trouble remembering which command opens which archive (.tar.gz, .bz2, .zip ...). This is why I have a handy little utility the called "unpack" that knows by the file extension which command to call:

#!/usr/bin/env python
'''Unpack/show compressed files

Create a symbolic link called "zview" to this file or specify -l to view
'''

__author__ = "Miki Tebeka <miki.tebeka@gmail.com>"

from os import system
from os.path import splitext, isfile
from operator import itemgetter

def _stdout(cmd, filename):
    return "%s '%s' > '%s'" % (cmd, filename, splitext(filename)[0])

def bz(filename):
    return _stdout("bzip2 -d -c", filename)

def gz(filename):
    return _stdout("gunzip -c", filename)

class Archive:
    extensions = {} # Extension -> instance

    def __init__(self, unpack, list, extensions):
        self.unpack = unpack
        self.list = list
        for ext in extensions:
            Archive.extensions[ext] = self

Archive("tar -xzvf", "tar -tzf", [".tar.gz", ".tgz", ".tar.z"]),
Archive("tar -xjvf", "tar -tjf", [".tar.bz", ".tar.bz2"]),
Archive(bz, "", [".bz", ".bz2"]),
Archive("tar -xvf", "tar -tf", [".tar"]),
Archive("unzip", "unzip -l", [".zip", "jar", "egg"]),
Archive("unarj e", "unarj l", [".arj"]),
Archive(gz, "", [".gz", ".Z"]),
Archive("unrar x", "unrar lb", [".rar"]),
Archive("7za x", "7za l", [".7z"])

def find_archive(filename):
    # Find *longest* matching extension
    for ext in sorted(Archive.extensions, reverse=1, key=len):
        if filename.lower().endswith(ext):
            return Archive.extensions[ext]

def main(argv=None):
    if argv is None:
        import sys
        argv = sys.argv

    from optparse import OptionParser
    parser = OptionParser(usage="usage: %prog [options] FILE")
    parser.add_option("-s", "--show", help="just show command, don't run",
        dest="show", action="store_true", default=0)
    parser.add_option("-l", "--list", help="list files in archive", 
        dest="list", action="store_true", default=0)


    opts, args = parser.parse_args(argv[1:])
    if len(args) != 1:
        parser.error("wrong number of arguments") # Will exit

    infile = args[0]
    if (not opts.show) and (not isfile(infile)):
        raise SystemExit("error: can't find %s" % infile)

    archive = find_archive(infile)
    if not archive:
        raise SystemExit("error: don't know how to handle %s" % infile)
    list = opts.list or ("zview" in __file__)
    command = archive.list if list else archive.unpack

    infile =  infile.replace("'", "\\'")

    if callable(command):
        command = command(infile)
    else:
        command = "%s '%s'" % (command, infile)

    if opts.show:
        print command
        raise SystemExit

    raise SystemExit(system(command))

if __name__ == "__main__":
    main()

There is also a similar one for viewing zip content.

ecut

cut is a nice littiel utility that cuts a line to several fields and displays only some of them. It's main drawback is that cut splits to fields according to fixed separator, there are times when you want to split accoring to a regular expression - enters ecut.

ecut let's you split fields by regular expression (defaulting to \s+), it also handles Python's array indexing so that you can write someting like:

ls -l | tail +2 | ecut -f -2 | sort -r | head -1

(which is a weird way of finding the last time someting was modified in the current directory :)

ecut code is simple:

#!/usr/bin/env python
'''cut with regular expressions'''

__author__ = "Miki Tebeka "
__license__ = "BSD"
__version__ = "0.2.0"

from sys import stdin
import re
from optparse import OptionParser

progname = "ecut" # Program name
is_range = re.compile("\d+-\d+").match

class FatalError(SystemExit):
    def __init__(self, message):
        error_message = "%s: error: %s" % (progname, message)
        SystemExit.__init__(self, error_message)

parser = OptionParser("usage: %prog [OPTIONS] [FILE]",
            version="%%prog %s" % __version__)
default_delimiter = r"\s+"
parser.add_option("-d", "--delimiter", dest="delim",
        default=default_delimiter, 
        help="delimiter to use (defaults to '%s')" % default_delimiter)
parser.add_option("-f", "--fields", dest="fields", default=[],
        help="comma seperated list of fields to print", metavar="LIST")
parser.add_option("--output-delimiter", dest="out_delim", default=" ",
        help="output delimiter", metavar="STRING")
parser.add_option("-s", "--only-delimited", dest="only_delim", default=0,
        help="do not print lines not containing delimiters",
        action="store_true")

opts, args = parser.parse_args()
if not opts.fields:
    raise FatalError("no fields given")

# Compile the delimiter
try:
    split = re.compile(opts.delim).split
except re.error, e:
    raise FatalError("bad regular expression (%s)" % e.args[0])

if not args:
    infiles = ["-"]
else:
    infiles = args

# Prase fields, we substract 1 since f1 is the 1'st field
fields = []
for field in opts.fields.split(","):
    try:
        if not is_range(field):
            fields.append(int(field))
        else:
            fs = field.split("-")
            if len(fs) != 2:
                raise ValueError
            if fs[0] and fs[1]:
                fields.append((int(fs[0]) - 1, int(fs[1]) - 1)) # Full range
            elif not fs[0]:
                fields.append((0, int(fs[1]) - 1)) # 0-M
            else: # M-end
                fields.append((int(fs[0]) - 1, -1))
    except ValueError:
        raise FatalError("bad field: %s" % field)

inttype = type(1) # Ingeter type

# Process input files
for file in infiles:
    if file == "-":
        info = stdin
    else:
        try:
            info = open(file)
        except IOError, e:
            raise FatalError("can't open %s - %s" % (file, e.strerror))

    for line in info:
        out = []
        fs = filter(lambda x: x, split(line))
        max = len(fs) - 1

        if opts.only_delim and (len(fs) == 1):
            continue

        for field in fields:
            if (type(field) == inttype): # Simple field
                if field <= max:
                    out.append(fs[field])
            else: # Range
                start = field[0]
                if field[1] == -1:
                    end = max
                else:
                    end = min(field[2], max)
                for i in range(start, end + 1):
                    out.append(fs[i])
        print opts.out_delim.join(out)

GmailTray

Didn't find any good email notification that will support
Gmail for Your Domain so I wrote one.

Very basic, but does exactly what I want. Took me about 2 hours to get the initial version up and running.

Basic design:

• Store visited emails UIDL in pysqlite database (which will be in 2.5 standard library).
  
• Have tray icon change color when there are new messges (I hate popups).
  
• Open speficifed web page to view items
  
• Small configuration screen

All of this was done using:

• poplib for checking mail and getting UIDLs
  
• pysqlite as UIDL database
  
• wxPython for GUI

Source code is only 226 lines of code.

After that just some py2exe and InnoSetup. We now have a cool looking windows application with an installer.

SCons

SCons is a great make replacement written in Python. It offers the following goodies (and many more):

• Cross platform. Same script will compile your sources for Linux or Windows or ...


• Support many tools out of the box (such as gcc/VC/java/tar ...)


• Calculate if a file has change by digital signature
(and not by time stamp)

• Automatically cleans after itself (no more make clean)
  

Scons Makefile is called SConstruct, here is a simple one:

Program("hw", ["hw.c"])

Which tells scons to build an executable called hw from the source file hw.c

Calling scons on Linux will produce:

[15:10] /tmp/hw $scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o hw.o -c hw.c
gcc -o hw hw.o
scons: done building targets.
[15:10]

and on Windows it finds VC and invokes it:

C:\Temp\hw>scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
cl /nologo /c hw.c /Fohw.obj
hw.c
link /nologo /OUT:hw.exe hw.obj
scons: done building targets.
C:\Temp\hw>

Cleaning is simple as well:

[15:19] /tmp/hw $scons -c
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Cleaning targets ...
Removed hw.o
Removed hw
scons: done cleaning targets.
[15:19] /tmp/hw $

All of this in one line of SConstruct.

(Shameless plug: See here for a longer article on the subject)

Better dir

The builtin dir command is very useful. However it does not help that much when an object has many attributes.

I have a little function called la in my $HOME/.pythonrc.py

def la(obj, key=None, ignore_case=1):
    '''List all attributes of object'''
    import re
    if key:
        if ignore_case:
            flags = re.I
        else:
            flags = 0
        func = re.compile(key, flags).search

    else:
        func = None
    print "\n".join(filter(func, dir(obj)))

This prints the attributes of an object one per line and also gives you the ability to search for specific items.

>>> from user import la
>>> import wx
>>> la(wx, "open")
ART_FILE_OPEN
ART_FOLDER_OPEN
EVT_MENU_OPEN
ID_OPEN
OPEN
Process_Open
wxEVT_MENU_OPEN
>>> 

(Sun Aug 27 14:03:24 JDT 2006: Added regular expression to "la")

Script Template

Sorry for the long delay, times are interesting around here.

Since many of my scripts share the same structure, I have a little template that I use for new scripts. I have a makepy command that copies this template to a new script.

#!/usr/bin/env python
''' '''

# Miki Tebeka 

def main(argv=None):
    if argv is None:
        import sys
        argv = sys.argv

    from optparse import OptionParser

    parser = OptionParser("usage: %prog ")

    opts, args = parser.parse_args(argv[1:])
    if len(args) != 1:
        parser.error("wrong number of arguments") # Will exit

if __name__ == "__main__":
    main()

As you can see, there is a place holder for the script documentation string. There is also a main function (inspired by Guido) and also a template for calling optparse command line parser module (most of my scripts take one argument, so this is the default).

Easy Web Scraping

This is a little script I use to email myself the latest Reality Check comic from a cron job.

#!/usr/bin/python
# Send new "Reality Check" image in email

from urllib import urlopen
import re
from smtplib import SMTP
from email.MIMEImage import MIMEImage
from email.MIMEMultipart import MIMEMultipart
from time import ctime

# My email
MYMAIL = "miki.tebeka@gmail.com"
# Find email image name
find_image = re.compile("reality\d+\.gif", re.M).search
BASE_URL = "http://www.unitedmedia.com/comics/reality"

def send_new():
    '''Send new image in email'''
    # Find
    im = find_image(urlopen(BASE_URL).read())
    if not im:
        raise ValueError("error: can't find image file in web page")
    image = im.group()

    # Full image URL
    url = BASE_URL + "/archive/images/" + image
    # Read image data
    image_data = urlopen(url).read()

    # Send it in email
    msg = MIMEMultipart()
    msg["Subject"] = "Reality check for %s " % ctime()
    msg["To"] = MYMAIL
    msg["From"] = MYMAIL
    att = MIMEImage(image_data)
    att.add_header("Content-Disposition", "attachment", filename=image)
    msg.attach(att)

    s = SMTP("my-mail-host")
    s.sendmail(MYMAIL, [MYMAIL], msg.as_string())
    s.close()

if __name__ == "__main__":
    try:
        send_new()
    except Exception, e:
        raise SystemExit("error: %s" % e)

Finding the right Python interpreter in makefile

Sometimes I need to run python from a makefile. This little trick helps me find the python interpreter to use in a cross platform way. Have a file called pyexe.py with the following content:

#!/usr/bin/env python
from sys import executable
print executable

Make sure the file is executable on *NIX world and that there is a .py file association on Microsoft land.

Then in the makefile just write

PYTHON = $(shell pyexe.py)

Quick ID Generator

Sometimes I find myself in the need to assign unique id for objects, count from itertools is a quick solution.

from itertools import count

next_id = count(0).next

print next_id() # 0
print next_id() # 1
print next_id() # 2

Using distutils to build SWIG packages

SWIG is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages.

Using SWIG makes connecting libraries written in C/C++ to Python very simple.
However you also need to compile the SWIG generated sources with all the right compiler flags (for finding the Python header files and libraries).

We can use distutils to save us this work as well.

Assume our C library is hello.c and our SWIG interface definition is in hello.i, write the following code in setup.py

from distutils.core import setup, Extension

setup(
    ext_modules = [
        Extension("_hello", sources=["hello.c", "hello.i"])
    ]
)

Note the underscore in the module name, SWIG generated a hello.py which will call import _hello somewhere.

To compile run python setup.py build_ext -i.

For a long article on SWIG and Python see here.

EDIT (3/2010): UnixReview is no longer with us, I'll try to locate the article.

Visitor Design Pattern

The Visitor Design Pattern can be written a bit differently in Python using its rich introspection abilities.

(This is a modified version of the code found in compiler/visitor.py in the Python standard library)

wxPython calculator in 50 lines of code

Using Python Syntax in Configuration Files

As a rule it thumb, it's best to place as many configuration options in a configuration file. This way you don't need to edit/compile the code to change its behavior.

However writing a parser for configuration files takes time, and the standard ones (.ini and .xml) have their limitations. Lucky for us each Python program comes with the full interpreter bundled. We can do stuff like:

# settings.py
from sys import platform

if platform == "win32":
 APP_HOME = "c:\\my_cool_app"
else:
 APP_HOME = "/opt/my_cool_app"

Then using __import__ in our application we can load the configuration file and parse it without a sweat:

# my_cool_app.py
settings = __import__("settings.py")
print settings.APP_HOME

To view this approach taken to extreme, have a look at http://www.unixreview.com/documents/s=9133/ur0404e/

"Hard" breakpoints in Python

Sometimes, when debugging, you want a to stop at abreakpoint no matter what.
Use the following code:

from pdb import set_trace

def buggy_function():
    pass
    pass
    set_trace() # Break here
    pass
    pass

if __name__ == "__main__":
    buggy_function()

When you run the script with python (not with pdb), set_trace will cause the program to stop at this line and pdb console will show.

BTW: You can do the same trick on MSVC if you write __asm int 3

Here We Go

Starting a blog! How original :)

Welcome!

I plan to post a weekly tip on Python programming and programming in general, hope you'll find it useful.

Miki