Easy Web Scraping

This is a little script I use to email myself the latest Reality Check comic from a cron job.

#!/usr/bin/python
# Send new "Reality Check" image in email

from urllib import urlopen
import re
from smtplib import SMTP
from email.MIMEImage import MIMEImage
from email.MIMEMultipart import MIMEMultipart
from time import ctime

# My email
MYMAIL = "miki.tebeka@gmail.com"
# Find email image name
find_image = re.compile("reality\d+\.gif", re.M).search
BASE_URL = "http://www.unitedmedia.com/comics/reality"

def send_new():
    '''Send new image in email'''
    # Find
    im = find_image(urlopen(BASE_URL).read())
    if not im:
        raise ValueError("error: can't find image file in web page")
    image = im.group()

    # Full image URL
    url = BASE_URL + "/archive/images/" + image
    # Read image data
    image_data = urlopen(url).read()

    # Send it in email
    msg = MIMEMultipart()
    msg["Subject"] = "Reality check for %s " % ctime()
    msg["To"] = MYMAIL
    msg["From"] = MYMAIL
    att = MIMEImage(image_data)
    att.add_header("Content-Disposition", "attachment", filename=image)
    msg.attach(att)

    s = SMTP("my-mail-host")
    s.sendmail(MYMAIL, [MYMAIL], msg.as_string())
    s.close()

if __name__ == "__main__":
    try:
        send_new()
    except Exception, e:
        raise SystemExit("error: %s" % e)

Finding the right Python interpreter in makefile

Sometimes I need to run python from a makefile. This little trick helps me find the python interpreter to use in a cross platform way. Have a file called pyexe.py with the following content:

#!/usr/bin/env python
from sys import executable
print executable

Make sure the file is executable on *NIX world and that there is a .py file association on Microsoft land.

Then in the makefile just write

PYTHON = $(shell pyexe.py)

Quick ID Generator

Sometimes I find myself in the need to assign unique id for objects, count from itertools is a quick solution.

from itertools import count

next_id = count(0).next

print next_id() # 0
print next_id() # 1
print next_id() # 2

Using distutils to build SWIG packages

SWIG is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages.

Using SWIG makes connecting libraries written in C/C++ to Python very simple.
However you also need to compile the SWIG generated sources with all the right compiler flags (for finding the Python header files and libraries).

We can use distutils to save us this work as well.

Assume our C library is hello.c and our SWIG interface definition is in hello.i, write the following code in setup.py

from distutils.core import setup, Extension

setup(
    ext_modules = [
        Extension("_hello", sources=["hello.c", "hello.i"])
    ]
)

Note the underscore in the module name, SWIG generated a hello.py which will call import _hello somewhere.

To compile run python setup.py build_ext -i.

For a long article on SWIG and Python see here.

EDIT (3/2010): UnixReview is no longer with us, I'll try to locate the article.

Visitor Design Pattern

The Visitor Design Pattern can be written a bit differently in Python using its rich introspection abilities.

(This is a modified version of the code found in compiler/visitor.py in the Python standard library)

wxPython calculator in 50 lines of code

Using Python Syntax in Configuration Files

As a rule it thumb, it's best to place as many configuration options in a configuration file. This way you don't need to edit/compile the code to change its behavior.

However writing a parser for configuration files takes time, and the standard ones (.ini and .xml) have their limitations. Lucky for us each Python program comes with the full interpreter bundled. We can do stuff like:

# settings.py
from sys import platform

if platform == "win32":
 APP_HOME = "c:\\my_cool_app"
else:
 APP_HOME = "/opt/my_cool_app"

Then using __import__ in our application we can load the configuration file and parse it without a sweat:

# my_cool_app.py
settings = __import__("settings.py")
print settings.APP_HOME

To view this approach taken to extreme, have a look at http://www.unixreview.com/documents/s=9133/ur0404e/

"Hard" breakpoints in Python

Sometimes, when debugging, you want a to stop at abreakpoint no matter what.
Use the following code:

from pdb import set_trace

def buggy_function():
    pass
    pass
    set_trace() # Break here
    pass
    pass

if __name__ == "__main__":
    buggy_function()

When you run the script with python (not with pdb), set_trace will cause the program to stop at this line and pdb console will show.

BTW: You can do the same trick on MSVC if you write __asm int 3

Here We Go

Starting a blog! How original :)

Welcome!

I plan to post a weekly tip on Python programming and programming in general, hope you'll find it useful.

Miki