avrocat

After playing a bit with simpleavro, I've decided it'll be better done in the Unix tradition of "doing one thing well" and using pipes. Hence avrocat was born, a "cat" like utility for avro files.

usage: avrocat [-h] [-n COUNT] [-s SKIP] [-f {json,csv}] [--header]
               [--filter FILTER] [--schema]
               filename

`cat` for Avro files

positional arguments:
  filename              avro file (- for stdin)

optional arguments:
  -h, --help            show this help message and exit
  -n COUNT, --count COUNT
                        number of records to print
  -s SKIP, --skip SKIP  number of records to skip
  -f {json,csv}, --format {json,csv}
                        record format
  --header              print CSV header
  --filter FILTER       filter records (e.g. r['age']>1)
  --schema              print schema

EDIT: Seems like avrocat is finding itself into the "avro" package.

simpleavro

simpleavro 0.1.0 is out!

Delicous Monkey

Sadly, the official Delicious Firefox addon does not support Firefox 4.
I couldn't find any other addon online the gives me keyboard shortcut for saving the current page in Delicious.
Lucky for me, there is GreaseMonkey. Below is a short script to have CTRL-M save the current page in Delicious.

Convert PDF to JPEG

The wife needed a quick and dirty solution to convert PDF files to JPEG. Below is a quick UI (using Tkinter) about ImageMagic's "convert" utility.

A simpler named decorator

Everybody loves decorators. However there's is a difference if a decorator is called with or without () (as in @logged vs @logged("foo")). In the first case, the logged will be called with the function it decorates. In the second case, logged is called with a string and should return a function that accepts the function to be decorated. Below is a simple solution to support both cases in one decorator.

A Whirlwind of Python

I gave a fast paced talk about Python at work, you can download it here. (Unzip and open index.html)

Inside the zip file are also the files used to create the talk. I've used s5 as framework and generated slides from Python source files using Pygments.

Convert oggs to mp3 - the fast way

I've created a quick script to convert all .ogg files in a given directory to .mp3 (using oggdec and lame). However it was running too slow to my taste, a good excuse to play with multiprocessing.Pool. On a short list of 4 .ogg files the processing time went from 32 seconds to 12 seconds.

Adding XML support to ctags

I use ctags a lot with Vim. It let me jump to definitions quickly.

At work we have a lot of spring XML configuration files, and it was pretty easy to add support for XML to ctags. Just add the following to ~/.ctags.

--langdef=XML
--langmap=XML:.xml
--regex-XML=/id="([a-zA-Z0-9_]+)"/\1/d,definition/

AppEngine Work Environment

I'm doing a little AppEngine project (boy, my wife is one tough customer :). On the way I've developed some scripts to help me with my work flow (which is mostly coding with Vim and trying stuff on the REPL).

I have python2.5 (which is the version AppEngine uses) at /opt/python2.5 and I've "pip installed" pyflakes and ipython. The AppEngine SDK is at /opt/google_appengine.

gae.py


repl.sh


run-local.sh


check.sh


push.sh


pypath.sh

JSON decorator for CherryPy

#!/usr/bin/env python

from functools import wraps
import json
from cherrypy import response, expose

def jsonify(func):
    '''JSON decorator for CherryPy'''
    @wraps(func)
    def wrapper(*args, **kw):
        value = func(*args, **kw)
        response.headers["Content-Type"] = "application/json"
        return json.dumps(value)

    return wrapper

def example():
    from cherrypy import quickstart
    from datetime import datetime
    class Time:
        @expose
        @jsonify
        def index(self):
            now = datetime.now()
            return {
                "date" : now.strftime("%Y-%m-%d"),
                "time" : now.strftime("%H:%M"),
                "day" : now.strftime("%A"),
            }

    quickstart(Time())

if __name__ == "__main__":
    example()

$curl -i localhost:8080
HTTP/1.1 200 OK
Date: Mon, 31 Jan 2011 03:18:34 GMT
Content-Length: 56
Content-Type: application/json
Server: CherryPy/3.1.2

{"date": "2011-01-30", "day": "Sunday", "time": "19:18"}

Having hg output svn diffs

In my current workplace we use Subversion as the main VCS. I use Mercurial on top of it to get easy feature branches. My problem was that we use svn patches in our review process, and mercurial (hg diff -r default) gave me different format patches. The solution (as suggested by durin42 on #mercurial) was to use hgsubversion.

After installing hgsubversion using pip (or easy_install), you need to add it to your ~/.hgrc:

    [extensions]
    hgsubversion =

Then you can create patches using "hg diff -svn -r default", I use the following genpatch script for that:

#!/bin/bash
# Generate svn style diff for current hg feature branch

branch=$(hg branch)
if [ -z $branch ]; then
    echo "error: not a mercurial repo" 1>&2
    exit 1
fi

if [ "$branch" == "default" ]; then
    echo "error: in default branch" 1>&2
    exit 1
fi

hg diff --svn -r default > ${branch}.patch

gcalc - Command line interface to Google calculator

#!/usr/bin/env python
'''Command line interface to Google calculator
    gcalc 100f c -> 37.7777778 degrees Celsius
'''

# Idea taken from http://bit.ly/dVL4H3

import json
from urllib import urlopen
import re

def main(argv=None):
    import sys
    from optparse import OptionParser

    argv = argv or sys.argv

    parser = OptionParser("%prog FROM TO\n\t%prog 100f c")
    opts, args = parser.parse_args(argv[1:])
    if len(args) != 2:
        parser.error("wrong number of arguments") # Will exit

    url = "http://www.google.com/ig/calculator?q=%s=?%s" % tuple(args)
    try:
        # We decode to UTF-8 since Google sometimes return funny stuff
        result = urlopen(url).read().decode("utf-8", "ignore")
        # Convert to valid JSON: {foo: "1"} -> {"foo" : "1"}
        result = re.sub("([a-z]+):", '"\\1" :', result)
        result = json.loads(result)
    except (ValueError, IOError), e:
        raise SystemExit("error: %s" % e)

    if result["error"]:
        raise SystemExit("error: %s" % result["error"])

    print result["rhs"]

if __name__ == "__main__":
    main()

svncommit

#!/bin/bash
# Template for "svn commit"
# Add "export SVN_EDITOR=/path/to/this/file" to your .bashrc

# Fail one first error
set -e

filename=$1
editor=${EDITOR-vim}

mv $filename /tmp

# The template
cat << EOF > $filename
Bug # 
Reviewed-by:

EOF

# Add file list to template
cat /tmp/$filename >> $filename
mtime=$(stat -c %y $filename)
$editor $filename

# Restore old file so svn will see it didn't change
if [ "$(stat -c %y $filename)" == "$mtime" ]; then
    mv -f /tmp/$filename $filename
fi

Delays

Sorry for long pause.

I just stared a new job which is not in Python. My interested was also shifted to Clojure.

I'll try to post here more from time to time, but the frequency be lower for the seen future.

In the meanwhile, keep on hacking!

Getting last N items of iterator

islice allow you to get the first N items of an iterator, but since it doesn't accept negative indexes - you can't use it to get the last N. The solution is very simple - use deque.

Configuring Ubuntu For Asus Eee

Lately I lost my main laptop (belonged to the workplace), my older one allows me to use her Eee for now. The Eee came with Windows 7 starter edition, which I grew tired of fast. I've decided to try Ubuntu on it. Trying not to cause too much damage, I used Wubi.

Installing the main distro went without a problem, and after a reboot I had default Gnome desktop up. What I'm describing next is my attempt to get as much screen real estate as possible. Note that I'm spending most of my time in Firefox and in the shell.

General Setup

• Delete the bottom panel (right click and "Delete This Panel")
• Auto hide the top panel (right click, "Properties" and "Autohide")
• Install "Gnome Do"

Firefox Setup

• Right click on the toolbar, "Customize..." and check "Use small icons"
• Click on the "View" menu and leave only the navigation toolbar
• Uncheck the "Status bar" in the "View" menu
• Install the "Hide Menubar" Firefox addon (clicking on "ALT" will show the menu)
• I use GMail and Google Reader, so "Better GMail" and "Better GReader" addons helped
• Install "Adblock Plus" to get more content inside web pages
• This is just a personal preference, but I think installing "Chromifox Extreme" theme freed up some space as well

Terminal Setup

In the "Edit" menu click on "Profile preferences". And on the "General" tab, uncheck "Show menubar by default on new terminals" (use SHIFT-F10 to get a context menu to enable it)

Two More Things

If your kids play webkinz, install Google Chrome. It works much better there (there's always a scroll bar they can use).

As you probably figured out, I am looking for a job right now. If you know someone who is hiring, please point them to my resume.

translate

curl and couchdb - a love story

CouchDB API is JSON over HTTP. Which makes curl my default tool to play with the database.

Here's a small demo:

import_any

Import any file as a Python module. This is a nice big security risk, so beware ...

from types import ModuleType

def import_any(path, name=None):
    module = ModuleType(name or "dummy")
    execfile(path, globals(), module.__dict__)

    return module

Stress Is Good For You