Don't Forget Your Process When Selecting Tools

When selecting tools, there are many things you should consider - prices, value added, integration cost, health ...

However many teams forgot to check how well does the tool integrate with your current development process. A good tool that does not play well with your process will cause many problem and might alter your process a direction you don't want.

For example, we have a process (like many others probably) which involves code reviews. We have a tool with does not play well with code reviews - it has binary projects. The process with this tool is different, much slower and more error prone than the rest of the tools.

Next time you pick a tool to work with, think how well will it play with your current process and take it into consideration.

Note: It might be OK to change the process if the tool is worth it, just make sure it does before you integrate it.

Advice For New Managers

Here's some of the advice I give new managers. I try to talk to them before they start their first management job - let me be the first to ruin their career :)

Disclaimer: I'm not a manager, but I was managed by many and was an officer for many years in the IDF which is very informal kind of army.

Here they are, without any particular order.

Style

Every manager has their own style. From fire-and-forget to micromanagement and everything in the middle. Your style will probably be different from another manager (even from your own), your management style will change over time and from team to team. Play to your strengths - organizational, people skills, ....

Don't be afraid to try until you find a style that both you and your team are comfortable in. As long as you're open about what you're doing, you team will support you. It's hard to measure the effectivenehess of each style, but if you can - do that.

Let Go

You are a manager now, not a developer. I've seen very few managers in my long career that managed to do both effectively. Learn to trust your team and be a manager first and developer last.

Think about the things you liked and disliked in your past managers (and the current one) and figure out what you should do.

Manage Your Time

Management is very different from development, it's much more interruption driven. However you need to allocate time for yourself. I've seen the best officers in the army taking up to 30 minutes to think while all hell breaking loose around them (number 2 was in charge in the meanwhile). If they can do it - I'm sure you can.

Here's a great talk by John Cleese (of the Monty Python fame) about creativity which covers this topic as well.

Know Your Place

In officer training over at IDF, they teach you that an officer should be where he/she's most needed. Same goes for you - think about what you are the most effective at solving the more important things and be there.

Develop A Mantra

Find a sentence which will help you focus, and ask it repeatedly throughout the day. I found out that "Why aren't we deploying?" help our team focus and deliver. However find one that fits your goals/team/company ....

You Team Comes First

If you show your team they come first (mostly by "buffering" them from all the management noise), they will be loyal to you. Loyalty works both ways or it doesn't work.A loyal and jelled team is hard to build but when you get there - they will rock.

That's about it. Don't be nervous and try to enjoy a new position. In the worse case know that management is not for everybody (I personally don't like it). There's no shame in saying "this is not for me" and getting back to to fun stuff. (My brother did just that, he managed for a year and then said to the company - "Either I'm back to development or I'm out". He was back in development).

I hope that you found this useful.

Vi Keybindings in SBCL REPL

I'm (re-)reading "Practical Common Lisp" and going over the exercises/code with SBCL. I found the below script to help me with my Vi addiction :)

Error Handling in Python

I gave a talk at work about error handling in Python.
You can view it here.

What do you think? Did I miss something?

It's worth noting that after a while working with Go and the way it handles errors (if value, err = foo(); err != nil ...). If find my Go programs more verbose but more robust at the same time.

XFCE and Volume Keys

I'm currently playing around with XFCE. I have PulseAudio and wanted the volume keys to work (I'm a keyboard kinda guy. :). Here's a solution:

• Open Settings -> Keyboard -> Application Shortcuts
• Click on "Add"
• Type pactl set-sink-volume 0 +10% and then OK
• Hit the "up volume" button on your keyboard
• Click on "Add" (again)
• Type pactl set-sink-volume 0 -- -10% and then OK
• The -- is not a typo
• Clock on "Add"

Your volume keys sound work now.

Oh, and I find installing they "Greybird" theme [1] (from Ubuntu) makes things much nicer :)

[1] Install with yaourt -S xfce-theme-greybird and then select in Settings -> Appearance

Google Bookmark Command Line Utility

A little utility to save URLs in Google Bookmarks. (This will open a web browser with some fields populated).

IPython on OSX (with iTerm2)

Sadly, in the new workplace I have to use OSX (it's either that or Windows, running Linux and connecting to everything is a ... challenge).

Here's a little script I wrote to open IPython with a new iTerm2 window. I call this script ipy and place it at ~/bin. Then hitting APPLE+SPACE launches Spotlight. Writing "ipy" inside and hitting ENTER does the trick.

If you want VIm keybindings, add the following to ~/.inputrc

set editing-mode vi
tab: complete
set completion-ignore-case on
set blink-matching-paren on

Oh, and if you find out how to maximize the window - please share :)

Command line front end to Jinja2

Below is a little command line front end to the Jinja2 templating library.
(Very much in the spirit of the mako utility).

Want To Be More Productive? - Improve Your I/O

It always surprise me that people trying to makes themselves more productive overlook their I/O - reading and writing.

I still see many people slowly and inefficiently reading on the input side and pecking their way on the keyboard in the output side.

I guess the reason that people don't invest time in improving their I/O is that it's boring. I'll be the first to agree that learning speed reading or touch typing is boring, however the benefits you'll reap are big - the investment will pay itself every time you interact with your computer.

After getting the basics of speed reading and touch typing out of the way [1], there are many more ways you can improve your I/O. Here are few examples:

• Learn to efficiently skim through documents
• Learn to pick what to read
• Master your IDE
• Automate everything
• ...

[1] And you should never stop practicing.

Running "one off" Conversions In PostgreSQL

The other day we needed to runs a one-off job for converting datetime.weekday() values to day names. Here's what we came up with.

Getting Good Errors from Python Map/Reduce Jobs

At work, we use some Python map/reduce jobs (using Hadoop streaming).

Debugging can be difficult, since Hadoop does not keep the Python stacktraces and even if it does - it's very hard to find it. We decided to use crashlog, and now we get wonderful emails with detailed description of what went wrong.

Notes:

• The current mapper input file is in map_input_file environment variable
• Don't forget to add crashlog.py with -file (see here)
• You must add "." to PYTHONPATH in order to import
• import sys; sys.path.append('.') should do the trick
• Email is not the best solution for distributed logging (you get a lot of email when things go South). I'm going to play with graylog2 in the future.

Serving Dynamic Images with matplotlib

Here's an example of generating dynamic images using matplotlib in a web server (flask this time).

Show Location Of Hive Table in HDFS

Here's a script I use to find out where Hive is storing it's data on HDFS (I call it hiveloc).

Quickly Plotting Labeled Data

Here's a quick way to view some labeled data you have (taken from An Introduction to scikit-learn). It will reduce the data to two dimensions using PCA and then scatter plot it with different colors for each label.

Script to close a branch in mercurial (hg)

Mercurial (hg) does not let you delete branches (or alter history in any way). But you can close branches so they won't show in hg branches command.

Here's a script I use to close branches (we work with feature branches at work, and close them when work on the feature is done).

gittip on bitbucket/github

gittip is a cool idea, however currently there's no built-in way to add it to bitbucket/github projects.

One option I found that works it to add a clickable image to your README.md or README.rst.

See example here.

Markdown:

[](https://www.gittip.com/Miki%20Tebeka/)

ReStructuedText:

.. image:: http://i.imgur.com/lg9rx9w.png
   :alt: gittip
   :target: https://www.gittip.com/Miki%20Tebeka/

Notes:

1. You'll probably want to change gittip user id :)
2. There's a discussion on gittip bug tracker on the right way to do this.
3. Unofficial gittip image generated using cooltext.

import "C" slides

Last night I gave a talk about using C from Go at the L.A. Gophers meetup.

You can view the slides here. (Note that "run" won't work due to security restrictions, you can download the slides here and run it locally using the present tool).

Investigating Hash Distribution

A college asked me for a hash function on strings that return an integer between 0 to N. Before diving in, I decided to take the lazy path and check if Python's hash function is good enough.

Luckily, ipython notebook --pylab=inline makes that a breeze.

Check out the notebook here.

And yes, we decided to stick with this solution. I guess we're at least 1/3 programmers.

zipstream - Zip File InputFormat for Hadoop Streaming

At work, we store logs as a single CSV inside a zip file in HDFS (history, that's why :).

Looking around, I couldn't find a FileInput library that works with Hadoop streaming on CDH4 (the version we're using).

So I wrote one, hope you'll find it useful (you can download the jar directly from here.)

Here's an example how to use it:

Abusing namedtuple - Yet Another Enum

There's a discussion over at python-ideas about enum. This prompted me to write yet another implementation of enum, this time abusing namedtuple.

try lock

At work, we have several functions that can run only one at a time. We call this "try lock" (or trylock), and had it forever in the Java code.

When we started a Python project, we wanted this functionality. A decorator seems like the right solution. The below try_lock decorator has an optional function that lets you get a finer grained solution on what to lock. It gets the function arguments and returns a key to lock on. If you don't specify keyfn, then there will be just one lock for the function.

whoops - A WebHDFS Library and Client

Just released whoops 0.1.0 which is a WebHDFS library and a command line client for Python.

Timing Your Code

It's a good idea to time portions of your code and have some metric you monitor. This way you can see trends and solve bottlenecks before someone notices (hopefully). Timing functions is easy with decorators, but sometimes you want to time a portion of a function. For this you can use a context manager.

fastavro got snappy support

fastavro 0.7.1 introduces snappy support! (also 3.3 egg)

Last Letter Frequency

I was playing a game with my child where you say a word, then the other person need to say a word which starts with the last letter of the word you said, then you need to say a word with their last letter ...

We noticed that many words end with S and E, which made me curious about the frequency of the last letter in English words. matplotlib makes it super easy to visualize the results.

Python For Data Analysis

Just finished reading Python For Data Analysis, it's a great book with lots of practical examples. Highly recommended.

Mocking HTTP Servers

Sometimes, httpbin is not enough, and you need your own custom HTTP server for testing.
Here's a small example on how to do that using the built in SimpleHTTPServer (thanks @noahsussman for reminding me).

http://httpbin.org

Sometimes you need to write an HTTP server to debug the client you are writing.

One quick way to avoid this is to use http://httpbin.org/. It supports most of the common HTTP verbs and mostly return the variables you send in.

For example (note the args field in the reply):

$ curl -i 'http://httpbin.org/get?x=1&y=2'
HTTP/1.1 200 OK
Content-Type: application/json
Date: Mon, 15 Oct 2012 21:50:27 GMT
Server: gunicorn/0.13.4
Content-Length: 386
Connection: keep-alive

{
  "url": "http://httpbin.org/get?x=1&y=2",
  "headers": {
    "Content-Length": "",
    "Connection": "keep-alive",
    "Accept": "*/*",
    "User-Agent": "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3",
    "Host": "httpbin.org",
    "Content-Type": ""
  },
  "args": {
    "y": "2",
    "x": "1"
  },
  "origin": "75.82.8.111"
}

Cleanup After Your Tests - But Be Lazy

It's a nice practice to clean after your tests. It's good for various reasons like disk space, "pure" execution environment and others.

However if you clean up to eagerly it'll make your debugging much harder. The data just won't be there to see what went wrong.

The solution we found is pretty simple:

• Try to place all your test output in one location
• Nuke this location when starting the tests

This way all the information is available after an error, and you don't accumulate too much junk (just one test run junk at a time).

Data Wrangling With Python

I just gave a talk at work called "Data Wrangling With Python" which gives an overview on the scientific Python ecosystem. You can view it here.

Using Hadoop Streaming With Avro

One of the way to use Python with Hadoop is via Hadoop Streaming. However it's geared mostly toward text based format and at work we use mostly Avro.

Took me a while to figure the magic, but here it is. Note that the input to the mapper is one JSON object per line.

Note it's a bit old (Avro is now at 1.7.4), originally from here.

Setting Maching Learning on OSX

Setting up machine learning tools (numpy, scipy, matplotlib, scikit-learn, ...) can be a pain (why can they just use a decent OS? :).

We are lucky to have Ben Kim now with us at Adconion, and he posted the following:

Mac OS X Lion Software Installs

1. Install compilers
   1. Install XCode 4.x from the App Store
      1. Install Command Line Tools in Preferences/Download
   2. Install gcc, g++, and gfortran compilers
      1. Download tar file
      2. Extract to /
         1. tar -xvf abc.tar -C /
   3. Reference http://sites.google.com/site/dwhipp/tutorials/mac_compilers
2. Install Homebrew
   1. Run the install command using ruby

      1. ruby <(curl -fsSkL raw.github.com/mxcl/homebrew/go)

   2. brew doctor
      1. chown /usr/local folders listed
      2. Place /usr/local/bin before /usr/bin in path
   3. Reference https://github.com/mxcl/homebrew/wiki/installation
3. Install python using brew
   1. brew install readline sqlite gdbm pkg-config
   2. brew install python
   3. Note: Mac OS X Lion comes with old version 2.7.1 of python (python --version) 
4. Set PATH in .bash_profile
   1. vim ~/.bash_profile
      1. export PATH=/usr/local/share/python:/usr/local/bin:$PATH
5. Create symlinks
   1. Within /(System/)?Library/Frameworks/Python.framework/Versions, sudo rm Current
   2. Within the above directories, ln -s /usr/local/Cellar/python/2.7.3 Current
6. Install pip, if necessary, using easy_install
   1. sudo easy_install pip
7. Using pip (sudo pip install [--upgrade] abc)
   1. Install nose
   2. Install numpy
   3. Install scipy with environmental variables settings
      1. sudo CC=clang CXX=clang FFLAGS=-ff2c pip install [--upgrade] scipy
   4. Install scikit-learn
   5. Install pandas
8. Install matplotlib
   1. Download source from repo: https://github.com/matplotlib/matplotlib
   2. cd ~Downloads/matplotlib-*
   3. python setup.py build
   4. python setup.py install
9. Install VW (Vowpal Wabbit)
   1. Install boost
      1. Download tar file
      2. mv boost extracted folder to /usr/local
      3. export BOOST_ROOT environmental variable
      4. cd to boost directory
      5. make and install
         1. sudo ./bootstrap
         2. sudo ./bjam install
      6. Download bjam
      7. mv to directory in PATH
         1. mv bjam /usr/local/bin
      8. Set bjam toolset to darwin
         1. bjam toolset=darwin stage
      9. Reference http://www.boost.org/doc/libs/1_41_0/more/getting_started/unix-variants.html#expected-build-output
   2. cd to VW directory
      1. make and test
         1. make
         2. make test

A "cd" context manager

Packages Download Url

Inspired by StackOverflow question:

Show Dependecies of Azkaban Jobs

We're using Azkaban at work to schedule Hadoop jobs. It's hard to view the job dependencies without deploying, so here's a little script that will show you job dependencies as an image. It uses dot (from Graphviz) to produce the image.

Python Based Assembler

Got reminded of a project I did while back. It's a Python based assembler. The main idea is that the assembly file is actually a Python file with pre-set functions (assembly instruction). In this manner, I managed to skip lexing, parsing and other things and deliver a working assembler in two days. You can view the presentation I gave on this here.

The Assembler

Example Input

Using travis-ci with bitbucket

travis-ci is a great service. My problem is that it works only with github while I mainly use bitbucket (and please, let's not get into hg/git debate - hg is way better :).

The way I found to make this work is to mirror my bitbucket projects on github using hg-git. Below is an example from fastavro.

First, you need to install hg-git. It's available from PyPI, "pip install hg-git" will do the trick, (or "easy_install hg-git" if you don't have pip).

Then create a repository on github to mirror the one on bitbucket. After that tell travis-ci to watch this repository.

Next step is to enable hg-git in your repository, edit .hg/hgrc and add the following:

[extensions]
hgext.bookmarks =
hggit =

Then "bootstrap" it with the following command:

hg bookmark -r default master

Next step is to create .travis.yml, For fastavro I have both Python 2.7 and 3.2.

Last step, is to make sure every time we push to bitbucket, changes are pushed to github as well. This is done with an outgoing hook in .hg/hgrc

[hooks]
outgoing = hg push git+ssh://git@github.com/tebeka/fastavro.git || true

(The || true is there since hg push will exit with non-zero value sometimes)

That's all. Now fastavro has continuous integration that runs both on Python 2.7 and 3.2.

Twitter Post Frequency

Sometime I see interesting new people on Twitter. However before adding them I'd like to know what is their post frequency so I won't get spammed. Below is a simple script to do that:

A lambda Gotcha

Quick, what is the output of the following?

In [1]: callbacks = [lambda: i for i in range(10)]

In [2]: [c() for c in callbacks]

The right answer is:

Out[2]: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

This is due to the fact that i is bound to the same variable in all the lambdas, and has the final value of 9.

There are two ways to overcome this. The first is to use the fact the default arguments are evaluated at function creation time (which is another known gotcha).

In [3]: callbacks = [lambda i=i: i for i in range(10)]
In [4]: [c() for c in callbacks]
Out[4]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

The second is to create a function generator function:
In [5]: def make_callback(i):
   ...:     return lambda: i
   ...:
In [6]: callbacks = [make_callback(i) for i in range(10)]
In [7]: [c() for c in callbacks]
Out[7]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Reading Avro Files Faster than Java

At work, we use a lot of Avro. One of the problems we faced was that the Python Avro package is very slow comparing to the Java one. The goal then was to write fastavro which is a subset of the avro package and should be at least as fast as Java. In this post I'll show how fastavro became faster than Java and also Python 3 compatible.

Going Fast
The Python avro package uses classes and properties heavily. This might allow for nice design but since function calls in Python are expensive it has a cost. The approach was to strip down most of the code in the avro package to one simple module, eliminating as many function calls as possible along the way and using only built in types. After some tweaking, fastavro was churning through the 10K records benchmark in about 2.6seconds (comparing to 13.9 seconds of the avro package). It was a nice speedup but the goal was to be as fast as Java (which was doing about 1.8sec).


Going Faster
Enter Cython. fastavro compiles the Python code without any specific Cython code. This way on machines that do not have a compiler users can still use fastavro. This complicated the build process a bit since now the C extension is generated using Cython in external Makefile. The code in fastavro first tries to import the C extension and if it fails imports the pure Python one.

This approach gave a 2x speedup (benchmark of 10K records done in 1.5seconds). Again, this is without any Cython specific code.

Python 3 Support
The initial Python 3 support was written on the first day of PyCon. However after hearing Robert Brewer's excellent talk I decided to take his advice and write a small compatibility layer (six was not used for various reasons).

As Robert said, this approach made fastavro better with strings, unicode and other things which were glossed over the 2.X only code. The build system was simplified a lot comparing to the one with the initial Python 3 support.

End Result
The end result is a package that reads Avro faster than Java and supports both Python 2 and Python 3. Using Cython and a little bit of work the was achieved without too much effort.

As usual, the code can be found on bitbucket.

Super Simple Mocking

There are many mocking libraries for Python out there. Due to the dynamic nature of Python I find them an overkill. Below is a super simple mocking library that works for me.


Note that for some types (such as C extensions, objects with __slots__ ...) this will not work since they do not have a __dict__.

EDIT: Following HackerNews comments , I've changed the interface to mock(obj, **kw).

Adding Key Navigation to Your Web Site

I don't like using the mouse and thankful for every web site that adds keyboard navigation (like Google Reader). Be nice to your users and add some yourself.

jQuery makes it super easy to add keyboard navigation to your site, below is a simple example adding keyboard navigation. "k/j" for up/down, "o" for opening an item and "?" for toggling help. JavaScript code starts at line 90.

fastavro with Cython

Added an optional step of compiling fastavro with Cython. Just doing that, with no Cython specific code reduced the time of processing 10K records from 2.9sec to 1.7sec. Not bad for that little work.

Also added a __main__.py so you can use fastavro to process Avro files:

• python -m fastavro weather.avro # Dump records in JSON format
• python -m fastavro --schema weather.avro # Dump schema

fastavro

Just released fastavro to PyPI. It has way less features than the official avro package, but according to my tests it's about 5 times faster.

How To Use A Chat To Amplify Your Team

A chat room is a great tool for any team. I've worked both in distributed and "local" team and in both cases a central chat room was one of the most effective tools we used. The chat room is the central location for updates, conversations, shared knowledge and more. It's also highly effective in many open source projects.


Starting is super easy, you can either set an internal server in your company with one of many open IRC/Jabber servers or try an external service (like Campfire, or HipChat).

The success of the chat room depends on the signal/noise ratio in the main room. Try to have every new conversation start in the main room, and only if it becomes to noisy move it to another room. 1/1 conversations are hiding information from the team and should be used only for personal communication.

Don't forget add services to the chat room. It's easy to add build notification, source commits, reminders and other smart bots. We use (Jenkins Jabber plugin) to get build notification. However be careful not to generate too much noise and tweak all these bots.

Logging and searching is another critical feature. Some services (like Campfire) provide it out of the box. If your chat server does not provide logging, it's pretty easy to add it with one of the many logging bots out there, and hooking search on top of the logs is pretty easy as well. (That's what we did with selenium.saucelabs.com which is a combination of Supybot and Omega).

Another issue is presence, people need to make it clear when they are "in" or "out" of the room. In some cases when the server logs it's as easy as login/logout. In other cases just notify the team your are away (we use the IRC /me lunching a lot). Stating that it's acceptable to be "out" of the room allows people to close the chat when they need to be in the zone.

So go ahead, set a chat for you team. Soon you'll wonder how you managed before.

Decimals Class Decorator

A friend at work had a problem where he wanted to make sure some attributes are always Decimal.

We considered using the excellent traits library, but it's an overkill and we didn't want to introduce another dependency. For me this looked like a job for properties but we wanted some way to generate the properties automatically. Lucky for us we now have class decorators (in the old days I'd probably use a metaclass).

branches

At work, our workflow involves JIRA and feature branches. We name the feature branches in the name of the issue they are solving.

However when doing hg branches it's hard to know what branch you want.
Below is a small script that annotates the output of hg branches with the issue description from JIRA (and also added the branch parent).

VirtualBox and USB

A little something I found out, writing it here in hope someone else will find it useful.

I'm on a Ubunbu system (11.10), and needed to connect to a USB device from an XP VM (wanted to sync and upgrade iPod touch, there is no iTunes for Linux). I couldn't connect the XP VM to any USB port (the USB menu was empty). The solution is to add yourself to the vboxusers group.

Open Users and Group, click on Manage Groups and double click on vboxusers and make sure your name is checked. After that logout and login, you should see the USB devices in the USB menu now.

Summarize tuples

Raymond asked "Group the list of tuples on a given field and sum or count selected other fields.", here's my solution using sqlite3. (original answer here).

Using dbus To Control Pidgin

I using Pidgin as my IM client, however after work I'd like to disconnect from our jabber server. Being the command like geek I am, I wrote the following. (Earlier version was written in bash using purple-remote .) You can learn more about Pidgin dbus interface here.

Finding Module Version

A friend at work asked me how can he find which version of Python module is currently installed. The quick answer is to use pip:
     
        pip freeze | grep <module>

Old Berlios Projects

Seems like Berlios is closing down. I've placed a backup of my old projects here. Let me know if you like to take ownership on one/several of them.

Using Generators in nose Tests

Sometimes you have many tests that do exactly the same thing but with different data. nose provides an easy way to do that with tests that return generators.

Feynman On The Importance Of Playing

I'm reading "Surely You're Joking, Mr. Feynman" (very good read).
What he says about burnout, playing and doing the things you love is priceless:

But when it came time to do some research. I couldn't get to work. I was a little tired; I was not interested; I couldn't do research! ...

And then I thought to myself, "You know, what they think of you is so fantastic, it's impossible to live up to it. You have no responsibility to live up to it!"... 

Then I had another thought; Physics disgusts me a little bit now, but I used to enjoy doing physics. Why did I enjoy it? I used to play with it. I used to do whatever I felt like doing - it didn't have to do with whether it was important for the development of nuclear physics...

So I get this new attitude ... I'm going to play with physics, whenever I want to, without worrying about any importance whatsoever.

Within a week I was in the cafeteria and some guy, fooling around, throws a plate in the air. ...

I had nothing to do, so I start to figure out the motion of the rotating plate...

And before I knew it (it was a very short time) I was "playing" - working, really - with the same old problem that I loved so much, that I had stopped working on when I went to Los Alamos; my thesis-type problems; all those old-fashioned wonderful things.

It was effortless. It was easy to play with these things. It was like uncorking a bottle: Everything flowed out effortlessly. ...

There was no importance to what I was doing, but ultimately there was. The diagrams and the whole business that I got the Nobel Prize for came from that piddling around with the wobbling plate.

Go out and play, I'm sure you'll make wonderful things.

EDIT: Thanks for HN for proof reading this. Also found a more complete excerpt here.

istail and ipeek

Prompted by a question in comp.lang.python:

"Top X" UI

Doing some experiments with data at work, one of the common tasks was "show top X items". However X was changing all the time. Finally I came out with a web based UI where you can set the top yourself. Hopefully someone else will find it useful.

Key points:

• Using jQuery, jQuery UI and flot
• Site is static, no backend needed
• Data is generated to a .js file

You can see a demo on word frequency of "Alice In Wonderland" here.


As usual, the code is in bitbucket.

Crucible Command Line Client

At work, we use Crucible (which I mostly like). However we do reviews post commit (using feature branches). And creating a review from a patch is too much clicks to my taste. Hence crucible command line client. You can install it with easy_install crucible and then run it.

crucible has many options, but you can save many defaults in ~/.cruciblerc (see example at top of the crucible script).

logio

The logging module is nice, one think I find myself missing here and there is the ability to treat logs as file objects. This help them "play nice" with many modules.

Here is a quick hack to do that, at the bottom there's an example on how to create CSV log files (and then we can use TimedRotatingFileHandler to rotate the logs).

PDB - The Movie

I did it again, this time showing how to work with PDB (the Python debugger).

Could it be the proximity to Hollywood that causes me to do these movies?

ingress

One of the many cool features in Twisted is "manhole". It lets you connect to any running server and get a shell (over ssh or telnet) in the server environment. This is very helpful when debugging.

It's very easy to implement this using SocketServer which is in the standard library, this way even if you're not running under Twised, you can still have this feature. I've created ingress just for this (easy_install ingress), using it is super easy:

import ingress
ingress.install()

The code itself is not that complicated. (Could have been simpler if there was a way to override writing to stdout in code.InteractiveConsole).

3 IDLE Tips - The Movie

I've managed to make a screencast, giving tips on how to use IDLE.
(Yeah, I know - it sucks, but that's how I learn ... ;)

avrocat

After playing a bit with simpleavro, I've decided it'll be better done in the Unix tradition of "doing one thing well" and using pipes. Hence avrocat was born, a "cat" like utility for avro files.

usage: avrocat [-h] [-n COUNT] [-s SKIP] [-f {json,csv}] [--header]
               [--filter FILTER] [--schema]
               filename

`cat` for Avro files

positional arguments:
  filename              avro file (- for stdin)

optional arguments:
  -h, --help            show this help message and exit
  -n COUNT, --count COUNT
                        number of records to print
  -s SKIP, --skip SKIP  number of records to skip
  -f {json,csv}, --format {json,csv}
                        record format
  --header              print CSV header
  --filter FILTER       filter records (e.g. r['age']>1)
  --schema              print schema

EDIT: Seems like avrocat is finding itself into the "avro" package.

simpleavro

simpleavro 0.1.0 is out!

Delicous Monkey

Sadly, the official Delicious Firefox addon does not support Firefox 4.
I couldn't find any other addon online the gives me keyboard shortcut for saving the current page in Delicious.
Lucky for me, there is GreaseMonkey. Below is a short script to have CTRL-M save the current page in Delicious.

Convert PDF to JPEG

The wife needed a quick and dirty solution to convert PDF files to JPEG. Below is a quick UI (using Tkinter) about ImageMagic's "convert" utility.

A simpler named decorator

Everybody loves decorators. However there's is a difference if a decorator is called with or without () (as in @logged vs @logged("foo")). In the first case, the logged will be called with the function it decorates. In the second case, logged is called with a string and should return a function that accepts the function to be decorated. Below is a simple solution to support both cases in one decorator.

A Whirlwind of Python

I gave a fast paced talk about Python at work, you can download it here. (Unzip and open index.html)

Inside the zip file are also the files used to create the talk. I've used s5 as framework and generated slides from Python source files using Pygments.

Convert oggs to mp3 - the fast way

I've created a quick script to convert all .ogg files in a given directory to .mp3 (using oggdec and lame). However it was running too slow to my taste, a good excuse to play with multiprocessing.Pool. On a short list of 4 .ogg files the processing time went from 32 seconds to 12 seconds.

Adding XML support to ctags

I use ctags a lot with Vim. It let me jump to definitions quickly.

At work we have a lot of spring XML configuration files, and it was pretty easy to add support for XML to ctags. Just add the following to ~/.ctags.

--langdef=XML
--langmap=XML:.xml
--regex-XML=/id="([a-zA-Z0-9_]+)"/\1/d,definition/

AppEngine Work Environment

I'm doing a little AppEngine project (boy, my wife is one tough customer :). On the way I've developed some scripts to help me with my work flow (which is mostly coding with Vim and trying stuff on the REPL).

I have python2.5 (which is the version AppEngine uses) at /opt/python2.5 and I've "pip installed" pyflakes and ipython. The AppEngine SDK is at /opt/google_appengine.

gae.py


repl.sh


run-local.sh


check.sh


push.sh


pypath.sh

JSON decorator for CherryPy

#!/usr/bin/env python

from functools import wraps
import json
from cherrypy import response, expose

def jsonify(func):
    '''JSON decorator for CherryPy'''
    @wraps(func)
    def wrapper(*args, **kw):
        value = func(*args, **kw)
        response.headers["Content-Type"] = "application/json"
        return json.dumps(value)

    return wrapper

def example():
    from cherrypy import quickstart
    from datetime import datetime
    class Time:
        @expose
        @jsonify
        def index(self):
            now = datetime.now()
            return {
                "date" : now.strftime("%Y-%m-%d"),
                "time" : now.strftime("%H:%M"),
                "day" : now.strftime("%A"),
            }

    quickstart(Time())

if __name__ == "__main__":
    example()

$curl -i localhost:8080
HTTP/1.1 200 OK
Date: Mon, 31 Jan 2011 03:18:34 GMT
Content-Length: 56
Content-Type: application/json
Server: CherryPy/3.1.2

{"date": "2011-01-30", "day": "Sunday", "time": "19:18"}

Having hg output svn diffs

In my current workplace we use Subversion as the main VCS. I use Mercurial on top of it to get easy feature branches. My problem was that we use svn patches in our review process, and mercurial (hg diff -r default) gave me different format patches. The solution (as suggested by durin42 on #mercurial) was to use hgsubversion.

After installing hgsubversion using pip (or easy_install), you need to add it to your ~/.hgrc:

    [extensions]
    hgsubversion =

Then you can create patches using "hg diff -svn -r default", I use the following genpatch script for that:

#!/bin/bash
# Generate svn style diff for current hg feature branch

branch=$(hg branch)
if [ -z $branch ]; then
    echo "error: not a mercurial repo" 1>&2
    exit 1
fi

if [ "$branch" == "default" ]; then
    echo "error: in default branch" 1>&2
    exit 1
fi

hg diff --svn -r default > ${branch}.patch

gcalc - Command line interface to Google calculator

#!/usr/bin/env python
'''Command line interface to Google calculator
    gcalc 100f c -> 37.7777778 degrees Celsius
'''

# Idea taken from http://bit.ly/dVL4H3

import json
from urllib import urlopen
import re

def main(argv=None):
    import sys
    from optparse import OptionParser

    argv = argv or sys.argv

    parser = OptionParser("%prog FROM TO\n\t%prog 100f c")
    opts, args = parser.parse_args(argv[1:])
    if len(args) != 2:
        parser.error("wrong number of arguments") # Will exit

    url = "http://www.google.com/ig/calculator?q=%s=?%s" % tuple(args)
    try:
        # We decode to UTF-8 since Google sometimes return funny stuff
        result = urlopen(url).read().decode("utf-8", "ignore")
        # Convert to valid JSON: {foo: "1"} -> {"foo" : "1"}
        result = re.sub("([a-z]+):", '"\\1" :', result)
        result = json.loads(result)
    except (ValueError, IOError), e:
        raise SystemExit("error: %s" % e)

    if result["error"]:
        raise SystemExit("error: %s" % result["error"])

    print result["rhs"]

if __name__ == "__main__":
    main()

svncommit

#!/bin/bash
# Template for "svn commit"
# Add "export SVN_EDITOR=/path/to/this/file" to your .bashrc

# Fail one first error
set -e

filename=$1
editor=${EDITOR-vim}

mv $filename /tmp

# The template
cat << EOF > $filename
Bug # 
Reviewed-by:

EOF

# Add file list to template
cat /tmp/$filename >> $filename
mtime=$(stat -c %y $filename)
$editor $filename

# Restore old file so svn will see it didn't change
if [ "$(stat -c %y $filename)" == "$mtime" ]; then
    mv -f /tmp/$filename $filename
fi

Delays

Sorry for long pause.

I just stared a new job which is not in Python. My interested was also shifted to Clojure.

I'll try to post here more from time to time, but the frequency be lower for the seen future.

In the meanwhile, keep on hacking!

Getting last N items of iterator

islice allow you to get the first N items of an iterator, but since it doesn't accept negative indexes - you can't use it to get the last N. The solution is very simple - use deque.

Configuring Ubuntu For Asus Eee

Lately I lost my main laptop (belonged to the workplace), my older one allows me to use her Eee for now. The Eee came with Windows 7 starter edition, which I grew tired of fast. I've decided to try Ubuntu on it. Trying not to cause too much damage, I used Wubi.

Installing the main distro went without a problem, and after a reboot I had default Gnome desktop up. What I'm describing next is my attempt to get as much screen real estate as possible. Note that I'm spending most of my time in Firefox and in the shell.

General Setup

• Delete the bottom panel (right click and "Delete This Panel")
• Auto hide the top panel (right click, "Properties" and "Autohide")
• Install "Gnome Do"

Firefox Setup

• Right click on the toolbar, "Customize..." and check "Use small icons"
• Click on the "View" menu and leave only the navigation toolbar
• Uncheck the "Status bar" in the "View" menu
• Install the "Hide Menubar" Firefox addon (clicking on "ALT" will show the menu)
• I use GMail and Google Reader, so "Better GMail" and "Better GReader" addons helped
• Install "Adblock Plus" to get more content inside web pages
• This is just a personal preference, but I think installing "Chromifox Extreme" theme freed up some space as well

Terminal Setup

In the "Edit" menu click on "Profile preferences". And on the "General" tab, uncheck "Show menubar by default on new terminals" (use SHIFT-F10 to get a context menu to enable it)

Two More Things

If your kids play webkinz, install Google Chrome. It works much better there (there's always a scroll bar they can use).

As you probably figured out, I am looking for a job right now. If you know someone who is hiring, please point them to my resume.

translate

curl and couchdb - a love story

CouchDB API is JSON over HTTP. Which makes curl my default tool to play with the database.

Here's a small demo:

import_any

Import any file as a Python module. This is a nice big security risk, so beware ...

from types import ModuleType

def import_any(path, name=None):
    module = ModuleType(name or "dummy")
    execfile(path, globals(), module.__dict__)

    return module

Stress Is Good For You

Tagging last good build in git

git (and mercurial) has a nice feature that you can "move" tags (using -f). At Sauce Labs we use that to have a track of our last good build and the last deployed revision.

In our continuous integration system, we mark the last good build in the last step which is executed only if the test suite passed.

tag-ok-build.sh

#!/bin/bash

# Error on 1'st failure
set -e

tag=last-green-build
revision=$(git rev-parse HEAD)

echo "Tagging $commit as $tag"
git tag -f $tag $commit
git pull origin master
git push --tags origin master

To deploy, we use the following script who also tags the last deployed version.

deploy.sh

#!/bin/bash
# Deploy, meaning sync from last successful build

tag=last-green-build

# Fetch get the latest changes from remote but does not update working
# directory, just the index
git fetch --tags

# Merge to the last successful bulid
git merge ${tag}

# Tag currently deployed revision
git tag -f deployed ${tag}
git push --tags

couchnuke

A simple script to "nuke" a couchdb server, use with care!

#!/bin/bash

# Delete all databases on a couchdb server

# Exit on 1'st error
set -e

if [ $# -gt 1 ]; then
    echo "usage: $(basename $0) [DB_URL]"
    exit 1
fi

URL=${1-http://localhost:5984}

echo -n "Delete *ALL* databases on $URL? [y/n] "
read ans
if [ "$ans" != "y" ]; then
    exit
fi

for db in $(curl -s $URL/_all_dbs | tr '",[]' ' \n  ');
do
    echo "Deleting $db"
    curl -X DELETE $URL/$db
done