IPython on OSX (with iTerm2)

Sadly, in the new workplace I have to use OSX (it's either that or Windows, running Linux and connecting to everything is a ... challenge).

Here's a little script I wrote to open IPython with a new iTerm2 window. I call this script ipy and place it at ~/bin. Then hitting APPLE+SPACE launches Spotlight. Writing "ipy" inside and hitting ENTER does the trick.

If you want VIm keybindings, add the following to ~/.inputrc

set editing-mode vi
tab: complete
set completion-ignore-case on
set blink-matching-paren on

Oh, and if you find out how to maximize the window - please share :)

Command line front end to Jinja2

Below is a little command line front end to the Jinja2 templating library.
(Very much in the spirit of the mako utility).

Want To Be More Productive? - Improve Your I/O

It always surprise me that people trying to makes themselves more productive overlook their I/O - reading and writing.

I still see many people slowly and inefficiently reading on the input side and pecking their way on the keyboard in the output side.

I guess the reason that people don't invest time in improving their I/O is that it's boring. I'll be the first to agree that learning speed reading or touch typing is boring, however the benefits you'll reap are big - the investment will pay itself every time you interact with your computer.

After getting the basics of speed reading and touch typing out of the way [1], there are many more ways you can improve your I/O. Here are few examples:

• Learn to efficiently skim through documents
• Learn to pick what to read
• Master your IDE
• Automate everything
• ...

[1] And you should never stop practicing.

Running "one off" Conversions In PostgreSQL

The other day we needed to runs a one-off job for converting datetime.weekday() values to day names. Here's what we came up with.

Getting Good Errors from Python Map/Reduce Jobs

At work, we use some Python map/reduce jobs (using Hadoop streaming).

Debugging can be difficult, since Hadoop does not keep the Python stacktraces and even if it does - it's very hard to find it. We decided to use crashlog, and now we get wonderful emails with detailed description of what went wrong.

Notes:

• The current mapper input file is in map_input_file environment variable
• Don't forget to add crashlog.py with -file (see here)
• You must add "." to PYTHONPATH in order to import
• import sys; sys.path.append('.') should do the trick
• Email is not the best solution for distributed logging (you get a lot of email when things go South). I'm going to play with graylog2 in the future.

Serving Dynamic Images with matplotlib

Here's an example of generating dynamic images using matplotlib in a web server (flask this time).

Show Location Of Hive Table in HDFS

Here's a script I use to find out where Hive is storing it's data on HDFS (I call it hiveloc).

Quickly Plotting Labeled Data

Here's a quick way to view some labeled data you have (taken from An Introduction to scikit-learn). It will reduce the data to two dimensions using PCA and then scatter plot it with different colors for each label.

Script to close a branch in mercurial (hg)

Mercurial (hg) does not let you delete branches (or alter history in any way). But you can close branches so they won't show in hg branches command.

Here's a script I use to close branches (we work with feature branches at work, and close them when work on the feature is done).

gittip on bitbucket/github

gittip is a cool idea, however currently there's no built-in way to add it to bitbucket/github projects.

One option I found that works it to add a clickable image to your README.md or README.rst.

See example here.

Markdown:

[](https://www.gittip.com/Miki%20Tebeka/)

ReStructuedText:

.. image:: http://i.imgur.com/lg9rx9w.png
   :alt: gittip
   :target: https://www.gittip.com/Miki%20Tebeka/

Notes:

1. You'll probably want to change gittip user id :)
2. There's a discussion on gittip bug tracker on the right way to do this.
3. Unofficial gittip image generated using cooltext.

import "C" slides

Last night I gave a talk about using C from Go at the L.A. Gophers meetup.

You can view the slides here. (Note that "run" won't work due to security restrictions, you can download the slides here and run it locally using the present tool).

Investigating Hash Distribution

A college asked me for a hash function on strings that return an integer between 0 to N. Before diving in, I decided to take the lazy path and check if Python's hash function is good enough.

Luckily, ipython notebook --pylab=inline makes that a breeze.

Check out the notebook here.

And yes, we decided to stick with this solution. I guess we're at least 1/3 programmers.

zipstream - Zip File InputFormat for Hadoop Streaming

At work, we store logs as a single CSV inside a zip file in HDFS (history, that's why :).

Looking around, I couldn't find a FileInput library that works with Hadoop streaming on CDH4 (the version we're using).

So I wrote one, hope you'll find it useful (you can download the jar directly from here.)

Here's an example how to use it:

Abusing namedtuple - Yet Another Enum

There's a discussion over at python-ideas about enum. This prompted me to write yet another implementation of enum, this time abusing namedtuple.

try lock

At work, we have several functions that can run only one at a time. We call this "try lock" (or trylock), and had it forever in the Java code.

When we started a Python project, we wanted this functionality. A decorator seems like the right solution. The below try_lock decorator has an optional function that lets you get a finer grained solution on what to lock. It gets the function arguments and returns a key to lock on. If you don't specify keyfn, then there will be just one lock for the function.

whoops - A WebHDFS Library and Client

Just released whoops 0.1.0 which is a WebHDFS library and a command line client for Python.

Timing Your Code

It's a good idea to time portions of your code and have some metric you monitor. This way you can see trends and solve bottlenecks before someone notices (hopefully). Timing functions is easy with decorators, but sometimes you want to time a portion of a function. For this you can use a context manager.

fastavro got snappy support

fastavro 0.7.1 introduces snappy support! (also 3.3 egg)

Last Letter Frequency

I was playing a game with my child where you say a word, then the other person need to say a word which starts with the last letter of the word you said, then you need to say a word with their last letter ...

We noticed that many words end with S and E, which made me curious about the frequency of the last letter in English words. matplotlib makes it super easy to visualize the results.

Python For Data Analysis

Just finished reading Python For Data Analysis, it's a great book with lots of practical examples. Highly recommended.