gittip on bitbucket/github

gittip is a cool idea, however currently there's no built-in way to add it to bitbucket/github projects.

One option I found that works it to add a clickable image to your README.md or README.rst.

See example here.

Markdown:

[](https://www.gittip.com/Miki%20Tebeka/)

ReStructuedText:

.. image:: http://i.imgur.com/lg9rx9w.png
   :alt: gittip
   :target: https://www.gittip.com/Miki%20Tebeka/

Notes:

1. You'll probably want to change gittip user id :)
2. There's a discussion on gittip bug tracker on the right way to do this.
3. Unofficial gittip image generated using cooltext.

import "C" slides

Last night I gave a talk about using C from Go at the L.A. Gophers meetup.

You can view the slides here. (Note that "run" won't work due to security restrictions, you can download the slides here and run it locally using the present tool).

Investigating Hash Distribution

A college asked me for a hash function on strings that return an integer between 0 to N. Before diving in, I decided to take the lazy path and check if Python's hash function is good enough.

Luckily, ipython notebook --pylab=inline makes that a breeze.

Check out the notebook here.

And yes, we decided to stick with this solution. I guess we're at least 1/3 programmers.

zipstream - Zip File InputFormat for Hadoop Streaming

At work, we store logs as a single CSV inside a zip file in HDFS (history, that's why :).

Looking around, I couldn't find a FileInput library that works with Hadoop streaming on CDH4 (the version we're using).

So I wrote one, hope you'll find it useful (you can download the jar directly from here.)

Here's an example how to use it:

Abusing namedtuple - Yet Another Enum

There's a discussion over at python-ideas about enum. This prompted me to write yet another implementation of enum, this time abusing namedtuple.

try lock

At work, we have several functions that can run only one at a time. We call this "try lock" (or trylock), and had it forever in the Java code.

When we started a Python project, we wanted this functionality. A decorator seems like the right solution. The below try_lock decorator has an optional function that lets you get a finer grained solution on what to lock. It gets the function arguments and returns a key to lock on. If you don't specify keyfn, then there will be just one lock for the function.

whoops - A WebHDFS Library and Client

Just released whoops 0.1.0 which is a WebHDFS library and a command line client for Python.

Timing Your Code

It's a good idea to time portions of your code and have some metric you monitor. This way you can see trends and solve bottlenecks before someone notices (hopefully). Timing functions is easy with decorators, but sometimes you want to time a portion of a function. For this you can use a context manager.

fastavro got snappy support

fastavro 0.7.1 introduces snappy support! (also 3.3 egg)

Last Letter Frequency

I was playing a game with my child where you say a word, then the other person need to say a word which starts with the last letter of the word you said, then you need to say a word with their last letter ...

We noticed that many words end with S and E, which made me curious about the frequency of the last letter in English words. matplotlib makes it super easy to visualize the results.

Python For Data Analysis

Just finished reading Python For Data Analysis, it's a great book with lots of practical examples. Highly recommended.

Mocking HTTP Servers

Sometimes, httpbin is not enough, and you need your own custom HTTP server for testing.
Here's a small example on how to do that using the built in SimpleHTTPServer (thanks @noahsussman for reminding me).

http://httpbin.org

Sometimes you need to write an HTTP server to debug the client you are writing.

One quick way to avoid this is to use http://httpbin.org/. It supports most of the common HTTP verbs and mostly return the variables you send in.

For example (note the args field in the reply):

$ curl -i 'http://httpbin.org/get?x=1&y=2'
HTTP/1.1 200 OK
Content-Type: application/json
Date: Mon, 15 Oct 2012 21:50:27 GMT
Server: gunicorn/0.13.4
Content-Length: 386
Connection: keep-alive

{
  "url": "http://httpbin.org/get?x=1&y=2",
  "headers": {
    "Content-Length": "",
    "Connection": "keep-alive",
    "Accept": "*/*",
    "User-Agent": "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3",
    "Host": "httpbin.org",
    "Content-Type": ""
  },
  "args": {
    "y": "2",
    "x": "1"
  },
  "origin": "75.82.8.111"
}

Cleanup After Your Tests - But Be Lazy

It's a nice practice to clean after your tests. It's good for various reasons like disk space, "pure" execution environment and others.

However if you clean up to eagerly it'll make your debugging much harder. The data just won't be there to see what went wrong.

The solution we found is pretty simple:

• Try to place all your test output in one location
• Nuke this location when starting the tests

This way all the information is available after an error, and you don't accumulate too much junk (just one test run junk at a time).

Data Wrangling With Python

I just gave a talk at work called "Data Wrangling With Python" which gives an overview on the scientific Python ecosystem. You can view it here.

Using Hadoop Streaming With Avro

One of the way to use Python with Hadoop is via Hadoop Streaming. However it's geared mostly toward text based format and at work we use mostly Avro.

Took me a while to figure the magic, but here it is. Note that the input to the mapper is one JSON object per line.

Note it's a bit old (Avro is now at 1.7.4), originally from here.

Setting Maching Learning on OSX

Setting up machine learning tools (numpy, scipy, matplotlib, scikit-learn, ...) can be a pain (why can they just use a decent OS? :).

We are lucky to have Ben Kim now with us at Adconion, and he posted the following:

Mac OS X Lion Software Installs

1. Install compilers
   1. Install XCode 4.x from the App Store
      1. Install Command Line Tools in Preferences/Download
   2. Install gcc, g++, and gfortran compilers
      1. Download tar file
      2. Extract to /
         1. tar -xvf abc.tar -C /
   3. Reference http://sites.google.com/site/dwhipp/tutorials/mac_compilers
2. Install Homebrew
   1. Run the install command using ruby

      1. ruby <(curl -fsSkL raw.github.com/mxcl/homebrew/go)

   2. brew doctor
      1. chown /usr/local folders listed
      2. Place /usr/local/bin before /usr/bin in path
   3. Reference https://github.com/mxcl/homebrew/wiki/installation
3. Install python using brew
   1. brew install readline sqlite gdbm pkg-config
   2. brew install python
   3. Note: Mac OS X Lion comes with old version 2.7.1 of python (python --version) 
4. Set PATH in .bash_profile
   1. vim ~/.bash_profile
      1. export PATH=/usr/local/share/python:/usr/local/bin:$PATH
5. Create symlinks
   1. Within /(System/)?Library/Frameworks/Python.framework/Versions, sudo rm Current
   2. Within the above directories, ln -s /usr/local/Cellar/python/2.7.3 Current
6. Install pip, if necessary, using easy_install
   1. sudo easy_install pip
7. Using pip (sudo pip install [--upgrade] abc)
   1. Install nose
   2. Install numpy
   3. Install scipy with environmental variables settings
      1. sudo CC=clang CXX=clang FFLAGS=-ff2c pip install [--upgrade] scipy
   4. Install scikit-learn
   5. Install pandas
8. Install matplotlib
   1. Download source from repo: https://github.com/matplotlib/matplotlib
   2. cd ~Downloads/matplotlib-*
   3. python setup.py build
   4. python setup.py install
9. Install VW (Vowpal Wabbit)
   1. Install boost
      1. Download tar file
      2. mv boost extracted folder to /usr/local
      3. export BOOST_ROOT environmental variable
      4. cd to boost directory
      5. make and install
         1. sudo ./bootstrap
         2. sudo ./bjam install
      6. Download bjam
      7. mv to directory in PATH
         1. mv bjam /usr/local/bin
      8. Set bjam toolset to darwin
         1. bjam toolset=darwin stage
      9. Reference http://www.boost.org/doc/libs/1_41_0/more/getting_started/unix-variants.html#expected-build-output
   2. cd to VW directory
      1. make and test
         1. make
         2. make test

A "cd" context manager

Packages Download Url

Inspired by StackOverflow question:

Show Dependecies of Azkaban Jobs

We're using Azkaban at work to schedule Hadoop jobs. It's hard to view the job dependencies without deploying, so here's a little script that will show you job dependencies as an image. It uses dot (from Graphviz) to produce the image.