Reading Passwords of OSX Keychain

OSX stores passwords in its keychain, which is sometimes useful when you forget a password. The command line security utility lets you access the keychain, however the output is somewhat cryptic.

Below is a little script to get user and password for a given domain.

    find-pass.py signup.netflix.com

Note you might get an image like the image on the right. Click either "Allow" or "Always Allow".

Dealing with Bad Memory

The family joke is that I was born senile (my joke is that I have 1bit memory). During the years I've developed some methods to help me be be productive with bad memory. Hope they'll help you as well (if you remember them :).

JFDI
That's the most effective method - do it at the moment you remember. It's pretty amazing how many things you can do "right now" without interrupting your flow too much. Once you did it - there's no need to remember.

Make it Impossible to Forget
Yeah, writing things down help - but I forget to look at my lists. However if you make things impossible to forget - then you won't forget. For example if I need to take something to the car, I'll place it at the front door - can't miss it when I go out.

Get Help
May they be electronic or human, get some help. I married a wife with a phenomonal memory, but it's not an option for everyone :) In this digital age you can find a good non human assistent to help you. Hiring a human assitenet doesn't have to be expensive - see Fancy Hands for example. 

Write It Down
We have a saying: "A short pencil is better than a long memory.". Write things down, it'll help you remember when you write then and later you can look them up. I use this blog as an memory of things that worked for me. Other things I use are pinboard, trello, GMail and Google Docs. Everything with a search function in it.
 
Forgive Yourself
You will forget things, learn to live with it. One of the worst things you can do it agonize over the things you forgot. It'll only add stress to your life without helping you to remember.  

Removing "noise" before matplotlib charts in IPython notebooks

By default, IPython notebook prints the value of the last expression. This works fine most of the time but sometimes the result of a chart is a long list of lines which is something you don't want to see - you just want to see the chart. Here are two options to fix this (you can also view the example notebook).

Decorators and Context Managers Workshop

I gave a workshop on decorators and context mangers at work.

Both are very powerful and will change the way you code in Python once you grok them. They mostly allow you to focus on application logic and write the "book keeping" code (logging, timing, resource allocation ...) in a separate place.

Here are the IPython notebooks:

• Context Managers
• Decorators

And the solutions (but try to solve the exercises first):

• Context Managers
• Decorators

 If you want to run locally, the code is here.

Searching and Viewing emoji

We started using emoji at work. We're using Adium with the emoji xtra.

Here's a little script to search and optionally show emojis (requires matplotlib).

Making "pip" Faster

We use pip to install packages at work. Mostly in combination with virtualenv and wheel packages. Lately I had to re-install again and again a bunch of virtual environments and looks for a way to speed things up.

The solution was to cache the pip downloads and also to keep in a local directory the bigger .whl packages. All done in ~/.pip/pip.conf. Most of the downloads are done once now and using local .whl means no more compilation (I'm looking at you PyYAML ...). I didn't measure the time difference, but for sure it feels faster.

A Simple Python Based Configuration with Overrides

Here's a simple Python based configuration system that supports overrides. We use a variant of it at work and it proves to be very handy. The fact that you can use all of Python in your configuration is very powerful. We use Puppet to write config_local.py in production/qa environments and keep the default good for development machines.

Import Google Bookmarks to Pinboard

I decided to give Pinboard a try. Sadly they lack support for importing Google Bookmarks.

I wrote a little utility to do just that (still waiting for pinboard support to get back to me ;)

Python Logging Overview

Gave a talk at work about Python Logging.

You can view it here.

logbot

We have an internal Jabber server at work, which is great for communication. However unlike HipChat/Campfire - it lacks central logging.

A couple of hours with sleekxmpp, Whoosh and flask solved the problem. logbot is out there - use with caution. (And pull requests are more than welcome ;)

"crashlog" is now on PyPi

FYI: Decided to make crashlog a PyPi package.
LMK if you find any problems.

JSON Handling with datetime Support

While working on some ETL testing code, we needed to load some synthetic data to a database. At first we thought of using YAML for the synthetic data, but it was very slow so we decided to switch to JSON which is still readable but way faster (loading 183 records took 0.68sec with YAML vs 0.008sec in JSON).

However we needed support for serializing datetime objects. Here's dtjson, which support datetime objects in JSON serialization. You can use it almost a drop-in replacement for json: import dtjson as json.

Adding ODBC Source on OSX

At work, we're using Netezza as our main data warehouse. It took me a while to figure out how to add the ODBC driver on OSX so that I'll be able to connect with pyodbc.

Here are the steps:

• Get the OSX Netezza ODBC driver and unpack it
• Not sure where do get it, we have it internally
• Extract the archive somewhere (say /opt/NetezzaODBCDriver)
•  Add  export DYLD_LIBRARY_PATH="${DYLD_LIBRARY_PATH}":/opt/NetezzaODBCDriver/lib/ to your ~/.zshrc (or ~/.bashrc)
• mkdir -p ~/Library/ODBC
• cp /opt/NetezzaODBCDriver/ini/*.ini ~/Library/ODBC
• Edit ~/Library/ODBC/odbcinst.ini to reflect were your driver is (Driver and Setup keys)
• In our example it'll be Driver = /opt/NetezzaODBCDriver/lib/libnzodbc.so (same for Setup) 

That's it, you can now connect to Netezza from OSX.

Here's a small example usage:

Don't Forget Your Process When Selecting Tools

When selecting tools, there are many things you should consider - prices, value added, integration cost, health ...

However many teams forgot to check how well does the tool integrate with your current development process. A good tool that does not play well with your process will cause many problem and might alter your process a direction you don't want.

For example, we have a process (like many others probably) which involves code reviews. We have a tool with does not play well with code reviews - it has binary projects. The process with this tool is different, much slower and more error prone than the rest of the tools.

Next time you pick a tool to work with, think how well will it play with your current process and take it into consideration.

Note: It might be OK to change the process if the tool is worth it, just make sure it does before you integrate it.

Advice For New Managers

Here's some of the advice I give new managers. I try to talk to them before they start their first management job - let me be the first to ruin their career :)

Disclaimer: I'm not a manager, but I was managed by many and was an officer for many years in the IDF which is very informal kind of army.

Here they are, without any particular order.

Style

Every manager has their own style. From fire-and-forget to micromanagement and everything in the middle. Your style will probably be different from another manager (even from your own), your management style will change over time and from team to team. Play to your strengths - organizational, people skills, ....

Don't be afraid to try until you find a style that both you and your team are comfortable in. As long as you're open about what you're doing, you team will support you. It's hard to measure the effectivenehess of each style, but if you can - do that.

Let Go

You are a manager now, not a developer. I've seen very few managers in my long career that managed to do both effectively. Learn to trust your team and be a manager first and developer last.

Think about the things you liked and disliked in your past managers (and the current one) and figure out what you should do.

Manage Your Time

Management is very different from development, it's much more interruption driven. However you need to allocate time for yourself. I've seen the best officers in the army taking up to 30 minutes to think while all hell breaking loose around them (number 2 was in charge in the meanwhile). If they can do it - I'm sure you can.

Here's a great talk by John Cleese (of the Monty Python fame) about creativity which covers this topic as well.

Know Your Place

In officer training over at IDF, they teach you that an officer should be where he/she's most needed. Same goes for you - think about what you are the most effective at solving the more important things and be there.

Develop A Mantra

Find a sentence which will help you focus, and ask it repeatedly throughout the day. I found out that "Why aren't we deploying?" help our team focus and deliver. However find one that fits your goals/team/company ....

You Team Comes First

If you show your team they come first (mostly by "buffering" them from all the management noise), they will be loyal to you. Loyalty works both ways or it doesn't work.A loyal and jelled team is hard to build but when you get there - they will rock.

That's about it. Don't be nervous and try to enjoy a new position. In the worse case know that management is not for everybody (I personally don't like it). There's no shame in saying "this is not for me" and getting back to to fun stuff. (My brother did just that, he managed for a year and then said to the company - "Either I'm back to development or I'm out". He was back in development).

I hope that you found this useful.

Vi Keybindings in SBCL REPL

I'm (re-)reading "Practical Common Lisp" and going over the exercises/code with SBCL. I found the below script to help me with my Vi addiction :)

Error Handling in Python

I gave a talk at work about error handling in Python.
You can view it here.

What do you think? Did I miss something?

It's worth noting that after a while working with Go and the way it handles errors (if value, err = foo(); err != nil ...). If find my Go programs more verbose but more robust at the same time.

XFCE and Volume Keys

I'm currently playing around with XFCE. I have PulseAudio and wanted the volume keys to work (I'm a keyboard kinda guy. :). Here's a solution:

• Open Settings -> Keyboard -> Application Shortcuts
• Click on "Add"
• Type pactl set-sink-volume 0 +10% and then OK
• Hit the "up volume" button on your keyboard
• Click on "Add" (again)
• Type pactl set-sink-volume 0 -- -10% and then OK
• The -- is not a typo
• Clock on "Add"

Your volume keys sound work now.

Oh, and I find installing they "Greybird" theme [1] (from Ubuntu) makes things much nicer :)

[1] Install with yaourt -S xfce-theme-greybird and then select in Settings -> Appearance

Google Bookmark Command Line Utility

A little utility to save URLs in Google Bookmarks. (This will open a web browser with some fields populated).

IPython on OSX (with iTerm2)

Sadly, in the new workplace I have to use OSX (it's either that or Windows, running Linux and connecting to everything is a ... challenge).

Here's a little script I wrote to open IPython with a new iTerm2 window. I call this script ipy and place it at ~/bin. Then hitting APPLE+SPACE launches Spotlight. Writing "ipy" inside and hitting ENTER does the trick.

If you want VIm keybindings, add the following to ~/.inputrc

set editing-mode vi
tab: complete
set completion-ignore-case on
set blink-matching-paren on

Oh, and if you find out how to maximize the window - please share :)

Command line front end to Jinja2

Below is a little command line front end to the Jinja2 templating library.
(Very much in the spirit of the mako utility).

Want To Be More Productive? - Improve Your I/O

It always surprise me that people trying to makes themselves more productive overlook their I/O - reading and writing.

I still see many people slowly and inefficiently reading on the input side and pecking their way on the keyboard in the output side.

I guess the reason that people don't invest time in improving their I/O is that it's boring. I'll be the first to agree that learning speed reading or touch typing is boring, however the benefits you'll reap are big - the investment will pay itself every time you interact with your computer.

After getting the basics of speed reading and touch typing out of the way [1], there are many more ways you can improve your I/O. Here are few examples:

• Learn to efficiently skim through documents
• Learn to pick what to read
• Master your IDE
• Automate everything
• ...

[1] And you should never stop practicing.

Running "one off" Conversions In PostgreSQL

The other day we needed to runs a one-off job for converting datetime.weekday() values to day names. Here's what we came up with.

Getting Good Errors from Python Map/Reduce Jobs

At work, we use some Python map/reduce jobs (using Hadoop streaming).

Debugging can be difficult, since Hadoop does not keep the Python stacktraces and even if it does - it's very hard to find it. We decided to use crashlog, and now we get wonderful emails with detailed description of what went wrong.

Notes:

• The current mapper input file is in map_input_file environment variable
• Don't forget to add crashlog.py with -file (see here)
• You must add "." to PYTHONPATH in order to import
• import sys; sys.path.append('.') should do the trick
• Email is not the best solution for distributed logging (you get a lot of email when things go South). I'm going to play with graylog2 in the future.

Serving Dynamic Images with matplotlib

Here's an example of generating dynamic images using matplotlib in a web server (flask this time).

Show Location Of Hive Table in HDFS

Here's a script I use to find out where Hive is storing it's data on HDFS (I call it hiveloc).

Quickly Plotting Labeled Data

Here's a quick way to view some labeled data you have (taken from An Introduction to scikit-learn). It will reduce the data to two dimensions using PCA and then scatter plot it with different colors for each label.

Script to close a branch in mercurial (hg)

Mercurial (hg) does not let you delete branches (or alter history in any way). But you can close branches so they won't show in hg branches command.

Here's a script I use to close branches (we work with feature branches at work, and close them when work on the feature is done).

gittip on bitbucket/github

gittip is a cool idea, however currently there's no built-in way to add it to bitbucket/github projects.

One option I found that works it to add a clickable image to your README.md or README.rst.

See example here.

Markdown:

[](https://www.gittip.com/Miki%20Tebeka/)

ReStructuedText:

.. image:: http://i.imgur.com/lg9rx9w.png
   :alt: gittip
   :target: https://www.gittip.com/Miki%20Tebeka/

Notes:

1. You'll probably want to change gittip user id :)
2. There's a discussion on gittip bug tracker on the right way to do this.
3. Unofficial gittip image generated using cooltext.

import "C" slides

Last night I gave a talk about using C from Go at the L.A. Gophers meetup.

You can view the slides here. (Note that "run" won't work due to security restrictions, you can download the slides here and run it locally using the present tool).

Investigating Hash Distribution

A college asked me for a hash function on strings that return an integer between 0 to N. Before diving in, I decided to take the lazy path and check if Python's hash function is good enough.

Luckily, ipython notebook --pylab=inline makes that a breeze.

Check out the notebook here.

And yes, we decided to stick with this solution. I guess we're at least 1/3 programmers.

zipstream - Zip File InputFormat for Hadoop Streaming

At work, we store logs as a single CSV inside a zip file in HDFS (history, that's why :).

Looking around, I couldn't find a FileInput library that works with Hadoop streaming on CDH4 (the version we're using).

So I wrote one, hope you'll find it useful (you can download the jar directly from here.)

Here's an example how to use it:

Abusing namedtuple - Yet Another Enum

There's a discussion over at python-ideas about enum. This prompted me to write yet another implementation of enum, this time abusing namedtuple.

try lock

At work, we have several functions that can run only one at a time. We call this "try lock" (or trylock), and had it forever in the Java code.

When we started a Python project, we wanted this functionality. A decorator seems like the right solution. The below try_lock decorator has an optional function that lets you get a finer grained solution on what to lock. It gets the function arguments and returns a key to lock on. If you don't specify keyfn, then there will be just one lock for the function.

whoops - A WebHDFS Library and Client

Just released whoops 0.1.0 which is a WebHDFS library and a command line client for Python.