"Manual" Breakpoints in Go

When debugging, sometimes you need to set conditional breakpoints. This option is available both in gdb and delve. However sometimes when the condition is complicated, it's hard or even impossible to set it. A way around is to temporary write the condition in Go and set breakpoint "manually".

I Python we do it with pdb.set_trace(), in Go we'll need to work a little harder. The main idea is that breakpoints are special signal called SIGTRAP.

Here's the code to do this:
You'll need tell the go tool not to optimize and keep variable information:

$ go build -gcflags "-N -l" manual-bp

Then run a gdb session

$ gdb manual-bp 

(gdb) run 

 When you hit the breakpoint, you'll be in assembly code. Exit two functions to get to your code

(gdb) fin
(gdb) fin

Then you'll be in your code and can run gdb commands

(gdb) p i
$1 = 3

This scheme also works with delve

$ dlv debug manual-bp.go 
(dlv) c 

Sadly delve don't have "fin" command so you'll need to hit "n" (next) until you reach your code. 

That's it, happy debugging.

Oh - and in the very old days we did about the same trick in C code. There we manually inserted asm("int $3)" to the code. You can do with with cgo but sending a signal seems easier.

Generate Relation Diagram from GAE ndb Model

Working with GAE, we wanted to create relation diagram from out ndb model. By deferring the rendering to dot and using Python's reflection this became an easy task. Some links are still missing since we're using ancestor queries, but this can be handled by some class docstring syntax or just manually editing the resulting dot file.

Testing numpy Code

Notebook here.

Work with AppEngine SDK in the REPL

Working again with AppEngine for Python. Here's a small code snippet that will let you work with your code in the REPL (much better than the previous solution).
What I do in IPython is:

In [1]: %run initgae.py

In [2]: %run app.py

And then I can work with my code and test things out.

Using ImageMagick to Generate Images

One of the exercises we did this week in the Python workshop used the term "bounding box diagonal". I had a hard time to explain it to the students without an image. Google image search didn't find anything great, so I decided to create such an image.

First I tried with drawing programs, but couldn't make the rectangle a square and the circle non-oval. Then I remembered imagemagick, I have it installed and mostly use it to resize images - but it can do much more. A quick look at the examples and some trial and error, and here's the result.

And here's the script that generated it:

Waiting for HTTP Server - Go Testing

I don't like mocking in tests. If I have a server to test, I prefer to start and instance and hit the API in my tests. Waiting for the server to start with a simple sleep is unpredictable. I prefer to start the server, try to hit an URL until it's OK or fail after a long timeout. This is a simple task with Go's select and time.After.

Slap a --help on it

Sometimes we write "one off" scripts to deal with certain task. However most often than not these scripts live more than just the one time. This is very common in ops related code that for some reason people don't apply the regular coding standards to.

It really upsets me when I try to see what a script is doing, run it with --help flag and it happily deletes the database while I wait :) It's so easy to add help support in the command line. In Python we do it with argparse, and we role our own in bash. Both cases it's extra 3 lines of code.

Please be kind to future self and add --help support to your scripts.

vfetch - Fetch Go Vendor Depedencies

Go 1.6 now supports vendoring. I found myself cloning dependencies to "vendor" directory, then cloning their dependencies ... This got old really fast so vfetch was born. It's a quick and dirty solution, uses "go get" with a temporary GOPATH to get the package and its dependencies, then uses rsync to copy them to the vendor directory.

Installing is the usual "go get github.com/tebeka/vfetch" then you can use "vfetch github.com/gorilla/mux".

Comment, ideas and pull requests are more than welcomed.

Super Simple nvim UI

I've been playing with neovim lately, enjoying it and a leaner RC file. nvim comes currently only in terminal mode and I wanted a way to spin a new window for it. Here's a super simple script (I call it e) to start a new xfce4-terminal window with nvim.

Removing String Columns from a DataFrame

Sometimes you want to work just with numerical columns in a pandas DataFrame. The rule of thumb is that everything that has a type of object is something not numeric (you can get fancier with numpy.issubdtype). We're going to use the DataFrame dtypes with some boolean indexing to accomplish this.

In [1]: import pandas as pd  

In [2]: df = pd.DataFrame([
   ...:     [1, 2, 'a', 3],
   ...:     [4, 5, 'b', 6],
   ...:     [7, 8, 'c', 9],
   ...: ])  

In [3]: df  
Out[3]: 
   0  1  2  3
0  1  2  a  3
1  4  5  b  6
2  7  8  c  9

In [4]: df.dtypes  
Out[4]: 
0     int64
1     int64
2    object
3     int64
dtype: object

In [5]: df[df.columns[df.dtypes != object]]
Out[5]: 
   0  1  3
0  1  2  3
1  4  5  6
2  7  8  9

In [6]:   

Forging Python - First Chapter is Up

Finally, first chapter of my upcoming book "Forging Python" is up. I'm doing it leanpub style so comments ans suggestions are more than welcomed.

I plan to finish the book this year, hopefully during the summer. However more than one person said I'm way too optimistic - time will tell :)

353Solutions - 2015 in Review

Happy new year!

First full calendar year that 353solutions is operating. Let's start with the numbers and then some insights and future goals.

Numbers

• 170 days of work in total
• Work day is a day where I billed someone for some part of it
• Can be and hour can be 24 hours (when teaching abroad)
• There were total of 251 work days in 2015
• There were some work days that are not billable (drafting syllabuses, answering emails ...) but not that many
• 111 of days consulting to 4 clients
• 1st Go project!!!
• 58 days teaching 14 courses
• Python at all levels and scientific Python (including new async workshop)
• In UK, Poland and Israel

Insights

• Social network provided almost all the work
• Keep investing in good friends (not just for work :)
• Workshops pay way more than consulting
• However can't work from home in workshops
• Consulting keeps you updated with latest tech
• Had to let go of a client due to draconian contract
• No regrets here, it was the right decision
• Super nice team. Sadly lawyers had final say the company
• Python and data science are big and in high demand
• Delegating overhead to the right person helps a lot
• Accounting, contracts ...

Future Goals

• Keep positioning in Python and Scientific Python area
• Drive more Go projects and workshops
• Works less days, have same revenue at end of year
• Start some "public classes" where we rent a class and people show up
• Some companies don't have big enough data science team
• Need to invest in advertising
• Publish my book (more on that later)

Using HAProxy to Prevent Deletes from Elasticsearch

At one of my clients, we wanted something quick and dirty to prevent deletes from Elasticsearch (shield is too expensive and would take too much time to integrate with our systems - we'll fix this technical debt later).

The quick solution was to place HAProxy in front of Elasticsearch and use its acl mechanism to prevent HTTP DELETE. Works like a charm.

Here's the HAProxy configuration and the docker-compose setup file I used to test the configuration.

Python's deque for Go

Working on a Go project with my friend Fabrizio, I've investigated ways to have a faster data structure to store history items with append and pop.

Got the idea to try implementing Python's deque in Go. The C implementation is pretty easy to read. The result is deque for Go, which implement a subset of the features from Python's deque but enough for our needs. And it's pretty fast too:

$ make compare
Git head is 765f6b0
cd compare && go test -run NONE -bench . -v
testing: warning: no tests to run
PASS
BenchmarkHistAppend-4  3000000        517 ns/op
BenchmarkHistList-4    2000000        702 ns/op
BenchmarkHistQueue-4   3000000        576 ns/op
BenchmarkHistDeque-4   3000000        423 ns/op
ok   _/home/miki/Projects/go/src/github.com/tebeka/deque/compare 8.505s

aenumerate - enumerate for async for

Python's new async/await syntax helps a lot with writing async code. Here's a little utility that provides the async equivalent of enumerate.

git - Creating Pull Request for master

A co-worker asked me for a code review (we're using Stash, but this can work for other systems as well), the problem was that he worked on master (started his own project) and not in development branch. The solution was to create an empty orphan branch and then a pull request from master to that branch (reverse the usual order).

Here's how to create such branch.

Go Tour Exercise Solutions

As a backup plan for the last Go Meetup, I wrote the solutions to the exercises in Go Tour and we discussed some of them.

You can find the solutions here.

re2 available on conda

We're using re2 to get some speed gains on the many regular expressions we're trying to match. So far building it was either a manual step or a script that ran when building docker container. I decided to create a conda package (we're using Miniconda as our Python environment).

I started with conda skeleton pypi re2 (you need to conda install conda-build first). Then after some tweaking to build.sh we were good to go.

The result - you can now conda install -c tebeka re2 (only 64bit linux supported currently).

The project is here, I'll gladly accept any comments/improvements.

Here's build.sh which patches re2 Makefile and added the library and header location to the Python build step.

fastavro moved to github

If you can't beat them ... :)

fastavro is now on github. I still prefer mercurial as an SCM but most of the pull requests I get are on github and it doesn't worth the effort of maintaining two repositories (though hg-git is a big help)

dockermon - A Docker Event Monitor

I'm currently working with the awesome team at CyberInt (and yes, they are hiring).

We're moving to a docker based environment. The old environment used Supervisor to monitor and relaunch daemons. We had an event listener that notified us on our HipChat room every time a daemon crashed and wanted the same feature with our docker containers.

We didn't find a ready solution, so we wrote one and made it open source. The project is called dockermon and is one Python script with no external dependencies and also Python 2 and 3 compatible.

Naming "with open" Variable

Python''s "with" statement is great for resource handling. However I find my self struggling with naming (and naming is important) the context manager variable.

When you write "with open(''/path/to/somethere'') as X", what''s the best name for X? In some cases it''s obvious, but in most cases I find myself using the generic "fo" (stands for "file object").

I decided to run a little script on Python''s 3.4 Lib directory and find out what is the most common name. Here are the results:

Seems like f is the most common, but I really don''t like single letter variables. I''ll go with the 2nd place - fp.

Here''s the script used to generate this chart:

353Solutions - A Year in Review

353Solutions was founded a bit more than a year ago. I wasn't planning on doing consulting, I'm a techie and love the development abstraction layer that companies give you and let you code most of the time. (If this is not the case in the company you're working at - consider finding a better one :)

However as the old saying goes - "Man plans and god laughs". I found myself owning a teaching/consulting company called 353Solutions. So far it's fun and provides for the family - what else can you ask for?

Here are results from a short retrospective we did lately.

The Numbers

• 6 clients
• 204 work days
• 204 hours teaching Python (7 courses)

Thoughts

I like working from home, however most companies I talked to wanted some office time. This is understandable since I don't only code but also do system and process design - these roles require more face to face communication. I'm still looking for something that will allow me to spend most of my time working from home.

Teaching is fun! I did that on and off most of my professional carrier, but now it's a big chunk of my time. I'm grateful to Raymond Hettinger who started me off and showed me what a top-notch class/workshop should look like. So far I'm mostly going to companies and teaching there, but just now we launched our own classes - it will  be awesome!

The downside for teaching is that it takes me away from home. For a limited amount this is great (I spend about a week every month teaching Python in the UK). However I'm looking for opportunities that will let me teach from home - stay tuned.

The social network is by far my biggest source of new jobs. Talking to other people - it's not just me. Investing time in making connections and keeping them will pay off. The main downside for is that people want to hire me and not 353Solutions. This means I need to work harder to market the other people who I work with - I can't do everything.

Learning to say "no" was the hardest thing for me. So many interesting things to do, so many cool companies ... But I like spending time with my family, friends and hobbies. You need to find the things that make you happy and pay enough, going cheap is not a good thing in most cases. What I did in some cases was to take less money and get equity instead. Something like "technical" investing in startups.

The main point I need to improve is marketing. It's not something I like to do but feel the need, especially now that we have our own classes. I'm learning and looking for the best thing that will get maximal impact with minimal amount of time. Or maybe hire someone for that? If you know a good option  - please let me know :)

Use contextlib.closing to Handle "Legacy" Resources

Python''s context managers (with statement) are very handy at handling resources. (You see way less finally in Python code due to them). Maybe objects in Python can be used as context mangers - files, locks, database drivers and more. But some objects still do not.

To handle these "legacy" objects you can use contextlib.closing function which will return a context manager that will call obj.close() once the context manager exists.

Here''s an example of using contextlib.closing with sockets. We''ll be doing a simple HTTP request (Yeah, you should probably use requests or urlopen - this is just an example :)

Note also the user of iter with sentinel to read chunks up to 1K from the socket.

Combining jQuery and Multi Components of React

React is a great library for generating reactive web UI (and mobile). Reacts works well if there are isolated components or a big one with hierarchy. However I wanted to have a page with several isolated react components that are updated from the same data. The solution I found is to use an observer pattern and have each components register a callback to handle data change. See the code below and a live demo here.

Note that I am a React newbie, if you know of a better way - please enlighten me.

Solving Project Euler Problem 8 with numpy

I''m teaching a course in scientific Python these days. Usually I give "homework" from project Euler (which I personally use every time I learn a new programming language). I thought it''ll be fun to solve the problem not just with Python but with what numpy has to offer as well.

Here''s an example solution for project Euler problem 8.

Docker + MiniConda = A Perfect Match

Working with one of my clients (who is hiring BTW), we decided to use Docker as deployment platform. Since many Linux systems now use Python for many utilities, it''s advisable to install your own Python next to the system one and use it.

Installing CPython from source requires some system packages, libraries, headers and some knowledge. The much easier path it to use MiniConda (from the wonderful people at Continuum). Not only the Python installation is super simple, but also the conda package manger will get you a lot of packages pre-compiled so you don''t have to install gcc and header files for C extensions. And if you can''t find the package you need with conda, pip is also available.

Here''s a little project to demonstrate how to do this. The application is an image server with has two entry points /edge for edge detection and /resize for image resizing. We''ll be using scikit-image and Pillow for image manipulation and Flask as web server. All of them can be conda installed.

Here''s the Dockerfile for the project. Build with docker build -tag imgsrv, Run with docker run -p 8080:8080 imgsrv (see Makefile).

Adding Server SSL Certificate on Linux

Here''s a small script to add a server SSL certificate on Linux. You can export the certificate from your browser. Inspired by this stackoverflow post.

Logging from Celery to logstash and a structured log (JSON)

I love Celery, and we''re using it at one of my customers. One thing we wanted to have is centralized logging, since you can have multiple workers on multiple machines. We looked at several solutions and the winner came out to be logstash + kibana (AKA ELK stack).

Here''s some code to log to logstash with Celery current task information (if available) and also to a structured log (every line is a JSON object) for backup in case of network issues.

Using supervisord to Manage You Daemons

Say you have some daemons running. You''d like to restart them automatically if they fail, grab logs from them and in general manage them - supervisord will do it all for you.

One of my (super cool) clients needed also to start/stop daemons when configuration changes. The solution was to have a script that updates supervisord.conf every time we have a configuration change and then selectively start/stop on the daemons that have change (by default if your run supervisorctl update, it will restart all the daemons).

For this example, I''ll assume that the daemon processes are python -m SimpleHTTPServer and I have a list of port I''d like to listen on. This list of ports might change.

Example Usage

    $ ./updated.py 8000 8001 8002

    $ supervisorctl status

    httpd-8000                       RUNNING   pid 31768, uptime 0:00:04

    httpd-8001                       RUNNING   pid 31767, uptime 0:00:04

    httpd-8002                       RUNNING   pid 31766, uptime 0:00:04

    $ ./updated.py 8000 8004 8002 # Remove 8001, add 8004

    httpd-8001: disappeared

    httpd-8004: available

    httpd-8001: stopped

    httpd-8001: removed process group

    httpd-8004: added process group

    $ supervisorctl status

    httpd-8000                       RUNNING   pid 31768, uptime 0:00:12

    httpd-8002                       RUNNING   pid 31766, uptime 0:00:12

    httpd-8004                       RUNNING   pid 31785, uptime 0:00:02

    $

    

python -m

python -m lets you run modules as scripts. If your module is just one .py file it'll be executed (which usually means code under if __name__ == '__main__'). If your module is a directory, Python will look for __main__.py (next to __init__.py) and will run it.

One of Python's mottoes is "batteries included", and this goes for python -m as well. Here are some (all?) of the gems hidden in the standard library. Sadly not all of them have help, but I poked around in the source code to see the usage.

json.tool

This is by far the one I use most, it'll indent nicely an JSON input in the standard output and very helpful in combination with curl.

$ curl -sL http://j.mp/1IuxaLD

[{"x":1,"y":2},{"x":3,"y":4},{"x":5,"y":6}]

$ curl -sL http://j.mp/1IuxaLD | python -m json.tool

[

    {

        "x": 1,

        "y": 2

    },

    {

        "x": 3,

        "y": 4

    },

    {

        "x": 5,

        "y": 6

    }

]

zipfile

zipfile will let you view, extract and create zip files - very much like the zip and unzip. Here's the help:

$ python -m zipfile -h

Usage:

    zipfile.py -l zipfile.zip        # Show listing of a zipfile

    zipfile.py -t zipfile.zip        # Test if a zipfile is valid

    zipfile.py -e zipfile.zip target # Extract zipfile into target dir

    zipfile.py -c zipfile.zip src ... # Create zipfile from sources

gzip

Like zipfile, let's you compress and decompress .gz files, like gzip/gunzip. By default it'll compress a file but with -d will decompress.

python -m gzip wordlist.txt  # Will create wordlist.txt.gz

python -m gzip -d wordlist.txt.gz  # Will extract to wordlist.txt

filecmp

Compare two directories.

$ python -m filecmp /tmp/a /tmp/b

diff /tmp/a /tmp/b

Only in /tmp/a : ['1']

Only in /tmp/b : ['2']

Identical files : ['4']

Differing files : ['3']

Encode/Decode

Several modules lets you encode/decode in various formats:

• base64
• uu
• encodings.rot_13
• binhex
• mimify
• quopri

For example

$ echo 'secertpassword' | python -m encodings.rot_13

frpregcnffjbeq

Servers

There are several servers that you can run, the ones I know are SimpleHTTPServer, CGIHTTPServer and smtpd (mail). If you quickly want to serve some files from a directory on your machine, just run:

python -m SimpleHTTPServer

Clients

Modules that provide simple clients to various protocols are:

• ftplib
• poplib
• nntplib
• smtplib (on localhost only)
• telnetlib

For example if you want to view Star Wars in text mode, do

$ python -m telnetlib towel.blinkenlights.nl

System Info

You can use platform to get some platform information (very much line uname -a) and locale to get locale information. Use mimetype to get the mime type of a file:

$ python -m mimetypes doc.html

type: text/html encoding: None

Python Utilties

• compileall will compile all Python files to .pyc
• dis will show bytecode for a file
• pdb will start the Python debugger on a given file (see here)
• pydoc will show documentation on a module/class/function
• site will print some site information (sys.path, USER_BASE ...)
• sysconfig will show many system related information (such as exec_prefix)
• tabnanny will tell you of you mix tabs and spaces (like starting python with -t or -tt)
• tokenize will print list of tokens in Python file

I mostly use pdb and pydoc, for example:

$ python -m pydoc os.remove

Help on built-in function remove in os:

os.remove = remove(...)

    remove(path)

    

    Remove a file (same as unlink(path)).

Profiling

There are several profiles and timers you can use from the command line:

• cProfile - Show profile information
• profile (use cProfile :)
• timeit - Time how long things run
• pstats - Print output of profiles
• trace - Show tracing information on run

Example:

$ python -m cProfile match.py

         28537 function calls (27503 primitive calls) in 0.057 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)

        1    0.000    0.000    0.000    0.000 :1()

        1    0.000    0.000    0.000    0.000 :1(ArgInfo)

        1    0.000    0.000    0.000    0.000 :1(ArgSpec)

   ...

$ pyton -m timeit 'import math; math.factorial(100)'

100000 loops, best of 3: 12.9 usec per loop

timeit has good help from the command line.

IDLE

You can start IDLE by running python -m idlelib.idle

ensurepip

Python 2.7.9 and 3.x comes with an easy way to install pip. Run python -m ensurepip and pypi is at your service.


That's about it ... What are you favorite python -m tools? Which ones did I miss?

EDIT: The good folks at comp.lang.python reminded me a few I forgot:

unittest

python -m unittest discover will run unittest in discovery mode. Just drop a new Python file starting with test and it'll be picked up next time you run the tests. You can also specify a specific test to run with python -m unittest test_file.py TestClass.test_method.

calendar

python -m calendar will show calendar of the current year. You can also run python -m calendar YEAR to display a specific year and python -m calendar YEAR MONTH to display a specific month.

Easter Eggs

python -m this will display the Python Zen

python -m antigravity will open XKCD comic web page (which my company is named after).

Quick View of matplotlib Styles

matploblit 1.4.2 added support for styles. I created a notebook to show how the ones bundled in look, you can view it here.

Calculate Distance in .kmz File

Here's a little script that calculates the distance of a path (series of GPS points) in a .kmz file (which I generate with My Tracks). It combines several elements - Zip file, XML (with namespaces), zip, sum and of course a bit of math.

The Versatile date Command

*nix systems comes with a date command line utility. Many people use it to view the current time, however there''s much more that date can do.

For example we run a daily job to process yesterday data. The job is a Python script that get the date as a paramter in YYYYMMDD format. This translates to one cron line:

@daily /path/to/job.py $(date --date=yesterday +%Y%m%d)

Here are some things you can do with date:


As a bonus, there is also a cal command which displays calender. Note that years are 4 digits so cal 4 14 will diaplay April for the year 14, not 2014.

cal has a handy -w switch that shows the work week as well.'

Generate QR Code Using Google Charts API

Here's a small utility to generate QR code image using Google Charts API.

Note the hand crafted Python 2/3 support, for more advanced stuff you might want to have a look at six. However for this script I wanted to stay without external dependencies.

Common Errors

"Experience is the name every one gives to their mistakes."

    - Oscar Wilde

Students are sometimes amazed by how I can tell the cause of exception without looking at the code. This is the result of making many errors in Python myself and also from observing many students doing mistakes. Here's a short list I've compiled of most likely cause of errors:

NameError

• You forgot to import a module
• You made a typo

AttributeError: 'NoneType' object has no attribute ...

• You forgot a return in your function

AttributeError

• Typo on dot lookup (obj.foo)
• Object is different type from what you think (str vs int)
• Object does not implement a dunder method (e.g. __len__)

There's also the 3'rd party didyoumean module, which might be interesting for beginners. It changes the default stack trace to add a hint about what might be the problem.

A Streaming Chart using Flask and flot

I was teaching a course on "Python Analytics" (pandas, scikit-learn, matplotlib ...) and was asked to provide an example of streaming chart - ones that updates periodically. I''ve showed a couple of examples one by generating image using matplotlib and another with bokeh-server. After a couple of days I remembered another way - using flot to render the chart. Here''s a small example using Flask as the web server (the code works both on Python 2 and 3).


Few comments:

• Don''t use debug=True in production :)
• For simplicity everything is in one file. However for larger application you might want to take the HTML template(s) out
• Data is in memory, a restart will wipe it out. If you need data persistence - pick a database (shelve, sqlite3 ...)
• A big shoutout to Continuum Analytics - Anaconda (and conda) has made my life so much easier teaching this workshop.

Resolve SSH Host Name

~/.ssh/config let''s you give meanigful names to hosts/ips. But sometimes you want the reverse lookup - what''s the ip of web1? Here''s a little Python script that does that.

EDIT: EAFP > LBYL

Archlinux Install Steps (on VirtualBox)

My favorite Linux distro for using under VirtualBox is Archlinux with XFCE window manager. It''s light, fast and has all the latest shiny new toys (just the way I like it :).

I found myself setting up VMs to try things out and wrote down the steps I use, this is a trimmed down version of the Installation part in the Archlinux beginners guide.

Be a Better Developer by Coding in Four Different Types of Lauguages

I like programming languages and find out that every time I learn a new language it improves my coding in the other ones as well. I learn new ways of doing things, different ways of thinking and it's great.

I usually tell new developers they need to write a (small) project in at least four types of languages - mainstream (procedural/OO), functional, logic based and assembly. Each of these types will give you a different way of solving problems and enrich your programming experience by and order of magnitude.

Here are my recommendations for each category.

Main Stream

By "main stream" I mean procedural/OO languages. There are tons of these and its up to what you're working with currently (though it might be a good excuse to learn a new language). I'm a Python expert, but pick anything - Go, JavaScript, Ruby, C, C++, Java, C# ...

(Yeah - I know they differ a lot. But thinking in most of them is probably the same. The main difference will probably be dynamic vs static typing).

Functional

Many choices here as well. Personally I like the Lisp family of languages, mostly Clojure and Scheme but you can check out a Common Lisp implementation (I think SBCL leads the pack currently), Haskell, ML and others.

Logic Based

If you haven't done logic programming - it'll blow your mind! It's a totally different way of thinking. Prolog is the main language, one free implementation is SWI but there are others as well.

Assembly

Learning assembly will give you a better understanding on how computers work and what are the abstractions other programming languages do for you. I recommend picking one that targets the machine you're working on.

Add That Trailing Comma

Lets say you wrote this simple code and at first things were going well.

Then after a while a friend came in and did a little fix.

But things started falling apart, after a lot of digging in - you find this.

Python will join two string together in this case, not what you wanted. Always leave a trailing comma.

Couple more things:

• As Dave Cheney pointed out, using this practice has the nice effect that one line change shows as one line change in the diff since you don''t have to add a comma to the previous line.
• Go probably learned from Python and made trailing comma mandatory.
• You can see more Python "gotchas" here.

draft2gist - Publish draftin.com Documents to gist

I've started playing with draftin.com, so far very nice.

draftin.com lets you publish documents to several sites, and if the site you want to publish to is not on the list - there are WebHooks.

I've written a small AppEngine service that is a WebHook for publishing draftin.com documents to gist. Feel free to use it and let me know if you find any errors.

Below is the server code, rest of the files are here.

HTTP Proxy Stripping Headers (go)

At one of my clients, we wanted to write an HTTP proxy that strips some of the headers sent from the target (backend). With the help of golang-nuts mailing list - this turned out to be pretty simple.

timeat on pypi

timeat is now on pypi. There's some extra code to get current time from NTP server. Should work both on Python 2.x and 3.x

timeat in Go

Wrote timeat, which shows time at a specific location, in Go as well. (For comparison the Python version is here). To install either go get bitbucket.org/tebeka/timeat or download the executables.

Working in a multi-timezone team, this script comes handy from time to time (as well as worldtimebuddy :).

Generating Byte Arrays for Assets in Go using xxd

Go's net/http server is pretty fast, but sometimes you want to get faster. One way to do that is to create a binary array in memory for static files (assets). Here''s how I generate the byte arrays automatically with xxd.

Note that this makes the build go slower as you have more and bigger assets.If this is a problem, take a look at nrsc ;)

Since the go toolchain does not support custom steps currently, I''m using make.

Makefile

httpd.go

Hook to Update Tag List when Changing git Branch

I use ctags with Vim to move around. At work we use git feature branches, which means code changes when you switch a branch. Here''s a very simple post-checkout hook to update the tag list whenever you switch branches.

Use dict to Speed Up Your Code

In Python, dictionary access is very fast. You can use that to get some speedup in your code  by replacing if/else with dictionary get.

In [1]: fn1 = lambda x: 1 if x == 't' else 0

In [2]: fn2 = {'t': 1, 'f': 0}.get

In [3]: %timeit fn1('y')  # Check "True" branch
10000000 loops, best of 3: 124 ns per loop

In [4]: %timeit fn2('y')
10000000 loops, best of 3: 79.6 ns per loop

In [5]: %timeit fn1('f')  # Check "False" branch
10000000 loops, best of 3: 125 ns per loop

In [6]: %timeit fn2('f')
10000000 loops, best of 3: 81.3 ns per loop

About 30% speedup - not bad :)

HTTPDir - Small OSX Utility to Serve a Directory over HTTP

HTTPDir is a small utility that lets you serve content of a directory over HTTP.

This is handy when you develop static sites that has reference to external resources. It is also aimed to people who are not comfortable with the command line.

HTTPDir is a simple Python script that uses Tkinter. It is packed in a format that OSX recognizes as an application. See the code here (look under HTTPDir.app/Contents/MacOS).

"timeat" updated

timeat updated to use the new(ish) Google Time Zone API.

$ timeat haifa
Haifa, Israel: Fri May 23, 2014 22:16
$ timeat paris
Paris, France: Fri May 23, 2014 21:17
Paris, TX, USA: Fri May 23, 2014 14:17
Paris, TN 38242, USA: Fri May 23, 2014 14:17
Paris, IL 61944, USA: Fri May 23, 2014 14:17
Paris, KY 40361, USA: Fri May 23, 2014 15:17

Oh, and there's also a Go version.

Important vs Urgent

This was a lesson I learned a long time ago, maybe other people will find it useful as well.

Look at the chart above, we divide tasks to four categories on the urgent/important dimensions. Let's look at each category:

1. Not urgent and important - this is where you (and most successful companies) should spent your time. This means investing in infrastructure, making your process more efficient ...
2. JFDI. However if you find that most of your time is spent there - you're doing something wrong.
3. Don't bother.
4. This is the big time sucker. Try to avoid at all cost.

Note: "If everything is important, then nothing is." - Patrick Lencioni

See more here, here and here.

Easy statsd metrics decorator and context manager

Easy statsd metrics decorator and context manager statsd is very handy with creating metrics. Here's a decorator and context manager that simplify the usage even more.
Note: If you want to easily test your code, you can use this Vagrant VM.

Running commands via HTTP

Ran asked about invoking commands via HTTP interface. Here's a quick answer I came up with (using Flask).

pypi→u

I was trying to see if there's an annoucement list for Celery (which we use at work) - didn't find it. One thing led to another... and I wrote pypi→u, a service that emails you if there's a new version of packages your interested in.

It run on AppEngine (with some bootstrap sprinkeld on). You can view the source here.

Note that pypi→u is very alpha, handle with care. Suggestions, comments and bug reports are welcomed.

Current email setup: osx + homebrew + sup + davmail + getmail

 At work we have an Exchange server for email (no POP3/IMAP) and most people use Outlook. However I prefer not to use outlook, here's what I came with:

• Sup as email client
• DavMail for POP3/SMTP
• getmail for fetching email
• Homebrew for unix-ish environment (yup, that's OSX for you)

 Setup is a bit complicated, but once you're there - email is a bliss (well, as much as email can be :)

DavMail

Install DavMail from the .dmg. Run it and point it to the exchange server URL and have it expose POP3 and SMTP. See ~/.davmail.properties (though it's easy to configure from the UI as well).

I added DavMail to my "Login Items" so it starts when I login.

Homebrew

See the site for documentation on how to install.

getmail

Install with "brew install getmail". I keep my mail in ~/Mail, see ~/.getmail/getmailrc.

I have a cron job to run getmail every 3 minutes
    */3 * * * * /usr/local/bin/getmail -n -q

sup

sup is based on ruby. I've installed rbenv and ruby 1.9. "gem install sup" should work. (Note that I had trouble linking to ncurses, and had to run "brew link ncurses" before installing sup). See ~/.sup for configuration example.

sendmail

Sending mail is done via the sendmail program that comes with OSX. A bit of configuration is needed though. See /etc/postfix/main.cf and /etc/postfix/sasl_passwd

After the above changes, run the following commands:
    sudo postmap /etc/postfix/sasl_passwd
    sudo postfix reload

That's it, you should be set to go with a decent email client.

I also use iTerm2, which lets me click on a link on sup terminal and open it

Reading Passwords of OSX Keychain

OSX stores passwords in its keychain, which is sometimes useful when you forget a password. The command line security utility lets you access the keychain, however the output is somewhat cryptic.

Below is a little script to get user and password for a given domain.

    find-pass.py signup.netflix.com

Note you might get an image like the image on the right. Click either "Allow" or "Always Allow".

Dealing with Bad Memory

The family joke is that I was born senile (my joke is that I have 1bit memory). During the years I've developed some methods to help me be be productive with bad memory. Hope they'll help you as well (if you remember them :).

JFDI
That's the most effective method - do it at the moment you remember. It's pretty amazing how many things you can do "right now" without interrupting your flow too much. Once you did it - there's no need to remember.

Make it Impossible to Forget
Yeah, writing things down help - but I forget to look at my lists. However if you make things impossible to forget - then you won't forget. For example if I need to take something to the car, I'll place it at the front door - can't miss it when I go out.

Get Help
May they be electronic or human, get some help. I married a wife with a phenomonal memory, but it's not an option for everyone :) In this digital age you can find a good non human assistent to help you. Hiring a human assitenet doesn't have to be expensive - see Fancy Hands for example. 

Write It Down
We have a saying: "A short pencil is better than a long memory.". Write things down, it'll help you remember when you write then and later you can look them up. I use this blog as an memory of things that worked for me. Other things I use are pinboard, trello, GMail and Google Docs. Everything with a search function in it.
 
Forgive Yourself
You will forget things, learn to live with it. One of the worst things you can do it agonize over the things you forgot. It'll only add stress to your life without helping you to remember.  

Removing "noise" before matplotlib charts in IPython notebooks

By default, IPython notebook prints the value of the last expression. This works fine most of the time but sometimes the result of a chart is a long list of lines which is something you don't want to see - you just want to see the chart. Here are two options to fix this (you can also view the example notebook).

Decorators and Context Managers Workshop

I gave a workshop on decorators and context mangers at work.

Both are very powerful and will change the way you code in Python once you grok them. They mostly allow you to focus on application logic and write the "book keeping" code (logging, timing, resource allocation ...) in a separate place.

Here are the IPython notebooks:

• Context Managers
• Decorators

And the solutions (but try to solve the exercises first):

• Context Managers
• Decorators

 If you want to run locally, the code is here.

Searching and Viewing emoji

We started using emoji at work. We're using Adium with the emoji xtra.

Here's a little script to search and optionally show emojis (requires matplotlib).

Making "pip" Faster

We use pip to install packages at work. Mostly in combination with virtualenv and wheel packages. Lately I had to re-install again and again a bunch of virtual environments and looks for a way to speed things up.

The solution was to cache the pip downloads and also to keep in a local directory the bigger .whl packages. All done in ~/.pip/pip.conf. Most of the downloads are done once now and using local .whl means no more compilation (I'm looking at you PyYAML ...). I didn't measure the time difference, but for sure it feels faster.

A Simple Python Based Configuration with Overrides

Here's a simple Python based configuration system that supports overrides. We use a variant of it at work and it proves to be very handy. The fact that you can use all of Python in your configuration is very powerful. We use Puppet to write config_local.py in production/qa environments and keep the default good for development machines.

Import Google Bookmarks to Pinboard

I decided to give Pinboard a try. Sadly they lack support for importing Google Bookmarks.

I wrote a little utility to do just that (still waiting for pinboard support to get back to me ;)

Python Logging Overview

Gave a talk at work about Python Logging.

You can view it here.

logbot

We have an internal Jabber server at work, which is great for communication. However unlike HipChat/Campfire - it lacks central logging.

A couple of hours with sleekxmpp, Whoosh and flask solved the problem. logbot is out there - use with caution. (And pull requests are more than welcome ;)

"crashlog" is now on PyPi

FYI: Decided to make crashlog a PyPi package.
LMK if you find any problems.

JSON Handling with datetime Support

While working on some ETL testing code, we needed to load some synthetic data to a database. At first we thought of using YAML for the synthetic data, but it was very slow so we decided to switch to JSON which is still readable but way faster (loading 183 records took 0.68sec with YAML vs 0.008sec in JSON).

However we needed support for serializing datetime objects. Here's dtjson, which support datetime objects in JSON serialization. You can use it almost a drop-in replacement for json: import dtjson as json.

Adding ODBC Source on OSX

At work, we're using Netezza as our main data warehouse. It took me a while to figure out how to add the ODBC driver on OSX so that I'll be able to connect with pyodbc.

Here are the steps:

• Get the OSX Netezza ODBC driver and unpack it
• Not sure where do get it, we have it internally
• Extract the archive somewhere (say /opt/NetezzaODBCDriver)
•  Add  export DYLD_LIBRARY_PATH="${DYLD_LIBRARY_PATH}":/opt/NetezzaODBCDriver/lib/ to your ~/.zshrc (or ~/.bashrc)
• mkdir -p ~/Library/ODBC
• cp /opt/NetezzaODBCDriver/ini/*.ini ~/Library/ODBC
• Edit ~/Library/ODBC/odbcinst.ini to reflect were your driver is (Driver and Setup keys)
• In our example it'll be Driver = /opt/NetezzaODBCDriver/lib/libnzodbc.so (same for Setup) 

That's it, you can now connect to Netezza from OSX.

Here's a small example usage:

Don't Forget Your Process When Selecting Tools

When selecting tools, there are many things you should consider - prices, value added, integration cost, health ...

However many teams forgot to check how well does the tool integrate with your current development process. A good tool that does not play well with your process will cause many problem and might alter your process a direction you don't want.

For example, we have a process (like many others probably) which involves code reviews. We have a tool with does not play well with code reviews - it has binary projects. The process with this tool is different, much slower and more error prone than the rest of the tools.

Next time you pick a tool to work with, think how well will it play with your current process and take it into consideration.

Note: It might be OK to change the process if the tool is worth it, just make sure it does before you integrate it.

Advice For New Managers

Here's some of the advice I give new managers. I try to talk to them before they start their first management job - let me be the first to ruin their career :)

Disclaimer: I'm not a manager, but I was managed by many and was an officer for many years in the IDF which is very informal kind of army.

Here they are, without any particular order.

Style

Every manager has their own style. From fire-and-forget to micromanagement and everything in the middle. Your style will probably be different from another manager (even from your own), your management style will change over time and from team to team. Play to your strengths - organizational, people skills, ....

Don't be afraid to try until you find a style that both you and your team are comfortable in. As long as you're open about what you're doing, you team will support you. It's hard to measure the effectivenehess of each style, but if you can - do that.

Let Go

You are a manager now, not a developer. I've seen very few managers in my long career that managed to do both effectively. Learn to trust your team and be a manager first and developer last.

Think about the things you liked and disliked in your past managers (and the current one) and figure out what you should do.

Manage Your Time

Management is very different from development, it's much more interruption driven. However you need to allocate time for yourself. I've seen the best officers in the army taking up to 30 minutes to think while all hell breaking loose around them (number 2 was in charge in the meanwhile). If they can do it - I'm sure you can.

Here's a great talk by John Cleese (of the Monty Python fame) about creativity which covers this topic as well.

Know Your Place

In officer training over at IDF, they teach you that an officer should be where he/she's most needed. Same goes for you - think about what you are the most effective at solving the more important things and be there.

Develop A Mantra

Find a sentence which will help you focus, and ask it repeatedly throughout the day. I found out that "Why aren't we deploying?" help our team focus and deliver. However find one that fits your goals/team/company ....

You Team Comes First

If you show your team they come first (mostly by "buffering" them from all the management noise), they will be loyal to you. Loyalty works both ways or it doesn't work.A loyal and jelled team is hard to build but when you get there - they will rock.

That's about it. Don't be nervous and try to enjoy a new position. In the worse case know that management is not for everybody (I personally don't like it). There's no shame in saying "this is not for me" and getting back to to fun stuff. (My brother did just that, he managed for a year and then said to the company - "Either I'm back to development or I'm out". He was back in development).

I hope that you found this useful.

Vi Keybindings in SBCL REPL

I'm (re-)reading "Practical Common Lisp" and going over the exercises/code with SBCL. I found the below script to help me with my Vi addiction :)

Error Handling in Python

I gave a talk at work about error handling in Python.
You can view it here.

What do you think? Did I miss something?

It's worth noting that after a while working with Go and the way it handles errors (if value, err = foo(); err != nil ...). If find my Go programs more verbose but more robust at the same time.

XFCE and Volume Keys

I'm currently playing around with XFCE. I have PulseAudio and wanted the volume keys to work (I'm a keyboard kinda guy. :). Here's a solution:

• Open Settings -> Keyboard -> Application Shortcuts
• Click on "Add"
• Type pactl set-sink-volume 0 +10% and then OK
• Hit the "up volume" button on your keyboard
• Click on "Add" (again)
• Type pactl set-sink-volume 0 -- -10% and then OK
• The -- is not a typo
• Clock on "Add"

Your volume keys sound work now.

Oh, and I find installing they "Greybird" theme [1] (from Ubuntu) makes things much nicer :)

[1] Install with yaourt -S xfce-theme-greybird and then select in Settings -> Appearance

Google Bookmark Command Line Utility

A little utility to save URLs in Google Bookmarks. (This will open a web browser with some fields populated).

IPython on OSX (with iTerm2)

Sadly, in the new workplace I have to use OSX (it's either that or Windows, running Linux and connecting to everything is a ... challenge).

Here's a little script I wrote to open IPython with a new iTerm2 window. I call this script ipy and place it at ~/bin. Then hitting APPLE+SPACE launches Spotlight. Writing "ipy" inside and hitting ENTER does the trick.

If you want VIm keybindings, add the following to ~/.inputrc

set editing-mode vi
tab: complete
set completion-ignore-case on
set blink-matching-paren on

Oh, and if you find out how to maximize the window - please share :)

Command line front end to Jinja2

Below is a little command line front end to the Jinja2 templating library.
(Very much in the spirit of the mako utility).

Want To Be More Productive? - Improve Your I/O

It always surprise me that people trying to makes themselves more productive overlook their I/O - reading and writing.

I still see many people slowly and inefficiently reading on the input side and pecking their way on the keyboard in the output side.

I guess the reason that people don't invest time in improving their I/O is that it's boring. I'll be the first to agree that learning speed reading or touch typing is boring, however the benefits you'll reap are big - the investment will pay itself every time you interact with your computer.

After getting the basics of speed reading and touch typing out of the way [1], there are many more ways you can improve your I/O. Here are few examples:

• Learn to efficiently skim through documents
• Learn to pick what to read
• Master your IDE
• Automate everything
• ...

[1] And you should never stop practicing.

Running "one off" Conversions In PostgreSQL

The other day we needed to runs a one-off job for converting datetime.weekday() values to day names. Here's what we came up with.

Getting Good Errors from Python Map/Reduce Jobs

At work, we use some Python map/reduce jobs (using Hadoop streaming).

Debugging can be difficult, since Hadoop does not keep the Python stacktraces and even if it does - it's very hard to find it. We decided to use crashlog, and now we get wonderful emails with detailed description of what went wrong.

Notes:

• The current mapper input file is in map_input_file environment variable
• Don't forget to add crashlog.py with -file (see here)
• You must add "." to PYTHONPATH in order to import
• import sys; sys.path.append('.') should do the trick
• Email is not the best solution for distributed logging (you get a lot of email when things go South). I'm going to play with graylog2 in the future.