Advent of Code 2017 #8 and Python's eval

I'm having fun solving Advent of Code 2017. Problem 8 reminded the power of Python's eval (and before you start commenting the "eval is evil" may I remind you of this :)

You can check The Go implementation that don't have eval and need to work harder.

Python → Go Cheat Sheet

I'm teaching and consulting in Go a lot lately, and I work a lot with Python as well. There's a big trend of rewriting backend services in Go and to help people coming from the Python world I've created a Go →Python cheatsheet.

The code in here, I'd love to hear if you have suggestion (via a PR ;)

Finding Where Most Activity Time Was Spent

View Notebook

Checking for Zero Values in Go

In Go, every type has a zero value. Which is the value a variable of this type get if it's not initialized. I had a configuration object of type map[string]interface{} and I needed to check if value exists and is not a zero value.

Here's a small piece of code that checks for zero values:

Generating Power Set using Bitmap

I was asked to write a function that generate a power set of items. At first I wrote a recursive algorithms but then another approach came to mind. When you calculate how many subsets there are, you can say that each item in the original set can either be or not be in a subset, which means 2^n subsets. This yes/no for including can be seen as a bitmask, and since we know that there are 2^n subsets we can use the number from 0 to 2^n-1 as bitmasks.

Who Touched the Code Last? (git)

Sometimes I'd like to know who to ask about a piece of code. I've developed a little Python script that shows the last people who touch a file/directory and the ones who touched it most.

Example output (on arrow project)
$ owners
Most: Wes McKinney (39.5%), Uwe L. Korn (15.3%), Kouhei Sutou (10.8%)
Last: Kengo Seki (31M), Uwe L. Korn (39M), Max Risuhin (23H)

So ask Wes or Uwe :)

Here's the code:

Go's append vs copy

When we'd like to concatenates slices in Go, most people usually reach out for append. Most of the time this solution is OK, however if you need to squeeze more performance - copy will be better.

EDIT: As Henrik Johansson suggested, if you pre-allocate the slice to append it'll fast as well.

scikit-learn Compatible Pipeline Steps

A client wanted a way to create a pipeline of transformations on DataFrames. Since they already work with scikit-learn, they were familiar with Pipelines. It took very little code to create a base class for a pipeline step that will work with DataFrames.

---

If you find this interesting, you might want to hire me ;)

Color Log Lines

There are several log handler out there the color log lines. However I prefer to leave the log as is and when I want colors I pipe the log via a utility that does that. Coloring log lines by levels can be easily done with a little awk.

Here's an example how it looks

353Solutions - 2016 in Review

Happy new year!

Let's see how 353solutions did in 2016. We'll start with the numbers and then some insights and future goals.

Numbers

• Total of 193 work days (days where customer were billed)
• Up 23 days from 2015
• Out of 261 work days in 2016
• Out of these 61 days are in workshops and the rest consulting
• 18 workshops
• Up from 14 in 2015
• Go workshops
• First docker workshop (teaching with ops specialist)
• First open enrollment class
• In Israel, UK, and Poland
• Only 4 out of the country (vs 8 last year, which is good - I prefer to fly less)
• Workshops range from 1 to 4 days
• Revenue up 10% from 2016
• Several new clients including GE, AOL, Outbrain, EMC and more

Insights

• Social network keeps providing almost all the work
• Doing good work will get you more work
• Workshops pay way more than consulting
• However can't work from home in workshops
• Consulting keeps you updated with latest tech
• Python and data science are big and in high demand
• However Go is getting traction
• Having someone else take care of the "overhead" (billing, contracts ...) is great help.

Last Year's Goals
We've set some goals in 2015, let's see how we did:

• Keep positioning in Python and Scientific Python area
• Done
• Drive more Go projects and workshops
• Done
• Works less days, have same revenue at end of year
• Failed here. Worked more and revenue went up
• Start some "public classes" where we rent a class and people show up
• Did only one this year
• Publish my book
• Failed here, though I'm close. Probably Q1 this year.

Goals for 2017

• Work less while keeping same revenue
• Work more from home
• Publish my book
• Publish a video course
• More open enrollment classes

Automatically Running BigQuery Flows

At a current project we're using Google's BigQuery to crunch some petabyte scale data. We have several SQL scripts that we need to run in specific order. The below script detects the table dependencies and run the SQL scripts in order. As a bonus you can run it with --view and it'll show you the dependency graph.

Using built-in slice for indexing

At one place I consult I saw something like the following code:
This is good and valid Python code, however we can use the slice built-in slice to do the same job.

Also when you're writing your own __getitem__ consider that key might be a slice object.

Weighted Rating

Notebook here.

First Contact with Data

Notebook here.

Simple Object Pools

Sometimes we need object pools to limit the number of resource consumed. The most common example is database connnections.

In Go we sometime use a buffered channel as a simple object pool.

In Python, we can dome something similar with a Queue. Python's context manager makes the resource handing automatic so clients don't need to remember to return the object.


Here's the output of both programs:

$ go run pool.go
worker 7 got resource 0
worker 0 got resource 2
worker 3 got resource 1
worker 8 got resource 2
worker 1 got resource 0
worker 9 got resource 1
worker 5 got resource 1
worker 4 got resource 0
worker 2 got resource 2
worker 6 got resource 1

$ python pool.py
worker 5 got resource 1
worker 8 got resource 2
worker 1 got resource 3
worker 4 got resource 1
worker 0 got resource 2
worker 7 got resource 3
worker 6 got resource 1
worker 3 got resource 2
worker 9 got resource 3
worker 2 got resource 1

"Manual" Breakpoints in Go

When debugging, sometimes you need to set conditional breakpoints. This option is available both in gdb and delve. However sometimes when the condition is complicated, it's hard or even impossible to set it. A way around is to temporary write the condition in Go and set breakpoint "manually".

I Python we do it with pdb.set_trace(), in Go we'll need to work a little harder. The main idea is that breakpoints are special signal called SIGTRAP.

Here's the code to do this:
You'll need tell the go tool not to optimize and keep variable information:

$ go build -gcflags "-N -l" manual-bp

Then run a gdb session

$ gdb manual-bp 

(gdb) run 

 When you hit the breakpoint, you'll be in assembly code. Exit two functions to get to your code

(gdb) fin
(gdb) fin

Then you'll be in your code and can run gdb commands

(gdb) p i
$1 = 3

This scheme also works with delve

$ dlv debug manual-bp.go 
(dlv) c 

Sadly delve don't have "fin" command so you'll need to hit "n" (next) until you reach your code. 

That's it, happy debugging.

Oh - and in the very old days we did about the same trick in C code. There we manually inserted asm("int $3)" to the code. You can do with with cgo but sending a signal seems easier.

Generate Relation Diagram from GAE ndb Model

Working with GAE, we wanted to create relation diagram from out ndb model. By deferring the rendering to dot and using Python's reflection this became an easy task. Some links are still missing since we're using ancestor queries, but this can be handled by some class docstring syntax or just manually editing the resulting dot file.

Testing numpy Code

Notebook here.

Work with AppEngine SDK in the REPL

Working again with AppEngine for Python. Here's a small code snippet that will let you work with your code in the REPL (much better than the previous solution).
What I do in IPython is:

In [1]: %run initgae.py

In [2]: %run app.py

And then I can work with my code and test things out.

Using ImageMagick to Generate Images

One of the exercises we did this week in the Python workshop used the term "bounding box diagonal". I had a hard time to explain it to the students without an image. Google image search didn't find anything great, so I decided to create such an image.

First I tried with drawing programs, but couldn't make the rectangle a square and the circle non-oval. Then I remembered imagemagick, I have it installed and mostly use it to resize images - but it can do much more. A quick look at the examples and some trial and error, and here's the result.

And here's the script that generated it:

Waiting for HTTP Server - Go Testing

I don't like mocking in tests. If I have a server to test, I prefer to start and instance and hit the API in my tests. Waiting for the server to start with a simple sleep is unpredictable. I prefer to start the server, try to hit an URL until it's OK or fail after a long timeout. This is a simple task with Go's select and time.After.

Slap a --help on it

Sometimes we write "one off" scripts to deal with certain task. However most often than not these scripts live more than just the one time. This is very common in ops related code that for some reason people don't apply the regular coding standards to.

It really upsets me when I try to see what a script is doing, run it with --help flag and it happily deletes the database while I wait :) It's so easy to add help support in the command line. In Python we do it with argparse, and we role our own in bash. Both cases it's extra 3 lines of code.

Please be kind to future self and add --help support to your scripts.

vfetch - Fetch Go Vendor Depedencies

Go 1.6 now supports vendoring. I found myself cloning dependencies to "vendor" directory, then cloning their dependencies ... This got old really fast so vfetch was born. It's a quick and dirty solution, uses "go get" with a temporary GOPATH to get the package and its dependencies, then uses rsync to copy them to the vendor directory.

Installing is the usual "go get github.com/tebeka/vfetch" then you can use "vfetch github.com/gorilla/mux".

Comment, ideas and pull requests are more than welcomed.

Super Simple nvim UI

I've been playing with neovim lately, enjoying it and a leaner RC file. nvim comes currently only in terminal mode and I wanted a way to spin a new window for it. Here's a super simple script (I call it e) to start a new xfce4-terminal window with nvim.

Removing String Columns from a DataFrame

Sometimes you want to work just with numerical columns in a pandas DataFrame. The rule of thumb is that everything that has a type of object is something not numeric (you can get fancier with numpy.issubdtype). We're going to use the DataFrame dtypes with some boolean indexing to accomplish this.

In [1]: import pandas as pd  

In [2]: df = pd.DataFrame([
   ...:     [1, 2, 'a', 3],
   ...:     [4, 5, 'b', 6],
   ...:     [7, 8, 'c', 9],
   ...: ])  

In [3]: df  
Out[3]: 
   0  1  2  3
0  1  2  a  3
1  4  5  b  6
2  7  8  c  9

In [4]: df.dtypes  
Out[4]: 
0     int64
1     int64
2    object
3     int64
dtype: object

In [5]: df[df.columns[df.dtypes != object]]
Out[5]: 
   0  1  3
0  1  2  3
1  4  5  6
2  7  8  9

In [6]:   

Forging Python - First Chapter is Up

Finally, first chapter of my upcoming book "Forging Python" is up. I'm doing it leanpub style so comments ans suggestions are more than welcomed.

I plan to finish the book this year, hopefully during the summer. However more than one person said I'm way too optimistic - time will tell :)

353Solutions - 2015 in Review

Happy new year!

First full calendar year that 353solutions is operating. Let's start with the numbers and then some insights and future goals.

Numbers

• 170 days of work in total
• Work day is a day where I billed someone for some part of it
• Can be and hour can be 24 hours (when teaching abroad)
• There were total of 251 work days in 2015
• There were some work days that are not billable (drafting syllabuses, answering emails ...) but not that many
• 111 of days consulting to 4 clients
• 1st Go project!!!
• 58 days teaching 14 courses
• Python at all levels and scientific Python (including new async workshop)
• In UK, Poland and Israel

Insights

• Social network provided almost all the work
• Keep investing in good friends (not just for work :)
• Workshops pay way more than consulting
• However can't work from home in workshops
• Consulting keeps you updated with latest tech
• Had to let go of a client due to draconian contract
• No regrets here, it was the right decision
• Super nice team. Sadly lawyers had final say the company
• Python and data science are big and in high demand
• Delegating overhead to the right person helps a lot
• Accounting, contracts ...

Future Goals

• Keep positioning in Python and Scientific Python area
• Drive more Go projects and workshops
• Works less days, have same revenue at end of year
• Start some "public classes" where we rent a class and people show up
• Some companies don't have big enough data science team
• Need to invest in advertising
• Publish my book (more on that later)

Using HAProxy to Prevent Deletes from Elasticsearch

At one of my clients, we wanted something quick and dirty to prevent deletes from Elasticsearch (shield is too expensive and would take too much time to integrate with our systems - we'll fix this technical debt later).

The quick solution was to place HAProxy in front of Elasticsearch and use its acl mechanism to prevent HTTP DELETE. Works like a charm.

Here's the HAProxy configuration and the docker-compose setup file I used to test the configuration.

Python's deque for Go

Working on a Go project with my friend Fabrizio, I've investigated ways to have a faster data structure to store history items with append and pop.

Got the idea to try implementing Python's deque in Go. The C implementation is pretty easy to read. The result is deque for Go, which implement a subset of the features from Python's deque but enough for our needs. And it's pretty fast too:

$ make compare
Git head is 765f6b0
cd compare && go test -run NONE -bench . -v
testing: warning: no tests to run
PASS
BenchmarkHistAppend-4  3000000        517 ns/op
BenchmarkHistList-4    2000000        702 ns/op
BenchmarkHistQueue-4   3000000        576 ns/op
BenchmarkHistDeque-4   3000000        423 ns/op
ok   _/home/miki/Projects/go/src/github.com/tebeka/deque/compare 8.505s

aenumerate - enumerate for async for

Python's new async/await syntax helps a lot with writing async code. Here's a little utility that provides the async equivalent of enumerate.

git - Creating Pull Request for master

A co-worker asked me for a code review (we're using Stash, but this can work for other systems as well), the problem was that he worked on master (started his own project) and not in development branch. The solution was to create an empty orphan branch and then a pull request from master to that branch (reverse the usual order).

Here's how to create such branch.

Go Tour Exercise Solutions

As a backup plan for the last Go Meetup, I wrote the solutions to the exercises in Go Tour and we discussed some of them.

You can find the solutions here.

re2 available on conda

We're using re2 to get some speed gains on the many regular expressions we're trying to match. So far building it was either a manual step or a script that ran when building docker container. I decided to create a conda package (we're using Miniconda as our Python environment).

I started with conda skeleton pypi re2 (you need to conda install conda-build first). Then after some tweaking to build.sh we were good to go.

The result - you can now conda install -c tebeka re2 (only 64bit linux supported currently).

The project is here, I'll gladly accept any comments/improvements.

Here's build.sh which patches re2 Makefile and added the library and header location to the Python build step.

fastavro moved to github

If you can't beat them ... :)

fastavro is now on github. I still prefer mercurial as an SCM but most of the pull requests I get are on github and it doesn't worth the effort of maintaining two repositories (though hg-git is a big help)

dockermon - A Docker Event Monitor

I'm currently working with the awesome team at CyberInt (and yes, they are hiring).

We're moving to a docker based environment. The old environment used Supervisor to monitor and relaunch daemons. We had an event listener that notified us on our HipChat room every time a daemon crashed and wanted the same feature with our docker containers.

We didn't find a ready solution, so we wrote one and made it open source. The project is called dockermon and is one Python script with no external dependencies and also Python 2 and 3 compatible.

Naming "with open" Variable

Python''s "with" statement is great for resource handling. However I find my self struggling with naming (and naming is important) the context manager variable.

When you write "with open(''/path/to/somethere'') as X", what''s the best name for X? In some cases it''s obvious, but in most cases I find myself using the generic "fo" (stands for "file object").

I decided to run a little script on Python''s 3.4 Lib directory and find out what is the most common name. Here are the results:

Seems like f is the most common, but I really don''t like single letter variables. I''ll go with the 2nd place - fp.

Here''s the script used to generate this chart:

353Solutions - A Year in Review

353Solutions was founded a bit more than a year ago. I wasn't planning on doing consulting, I'm a techie and love the development abstraction layer that companies give you and let you code most of the time. (If this is not the case in the company you're working at - consider finding a better one :)

However as the old saying goes - "Man plans and god laughs". I found myself owning a teaching/consulting company called 353Solutions. So far it's fun and provides for the family - what else can you ask for?

Here are results from a short retrospective we did lately.

The Numbers

• 6 clients
• 204 work days
• 204 hours teaching Python (7 courses)

Thoughts

I like working from home, however most companies I talked to wanted some office time. This is understandable since I don't only code but also do system and process design - these roles require more face to face communication. I'm still looking for something that will allow me to spend most of my time working from home.

Teaching is fun! I did that on and off most of my professional carrier, but now it's a big chunk of my time. I'm grateful to Raymond Hettinger who started me off and showed me what a top-notch class/workshop should look like. So far I'm mostly going to companies and teaching there, but just now we launched our own classes - it will  be awesome!

The downside for teaching is that it takes me away from home. For a limited amount this is great (I spend about a week every month teaching Python in the UK). However I'm looking for opportunities that will let me teach from home - stay tuned.

The social network is by far my biggest source of new jobs. Talking to other people - it's not just me. Investing time in making connections and keeping them will pay off. The main downside for is that people want to hire me and not 353Solutions. This means I need to work harder to market the other people who I work with - I can't do everything.

Learning to say "no" was the hardest thing for me. So many interesting things to do, so many cool companies ... But I like spending time with my family, friends and hobbies. You need to find the things that make you happy and pay enough, going cheap is not a good thing in most cases. What I did in some cases was to take less money and get equity instead. Something like "technical" investing in startups.

The main point I need to improve is marketing. It's not something I like to do but feel the need, especially now that we have our own classes. I'm learning and looking for the best thing that will get maximal impact with minimal amount of time. Or maybe hire someone for that? If you know a good option  - please let me know :)

Use contextlib.closing to Handle "Legacy" Resources

Python''s context managers (with statement) are very handy at handling resources. (You see way less finally in Python code due to them). Maybe objects in Python can be used as context mangers - files, locks, database drivers and more. But some objects still do not.

To handle these "legacy" objects you can use contextlib.closing function which will return a context manager that will call obj.close() once the context manager exists.

Here''s an example of using contextlib.closing with sockets. We''ll be doing a simple HTTP request (Yeah, you should probably use requests or urlopen - this is just an example :)

Note also the user of iter with sentinel to read chunks up to 1K from the socket.

Combining jQuery and Multi Components of React

React is a great library for generating reactive web UI (and mobile). Reacts works well if there are isolated components or a big one with hierarchy. However I wanted to have a page with several isolated react components that are updated from the same data. The solution I found is to use an observer pattern and have each components register a callback to handle data change. See the code below and a live demo here.

Note that I am a React newbie, if you know of a better way - please enlighten me.

Solving Project Euler Problem 8 with numpy

I''m teaching a course in scientific Python these days. Usually I give "homework" from project Euler (which I personally use every time I learn a new programming language). I thought it''ll be fun to solve the problem not just with Python but with what numpy has to offer as well.

Here''s an example solution for project Euler problem 8.

Docker + MiniConda = A Perfect Match

Working with one of my clients (who is hiring BTW), we decided to use Docker as deployment platform. Since many Linux systems now use Python for many utilities, it''s advisable to install your own Python next to the system one and use it.

Installing CPython from source requires some system packages, libraries, headers and some knowledge. The much easier path it to use MiniConda (from the wonderful people at Continuum). Not only the Python installation is super simple, but also the conda package manger will get you a lot of packages pre-compiled so you don''t have to install gcc and header files for C extensions. And if you can''t find the package you need with conda, pip is also available.

Here''s a little project to demonstrate how to do this. The application is an image server with has two entry points /edge for edge detection and /resize for image resizing. We''ll be using scikit-image and Pillow for image manipulation and Flask as web server. All of them can be conda installed.

Here''s the Dockerfile for the project. Build with docker build -tag imgsrv, Run with docker run -p 8080:8080 imgsrv (see Makefile).

Adding Server SSL Certificate on Linux

Here''s a small script to add a server SSL certificate on Linux. You can export the certificate from your browser. Inspired by this stackoverflow post.

Logging from Celery to logstash and a structured log (JSON)

I love Celery, and we''re using it at one of my customers. One thing we wanted to have is centralized logging, since you can have multiple workers on multiple machines. We looked at several solutions and the winner came out to be logstash + kibana (AKA ELK stack).

Here''s some code to log to logstash with Celery current task information (if available) and also to a structured log (every line is a JSON object) for backup in case of network issues.

Using supervisord to Manage You Daemons

Say you have some daemons running. You''d like to restart them automatically if they fail, grab logs from them and in general manage them - supervisord will do it all for you.

One of my (super cool) clients needed also to start/stop daemons when configuration changes. The solution was to have a script that updates supervisord.conf every time we have a configuration change and then selectively start/stop on the daemons that have change (by default if your run supervisorctl update, it will restart all the daemons).

For this example, I''ll assume that the daemon processes are python -m SimpleHTTPServer and I have a list of port I''d like to listen on. This list of ports might change.

Example Usage

    $ ./updated.py 8000 8001 8002

    $ supervisorctl status

    httpd-8000                       RUNNING   pid 31768, uptime 0:00:04

    httpd-8001                       RUNNING   pid 31767, uptime 0:00:04

    httpd-8002                       RUNNING   pid 31766, uptime 0:00:04

    $ ./updated.py 8000 8004 8002 # Remove 8001, add 8004

    httpd-8001: disappeared

    httpd-8004: available

    httpd-8001: stopped

    httpd-8001: removed process group

    httpd-8004: added process group

    $ supervisorctl status

    httpd-8000                       RUNNING   pid 31768, uptime 0:00:12

    httpd-8002                       RUNNING   pid 31766, uptime 0:00:12

    httpd-8004                       RUNNING   pid 31785, uptime 0:00:02

    $

    

python -m

python -m lets you run modules as scripts. If your module is just one .py file it'll be executed (which usually means code under if __name__ == '__main__'). If your module is a directory, Python will look for __main__.py (next to __init__.py) and will run it.

One of Python's mottoes is "batteries included", and this goes for python -m as well. Here are some (all?) of the gems hidden in the standard library. Sadly not all of them have help, but I poked around in the source code to see the usage.

json.tool

This is by far the one I use most, it'll indent nicely an JSON input in the standard output and very helpful in combination with curl.

$ curl -sL http://j.mp/1IuxaLD

[{"x":1,"y":2},{"x":3,"y":4},{"x":5,"y":6}]

$ curl -sL http://j.mp/1IuxaLD | python -m json.tool

[

    {

        "x": 1,

        "y": 2

    },

    {

        "x": 3,

        "y": 4

    },

    {

        "x": 5,

        "y": 6

    }

]

zipfile

zipfile will let you view, extract and create zip files - very much like the zip and unzip. Here's the help:

$ python -m zipfile -h

Usage:

    zipfile.py -l zipfile.zip        # Show listing of a zipfile

    zipfile.py -t zipfile.zip        # Test if a zipfile is valid

    zipfile.py -e zipfile.zip target # Extract zipfile into target dir

    zipfile.py -c zipfile.zip src ... # Create zipfile from sources

gzip

Like zipfile, let's you compress and decompress .gz files, like gzip/gunzip. By default it'll compress a file but with -d will decompress.

python -m gzip wordlist.txt  # Will create wordlist.txt.gz

python -m gzip -d wordlist.txt.gz  # Will extract to wordlist.txt

filecmp

Compare two directories.

$ python -m filecmp /tmp/a /tmp/b

diff /tmp/a /tmp/b

Only in /tmp/a : ['1']

Only in /tmp/b : ['2']

Identical files : ['4']

Differing files : ['3']

Encode/Decode

Several modules lets you encode/decode in various formats:

• base64
• uu
• encodings.rot_13
• binhex
• mimify
• quopri

For example

$ echo 'secertpassword' | python -m encodings.rot_13

frpregcnffjbeq

Servers

There are several servers that you can run, the ones I know are SimpleHTTPServer, CGIHTTPServer and smtpd (mail). If you quickly want to serve some files from a directory on your machine, just run:

python -m SimpleHTTPServer

Clients

Modules that provide simple clients to various protocols are:

• ftplib
• poplib
• nntplib
• smtplib (on localhost only)
• telnetlib

For example if you want to view Star Wars in text mode, do

$ python -m telnetlib towel.blinkenlights.nl

System Info

You can use platform to get some platform information (very much line uname -a) and locale to get locale information. Use mimetype to get the mime type of a file:

$ python -m mimetypes doc.html

type: text/html encoding: None

Python Utilties

• compileall will compile all Python files to .pyc
• dis will show bytecode for a file
• pdb will start the Python debugger on a given file (see here)
• pydoc will show documentation on a module/class/function
• site will print some site information (sys.path, USER_BASE ...)
• sysconfig will show many system related information (such as exec_prefix)
• tabnanny will tell you of you mix tabs and spaces (like starting python with -t or -tt)
• tokenize will print list of tokens in Python file

I mostly use pdb and pydoc, for example:

$ python -m pydoc os.remove

Help on built-in function remove in os:

os.remove = remove(...)

    remove(path)

    

    Remove a file (same as unlink(path)).

Profiling

There are several profiles and timers you can use from the command line:

• cProfile - Show profile information
• profile (use cProfile :)
• timeit - Time how long things run
• pstats - Print output of profiles
• trace - Show tracing information on run

Example:

$ python -m cProfile match.py

         28537 function calls (27503 primitive calls) in 0.057 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)

        1    0.000    0.000    0.000    0.000 :1()

        1    0.000    0.000    0.000    0.000 :1(ArgInfo)

        1    0.000    0.000    0.000    0.000 :1(ArgSpec)

   ...

$ pyton -m timeit 'import math; math.factorial(100)'

100000 loops, best of 3: 12.9 usec per loop

timeit has good help from the command line.

IDLE

You can start IDLE by running python -m idlelib.idle

ensurepip

Python 2.7.9 and 3.x comes with an easy way to install pip. Run python -m ensurepip and pypi is at your service.


That's about it ... What are you favorite python -m tools? Which ones did I miss?

EDIT: The good folks at comp.lang.python reminded me a few I forgot:

unittest

python -m unittest discover will run unittest in discovery mode. Just drop a new Python file starting with test and it'll be picked up next time you run the tests. You can also specify a specific test to run with python -m unittest test_file.py TestClass.test_method.

calendar

python -m calendar will show calendar of the current year. You can also run python -m calendar YEAR to display a specific year and python -m calendar YEAR MONTH to display a specific month.

Easter Eggs

python -m this will display the Python Zen

python -m antigravity will open XKCD comic web page (which my company is named after).

Quick View of matplotlib Styles

matploblit 1.4.2 added support for styles. I created a notebook to show how the ones bundled in look, you can view it here.

Calculate Distance in .kmz File

Here's a little script that calculates the distance of a path (series of GPS points) in a .kmz file (which I generate with My Tracks). It combines several elements - Zip file, XML (with namespaces), zip, sum and of course a bit of math.

The Versatile date Command

*nix systems comes with a date command line utility. Many people use it to view the current time, however there''s much more that date can do.

For example we run a daily job to process yesterday data. The job is a Python script that get the date as a paramter in YYYYMMDD format. This translates to one cron line:

@daily /path/to/job.py $(date --date=yesterday +%Y%m%d)

Here are some things you can do with date:


As a bonus, there is also a cal command which displays calender. Note that years are 4 digits so cal 4 14 will diaplay April for the year 14, not 2014.

cal has a handy -w switch that shows the work week as well.'

Generate QR Code Using Google Charts API

Here's a small utility to generate QR code image using Google Charts API.

Note the hand crafted Python 2/3 support, for more advanced stuff you might want to have a look at six. However for this script I wanted to stay without external dependencies.

Common Errors

"Experience is the name every one gives to their mistakes."

    - Oscar Wilde

Students are sometimes amazed by how I can tell the cause of exception without looking at the code. This is the result of making many errors in Python myself and also from observing many students doing mistakes. Here's a short list I've compiled of most likely cause of errors:

NameError

• You forgot to import a module
• You made a typo

AttributeError: 'NoneType' object has no attribute ...

• You forgot a return in your function

AttributeError

• Typo on dot lookup (obj.foo)
• Object is different type from what you think (str vs int)
• Object does not implement a dunder method (e.g. __len__)

There's also the 3'rd party didyoumean module, which might be interesting for beginners. It changes the default stack trace to add a hint about what might be the problem.

A Streaming Chart using Flask and flot

I was teaching a course on "Python Analytics" (pandas, scikit-learn, matplotlib ...) and was asked to provide an example of streaming chart - ones that updates periodically. I''ve showed a couple of examples one by generating image using matplotlib and another with bokeh-server. After a couple of days I remembered another way - using flot to render the chart. Here''s a small example using Flask as the web server (the code works both on Python 2 and 3).


Few comments:

• Don''t use debug=True in production :)
• For simplicity everything is in one file. However for larger application you might want to take the HTML template(s) out
• Data is in memory, a restart will wipe it out. If you need data persistence - pick a database (shelve, sqlite3 ...)
• A big shoutout to Continuum Analytics - Anaconda (and conda) has made my life so much easier teaching this workshop.

Resolve SSH Host Name

~/.ssh/config let''s you give meanigful names to hosts/ips. But sometimes you want the reverse lookup - what''s the ip of web1? Here''s a little Python script that does that.

EDIT: EAFP > LBYL

Archlinux Install Steps (on VirtualBox)

My favorite Linux distro for using under VirtualBox is Archlinux with XFCE window manager. It''s light, fast and has all the latest shiny new toys (just the way I like it :).

I found myself setting up VMs to try things out and wrote down the steps I use, this is a trimmed down version of the Installation part in the Archlinux beginners guide.

Be a Better Developer by Coding in Four Different Types of Lauguages

I like programming languages and find out that every time I learn a new language it improves my coding in the other ones as well. I learn new ways of doing things, different ways of thinking and it's great.

I usually tell new developers they need to write a (small) project in at least four types of languages - mainstream (procedural/OO), functional, logic based and assembly. Each of these types will give you a different way of solving problems and enrich your programming experience by and order of magnitude.

Here are my recommendations for each category.

Main Stream

By "main stream" I mean procedural/OO languages. There are tons of these and its up to what you're working with currently (though it might be a good excuse to learn a new language). I'm a Python expert, but pick anything - Go, JavaScript, Ruby, C, C++, Java, C# ...

(Yeah - I know they differ a lot. But thinking in most of them is probably the same. The main difference will probably be dynamic vs static typing).

Functional

Many choices here as well. Personally I like the Lisp family of languages, mostly Clojure and Scheme but you can check out a Common Lisp implementation (I think SBCL leads the pack currently), Haskell, ML and others.

Logic Based

If you haven't done logic programming - it'll blow your mind! It's a totally different way of thinking. Prolog is the main language, one free implementation is SWI but there are others as well.

Assembly

Learning assembly will give you a better understanding on how computers work and what are the abstractions other programming languages do for you. I recommend picking one that targets the machine you're working on.

Add That Trailing Comma

Lets say you wrote this simple code and at first things were going well.

Then after a while a friend came in and did a little fix.

But things started falling apart, after a lot of digging in - you find this.

Python will join two string together in this case, not what you wanted. Always leave a trailing comma.

Couple more things:

• As Dave Cheney pointed out, using this practice has the nice effect that one line change shows as one line change in the diff since you don''t have to add a comma to the previous line.
• Go probably learned from Python and made trailing comma mandatory.
• You can see more Python "gotchas" here.

draft2gist - Publish draftin.com Documents to gist

I've started playing with draftin.com, so far very nice.

draftin.com lets you publish documents to several sites, and if the site you want to publish to is not on the list - there are WebHooks.

I've written a small AppEngine service that is a WebHook for publishing draftin.com documents to gist. Feel free to use it and let me know if you find any errors.

Below is the server code, rest of the files are here.

HTTP Proxy Stripping Headers (go)

At one of my clients, we wanted to write an HTTP proxy that strips some of the headers sent from the target (backend). With the help of golang-nuts mailing list - this turned out to be pretty simple.

timeat on pypi

timeat is now on pypi. There's some extra code to get current time from NTP server. Should work both on Python 2.x and 3.x

timeat in Go

Wrote timeat, which shows time at a specific location, in Go as well. (For comparison the Python version is here). To install either go get bitbucket.org/tebeka/timeat or download the executables.

Working in a multi-timezone team, this script comes handy from time to time (as well as worldtimebuddy :).

Generating Byte Arrays for Assets in Go using xxd

Go's net/http server is pretty fast, but sometimes you want to get faster. One way to do that is to create a binary array in memory for static files (assets). Here''s how I generate the byte arrays automatically with xxd.

Note that this makes the build go slower as you have more and bigger assets.If this is a problem, take a look at nrsc ;)

Since the go toolchain does not support custom steps currently, I''m using make.

Makefile

httpd.go

Hook to Update Tag List when Changing git Branch

I use ctags with Vim to move around. At work we use git feature branches, which means code changes when you switch a branch. Here''s a very simple post-checkout hook to update the tag list whenever you switch branches.

Use dict to Speed Up Your Code

In Python, dictionary access is very fast. You can use that to get some speedup in your code  by replacing if/else with dictionary get.

In [1]: fn1 = lambda x: 1 if x == 't' else 0

In [2]: fn2 = {'t': 1, 'f': 0}.get

In [3]: %timeit fn1('y')  # Check "True" branch
10000000 loops, best of 3: 124 ns per loop

In [4]: %timeit fn2('y')
10000000 loops, best of 3: 79.6 ns per loop

In [5]: %timeit fn1('f')  # Check "False" branch
10000000 loops, best of 3: 125 ns per loop

In [6]: %timeit fn2('f')
10000000 loops, best of 3: 81.3 ns per loop

About 30% speedup - not bad :)

HTTPDir - Small OSX Utility to Serve a Directory over HTTP

HTTPDir is a small utility that lets you serve content of a directory over HTTP.

This is handy when you develop static sites that has reference to external resources. It is also aimed to people who are not comfortable with the command line.

HTTPDir is a simple Python script that uses Tkinter. It is packed in a format that OSX recognizes as an application. See the code here (look under HTTPDir.app/Contents/MacOS).

"timeat" updated

timeat updated to use the new(ish) Google Time Zone API.

$ timeat haifa
Haifa, Israel: Fri May 23, 2014 22:16
$ timeat paris
Paris, France: Fri May 23, 2014 21:17
Paris, TX, USA: Fri May 23, 2014 14:17
Paris, TN 38242, USA: Fri May 23, 2014 14:17
Paris, IL 61944, USA: Fri May 23, 2014 14:17
Paris, KY 40361, USA: Fri May 23, 2014 15:17

Oh, and there's also a Go version.

Important vs Urgent

This was a lesson I learned a long time ago, maybe other people will find it useful as well.

Look at the chart above, we divide tasks to four categories on the urgent/important dimensions. Let's look at each category:

1. Not urgent and important - this is where you (and most successful companies) should spent your time. This means investing in infrastructure, making your process more efficient ...
2. JFDI. However if you find that most of your time is spent there - you're doing something wrong.
3. Don't bother.
4. This is the big time sucker. Try to avoid at all cost.

Note: "If everything is important, then nothing is." - Patrick Lencioni

See more here, here and here.

Easy statsd metrics decorator and context manager

Easy statsd metrics decorator and context manager statsd is very handy with creating metrics. Here's a decorator and context manager that simplify the usage even more.
Note: If you want to easily test your code, you can use this Vagrant VM.

Running commands via HTTP

Ran asked about invoking commands via HTTP interface. Here's a quick answer I came up with (using Flask).

pypi→u

I was trying to see if there's an annoucement list for Celery (which we use at work) - didn't find it. One thing led to another... and I wrote pypi→u, a service that emails you if there's a new version of packages your interested in.

It run on AppEngine (with some bootstrap sprinkeld on). You can view the source here.

Note that pypi→u is very alpha, handle with care. Suggestions, comments and bug reports are welcomed.

Current email setup: osx + homebrew + sup + davmail + getmail

 At work we have an Exchange server for email (no POP3/IMAP) and most people use Outlook. However I prefer not to use outlook, here's what I came with:

• Sup as email client
• DavMail for POP3/SMTP
• getmail for fetching email
• Homebrew for unix-ish environment (yup, that's OSX for you)

 Setup is a bit complicated, but once you're there - email is a bliss (well, as much as email can be :)

DavMail

Install DavMail from the .dmg. Run it and point it to the exchange server URL and have it expose POP3 and SMTP. See ~/.davmail.properties (though it's easy to configure from the UI as well).

I added DavMail to my "Login Items" so it starts when I login.

Homebrew

See the site for documentation on how to install.

getmail

Install with "brew install getmail". I keep my mail in ~/Mail, see ~/.getmail/getmailrc.

I have a cron job to run getmail every 3 minutes
    */3 * * * * /usr/local/bin/getmail -n -q

sup

sup is based on ruby. I've installed rbenv and ruby 1.9. "gem install sup" should work. (Note that I had trouble linking to ncurses, and had to run "brew link ncurses" before installing sup). See ~/.sup for configuration example.

sendmail

Sending mail is done via the sendmail program that comes with OSX. A bit of configuration is needed though. See /etc/postfix/main.cf and /etc/postfix/sasl_passwd

After the above changes, run the following commands:
    sudo postmap /etc/postfix/sasl_passwd
    sudo postfix reload

That's it, you should be set to go with a decent email client.

I also use iTerm2, which lets me click on a link on sup terminal and open it

Reading Passwords of OSX Keychain

OSX stores passwords in its keychain, which is sometimes useful when you forget a password. The command line security utility lets you access the keychain, however the output is somewhat cryptic.

Below is a little script to get user and password for a given domain.

    find-pass.py signup.netflix.com

Note you might get an image like the image on the right. Click either "Allow" or "Always Allow".

Dealing with Bad Memory

The family joke is that I was born senile (my joke is that I have 1bit memory). During the years I've developed some methods to help me be be productive with bad memory. Hope they'll help you as well (if you remember them :).

JFDI
That's the most effective method - do it at the moment you remember. It's pretty amazing how many things you can do "right now" without interrupting your flow too much. Once you did it - there's no need to remember.

Make it Impossible to Forget
Yeah, writing things down help - but I forget to look at my lists. However if you make things impossible to forget - then you won't forget. For example if I need to take something to the car, I'll place it at the front door - can't miss it when I go out.

Get Help
May they be electronic or human, get some help. I married a wife with a phenomonal memory, but it's not an option for everyone :) In this digital age you can find a good non human assistent to help you. Hiring a human assitenet doesn't have to be expensive - see Fancy Hands for example. 

Write It Down
We have a saying: "A short pencil is better than a long memory.". Write things down, it'll help you remember when you write then and later you can look them up. I use this blog as an memory of things that worked for me. Other things I use are pinboard, trello, GMail and Google Docs. Everything with a search function in it.
 
Forgive Yourself
You will forget things, learn to live with it. One of the worst things you can do it agonize over the things you forgot. It'll only add stress to your life without helping you to remember.  

Removing "noise" before matplotlib charts in IPython notebooks

By default, IPython notebook prints the value of the last expression. This works fine most of the time but sometimes the result of a chart is a long list of lines which is something you don't want to see - you just want to see the chart. Here are two options to fix this (you can also view the example notebook).

Decorators and Context Managers Workshop

I gave a workshop on decorators and context mangers at work.

Both are very powerful and will change the way you code in Python once you grok them. They mostly allow you to focus on application logic and write the "book keeping" code (logging, timing, resource allocation ...) in a separate place.

Here are the IPython notebooks:

• Context Managers
• Decorators

And the solutions (but try to solve the exercises first):

• Context Managers
• Decorators

 If you want to run locally, the code is here.

Searching and Viewing emoji

We started using emoji at work. We're using Adium with the emoji xtra.

Here's a little script to search and optionally show emojis (requires matplotlib).

Making "pip" Faster

We use pip to install packages at work. Mostly in combination with virtualenv and wheel packages. Lately I had to re-install again and again a bunch of virtual environments and looks for a way to speed things up.

The solution was to cache the pip downloads and also to keep in a local directory the bigger .whl packages. All done in ~/.pip/pip.conf. Most of the downloads are done once now and using local .whl means no more compilation (I'm looking at you PyYAML ...). I didn't measure the time difference, but for sure it feels faster.

A Simple Python Based Configuration with Overrides

Here's a simple Python based configuration system that supports overrides. We use a variant of it at work and it proves to be very handy. The fact that you can use all of Python in your configuration is very powerful. We use Puppet to write config_local.py in production/qa environments and keep the default good for development machines.

Import Google Bookmarks to Pinboard

I decided to give Pinboard a try. Sadly they lack support for importing Google Bookmarks.

I wrote a little utility to do just that (still waiting for pinboard support to get back to me ;)

Python Logging Overview

Gave a talk at work about Python Logging.

You can view it here.

logbot

We have an internal Jabber server at work, which is great for communication. However unlike HipChat/Campfire - it lacks central logging.

A couple of hours with sleekxmpp, Whoosh and flask solved the problem. logbot is out there - use with caution. (And pull requests are more than welcome ;)

"crashlog" is now on PyPi

FYI: Decided to make crashlog a PyPi package.
LMK if you find any problems.

JSON Handling with datetime Support

While working on some ETL testing code, we needed to load some synthetic data to a database. At first we thought of using YAML for the synthetic data, but it was very slow so we decided to switch to JSON which is still readable but way faster (loading 183 records took 0.68sec with YAML vs 0.008sec in JSON).

However we needed support for serializing datetime objects. Here's dtjson, which support datetime objects in JSON serialization. You can use it almost a drop-in replacement for json: import dtjson as json.