Quick View of matplotlib Styles

matploblit 1.4.2 added support for styles. I created a notebook to show how the ones bundled in look, you can view it here.

Calculate Distance in .kmz File

Here's a little script that calculates the distance of a path (series of GPS points) in a .kmz file (which I generate with My Tracks). It combines several elements - Zip file, XML (with namespaces), zip, sum and of course a bit of math.

The Versatile date Command

*nix systems comes with a date command line utility. Many people use it to view the current time, however there''s much more that date can do.

For example we run a daily job to process yesterday data. The job is a Python script that get the date as a paramter in YYYYMMDD format. This translates to one cron line:

@daily /path/to/job.py $(date --date=yesterday +%Y%m%d)

Here are some things you can do with date:


As a bonus, there is also a cal command which displays calender. Note that years are 4 digits so cal 4 14 will diaplay April for the year 14, not 2014.

cal has a handy -w switch that shows the work week as well.'

Generate QR Code Using Google Charts API

Here's a small utility to generate QR code image using Google Charts API.

Note the hand crafted Python 2/3 support, for more advanced stuff you might want to have a look at six. However for this script I wanted to stay without external dependencies.

Common Errors

"Experience is the name every one gives to their mistakes."

    - Oscar Wilde

Students are sometimes amazed by how I can tell the cause of exception without looking at the code. This is the result of making many errors in Python myself and also from observing many students doing mistakes. Here's a short list I've compiled of most likely cause of errors:

NameError

• You forgot to import a module
• You made a typo

AttributeError: 'NoneType' object has no attribute ...

• You forgot a return in your function

AttributeError

• Typo on dot lookup (obj.foo)
• Object is different type from what you think (str vs int)
• Object does not implement a dunder method (e.g. __len__)

There's also the 3'rd party didyoumean module, which might be interesting for beginners. It changes the default stack trace to add a hint about what might be the problem.

A Streaming Chart using Flask and flot

I was teaching a course on "Python Analytics" (pandas, scikit-learn, matplotlib ...) and was asked to provide an example of streaming chart - ones that updates periodically. I''ve showed a couple of examples one by generating image using matplotlib and another with bokeh-server. After a couple of days I remembered another way - using flot to render the chart. Here''s a small example using Flask as the web server (the code works both on Python 2 and 3).


Few comments:

• Don''t use debug=True in production :)
• For simplicity everything is in one file. However for larger application you might want to take the HTML template(s) out
• Data is in memory, a restart will wipe it out. If you need data persistence - pick a database (shelve, sqlite3 ...)
• A big shoutout to Continuum Analytics - Anaconda (and conda) has made my life so much easier teaching this workshop.

Resolve SSH Host Name

~/.ssh/config let''s you give meanigful names to hosts/ips. But sometimes you want the reverse lookup - what''s the ip of web1? Here''s a little Python script that does that.

EDIT: EAFP > LBYL

Archlinux Install Steps (on VirtualBox)

My favorite Linux distro for using under VirtualBox is Archlinux with XFCE window manager. It''s light, fast and has all the latest shiny new toys (just the way I like it :).

I found myself setting up VMs to try things out and wrote down the steps I use, this is a trimmed down version of the Installation part in the Archlinux beginners guide.

Be a Better Developer by Coding in Four Different Types of Lauguages

I like programming languages and find out that every time I learn a new language it improves my coding in the other ones as well. I learn new ways of doing things, different ways of thinking and it's great.

I usually tell new developers they need to write a (small) project in at least four types of languages - mainstream (procedural/OO), functional, logic based and assembly. Each of these types will give you a different way of solving problems and enrich your programming experience by and order of magnitude.

Here are my recommendations for each category.

Main Stream

By "main stream" I mean procedural/OO languages. There are tons of these and its up to what you're working with currently (though it might be a good excuse to learn a new language). I'm a Python expert, but pick anything - Go, JavaScript, Ruby, C, C++, Java, C# ...

(Yeah - I know they differ a lot. But thinking in most of them is probably the same. The main difference will probably be dynamic vs static typing).

Functional

Many choices here as well. Personally I like the Lisp family of languages, mostly Clojure and Scheme but you can check out a Common Lisp implementation (I think SBCL leads the pack currently), Haskell, ML and others.

Logic Based

If you haven't done logic programming - it'll blow your mind! It's a totally different way of thinking. Prolog is the main language, one free implementation is SWI but there are others as well.

Assembly

Learning assembly will give you a better understanding on how computers work and what are the abstractions other programming languages do for you. I recommend picking one that targets the machine you're working on.

Add That Trailing Comma

Lets say you wrote this simple code and at first things were going well.

Then after a while a friend came in and did a little fix.

But things started falling apart, after a lot of digging in - you find this.

Python will join two string together in this case, not what you wanted. Always leave a trailing comma.

Couple more things:

• As Dave Cheney pointed out, using this practice has the nice effect that one line change shows as one line change in the diff since you don''t have to add a comma to the previous line.
• Go probably learned from Python and made trailing comma mandatory.
• You can see more Python "gotchas" here.

draft2gist - Publish draftin.com Documents to gist

I've started playing with draftin.com, so far very nice.

draftin.com lets you publish documents to several sites, and if the site you want to publish to is not on the list - there are WebHooks.

I've written a small AppEngine service that is a WebHook for publishing draftin.com documents to gist. Feel free to use it and let me know if you find any errors.

Below is the server code, rest of the files are here.

HTTP Proxy Stripping Headers (go)

At one of my clients, we wanted to write an HTTP proxy that strips some of the headers sent from the target (backend). With the help of golang-nuts mailing list - this turned out to be pretty simple.

timeat on pypi

timeat is now on pypi. There's some extra code to get current time from NTP server. Should work both on Python 2.x and 3.x

timeat in Go

Wrote timeat, which shows time at a specific location, in Go as well. (For comparison the Python version is here). To install either go get bitbucket.org/tebeka/timeat or download the executables.

Working in a multi-timezone team, this script comes handy from time to time (as well as worldtimebuddy :).

Generating Byte Arrays for Assets in Go using xxd

Go's net/http server is pretty fast, but sometimes you want to get faster. One way to do that is to create a binary array in memory for static files (assets). Here''s how I generate the byte arrays automatically with xxd.

Note that this makes the build go slower as you have more and bigger assets.If this is a problem, take a look at nrsc ;)

Since the go toolchain does not support custom steps currently, I''m using make.

Makefile

httpd.go

Hook to Update Tag List when Changing git Branch

I use ctags with Vim to move around. At work we use git feature branches, which means code changes when you switch a branch. Here''s a very simple post-checkout hook to update the tag list whenever you switch branches.

Use dict to Speed Up Your Code

In Python, dictionary access is very fast. You can use that to get some speedup in your code  by replacing if/else with dictionary get.

In [1]: fn1 = lambda x: 1 if x == 't' else 0

In [2]: fn2 = {'t': 1, 'f': 0}.get

In [3]: %timeit fn1('y')  # Check "True" branch
10000000 loops, best of 3: 124 ns per loop

In [4]: %timeit fn2('y')
10000000 loops, best of 3: 79.6 ns per loop

In [5]: %timeit fn1('f')  # Check "False" branch
10000000 loops, best of 3: 125 ns per loop

In [6]: %timeit fn2('f')
10000000 loops, best of 3: 81.3 ns per loop

About 30% speedup - not bad :)

HTTPDir - Small OSX Utility to Serve a Directory over HTTP

HTTPDir is a small utility that lets you serve content of a directory over HTTP.

This is handy when you develop static sites that has reference to external resources. It is also aimed to people who are not comfortable with the command line.

HTTPDir is a simple Python script that uses Tkinter. It is packed in a format that OSX recognizes as an application. See the code here (look under HTTPDir.app/Contents/MacOS).

"timeat" updated

timeat updated to use the new(ish) Google Time Zone API.

$ timeat haifa
Haifa, Israel: Fri May 23, 2014 22:16
$ timeat paris
Paris, France: Fri May 23, 2014 21:17
Paris, TX, USA: Fri May 23, 2014 14:17
Paris, TN 38242, USA: Fri May 23, 2014 14:17
Paris, IL 61944, USA: Fri May 23, 2014 14:17
Paris, KY 40361, USA: Fri May 23, 2014 15:17

Oh, and there's also a Go version.

Important vs Urgent

This was a lesson I learned a long time ago, maybe other people will find it useful as well.

Look at the chart above, we divide tasks to four categories on the urgent/important dimensions. Let's look at each category:

1. Not urgent and important - this is where you (and most successful companies) should spent your time. This means investing in infrastructure, making your process more efficient ...
2. JFDI. However if you find that most of your time is spent there - you're doing something wrong.
3. Don't bother.
4. This is the big time sucker. Try to avoid at all cost.

Note: "If everything is important, then nothing is." - Patrick Lencioni

See more here, here and here.

Easy statsd metrics decorator and context manager

Easy statsd metrics decorator and context manager statsd is very handy with creating metrics. Here's a decorator and context manager that simplify the usage even more.
Note: If you want to easily test your code, you can use this Vagrant VM.

Running commands via HTTP

Ran asked about invoking commands via HTTP interface. Here's a quick answer I came up with (using Flask).

pypi→u

I was trying to see if there's an annoucement list for Celery (which we use at work) - didn't find it. One thing led to another... and I wrote pypi→u, a service that emails you if there's a new version of packages your interested in.

It run on AppEngine (with some bootstrap sprinkeld on). You can view the source here.

Note that pypi→u is very alpha, handle with care. Suggestions, comments and bug reports are welcomed.

Current email setup: osx + homebrew + sup + davmail + getmail

 At work we have an Exchange server for email (no POP3/IMAP) and most people use Outlook. However I prefer not to use outlook, here's what I came with:

• Sup as email client
• DavMail for POP3/SMTP
• getmail for fetching email
• Homebrew for unix-ish environment (yup, that's OSX for you)

 Setup is a bit complicated, but once you're there - email is a bliss (well, as much as email can be :)

DavMail

Install DavMail from the .dmg. Run it and point it to the exchange server URL and have it expose POP3 and SMTP. See ~/.davmail.properties (though it's easy to configure from the UI as well).

I added DavMail to my "Login Items" so it starts when I login.

Homebrew

See the site for documentation on how to install.

getmail

Install with "brew install getmail". I keep my mail in ~/Mail, see ~/.getmail/getmailrc.

I have a cron job to run getmail every 3 minutes
    */3 * * * * /usr/local/bin/getmail -n -q

sup

sup is based on ruby. I've installed rbenv and ruby 1.9. "gem install sup" should work. (Note that I had trouble linking to ncurses, and had to run "brew link ncurses" before installing sup). See ~/.sup for configuration example.

sendmail

Sending mail is done via the sendmail program that comes with OSX. A bit of configuration is needed though. See /etc/postfix/main.cf and /etc/postfix/sasl_passwd

After the above changes, run the following commands:
    sudo postmap /etc/postfix/sasl_passwd
    sudo postfix reload

That's it, you should be set to go with a decent email client.

I also use iTerm2, which lets me click on a link on sup terminal and open it