Using ImageMagick to Generate Images

One of the exercises we did this week in the Python workshop used the term "bounding box diagonal". I had a hard time to explain it to the students without an image. Google image search didn't find anything great, so I decided to create such an image.

First I tried with drawing programs, but couldn't make the rectangle a square and the circle non-oval. Then I remembered imagemagick, I have it installed and mostly use it to resize images - but it can do much more. A quick look at the examples and some trial and error, and here's the result.

And here's the script that generated it:

Waiting for HTTP Server - Go Testing

I don't like mocking in tests. If I have a server to test, I prefer to start and instance and hit the API in my tests. Waiting for the server to start with a simple sleep is unpredictable. I prefer to start the server, try to hit an URL until it's OK or fail after a long timeout. This is a simple task with Go's select and time.After.

Slap a --help on it

Sometimes we write "one off" scripts to deal with certain task. However most often than not these scripts live more than just the one time. This is very common in ops related code that for some reason people don't apply the regular coding standards to.

It really upsets me when I try to see what a script is doing, run it with --help flag and it happily deletes the database while I wait :) It's so easy to add help support in the command line. In Python we do it with argparse, and we role our own in bash. Both cases it's extra 3 lines of code.

Please be kind to future self and add --help support to your scripts.

vfetch - Fetch Go Vendor Depedencies

Go 1.6 now supports vendoring. I found myself cloning dependencies to "vendor" directory, then cloning their dependencies ... This got old really fast so vfetch was born. It's a quick and dirty solution, uses "go get" with a temporary GOPATH to get the package and its dependencies, then uses rsync to copy them to the vendor directory.

Installing is the usual "go get github.com/tebeka/vfetch" then you can use "vfetch github.com/gorilla/mux".

Comment, ideas and pull requests are more than welcomed.

Super Simple nvim UI

I've been playing with neovim lately, enjoying it and a leaner RC file. nvim comes currently only in terminal mode and I wanted a way to spin a new window for it. Here's a super simple script (I call it e) to start a new xfce4-terminal window with nvim.

Removing String Columns from a DataFrame

Sometimes you want to work just with numerical columns in a pandas DataFrame. The rule of thumb is that everything that has a type of object is something not numeric (you can get fancier with numpy.issubdtype). We're going to use the DataFrame dtypes with some boolean indexing to accomplish this.

In [1]: import pandas as pd  

In [2]: df = pd.DataFrame([
   ...:     [1, 2, 'a', 3],
   ...:     [4, 5, 'b', 6],
   ...:     [7, 8, 'c', 9],
   ...: ])  

In [3]: df  
Out[3]: 
   0  1  2  3
0  1  2  a  3
1  4  5  b  6
2  7  8  c  9

In [4]: df.dtypes  
Out[4]: 
0     int64
1     int64
2    object
3     int64
dtype: object

In [5]: df[df.columns[df.dtypes != object]]
Out[5]: 
   0  1  3
0  1  2  3
1  4  5  6
2  7  8  9

In [6]:   

Forging Python - First Chapter is Up

Finally, first chapter of my upcoming book "Forging Python" is up. I'm doing it leanpub style so comments ans suggestions are more than welcomed.

I plan to finish the book this year, hopefully during the summer. However more than one person said I'm way too optimistic - time will tell :)

353Solutions - 2015 in Review

Happy new year!

First full calendar year that 353solutions is operating. Let's start with the numbers and then some insights and future goals.

Numbers

• 170 days of work in total
• Work day is a day where I billed someone for some part of it
• Can be and hour can be 24 hours (when teaching abroad)
• There were total of 251 work days in 2015
• There were some work days that are not billable (drafting syllabuses, answering emails ...) but not that many
• 111 of days consulting to 4 clients
• 1st Go project!!!
• 58 days teaching 14 courses
• Python at all levels and scientific Python (including new async workshop)
• In UK, Poland and Israel

Insights

• Social network provided almost all the work
• Keep investing in good friends (not just for work :)
• Workshops pay way more than consulting
• However can't work from home in workshops
• Consulting keeps you updated with latest tech
• Had to let go of a client due to draconian contract
• No regrets here, it was the right decision
• Super nice team. Sadly lawyers had final say the company
• Python and data science are big and in high demand
• Delegating overhead to the right person helps a lot
• Accounting, contracts ...

Future Goals

• Keep positioning in Python and Scientific Python area
• Drive more Go projects and workshops
• Works less days, have same revenue at end of year
• Start some "public classes" where we rent a class and people show up
• Some companies don't have big enough data science team
• Need to invest in advertising
• Publish my book (more on that later)

Using HAProxy to Prevent Deletes from Elasticsearch

At one of my clients, we wanted something quick and dirty to prevent deletes from Elasticsearch (shield is too expensive and would take too much time to integrate with our systems - we'll fix this technical debt later).

The quick solution was to place HAProxy in front of Elasticsearch and use its acl mechanism to prevent HTTP DELETE. Works like a charm.

Here's the HAProxy configuration and the docker-compose setup file I used to test the configuration.

Python's deque for Go

Working on a Go project with my friend Fabrizio, I've investigated ways to have a faster data structure to store history items with append and pop.

Got the idea to try implementing Python's deque in Go. The C implementation is pretty easy to read. The result is deque for Go, which implement a subset of the features from Python's deque but enough for our needs. And it's pretty fast too:

$ make compare
Git head is 765f6b0
cd compare && go test -run NONE -bench . -v
testing: warning: no tests to run
PASS
BenchmarkHistAppend-4  3000000        517 ns/op
BenchmarkHistList-4    2000000        702 ns/op
BenchmarkHistQueue-4   3000000        576 ns/op
BenchmarkHistDeque-4   3000000        423 ns/op
ok   _/home/miki/Projects/go/src/github.com/tebeka/deque/compare 8.505s

aenumerate - enumerate for async for

Python's new async/await syntax helps a lot with writing async code. Here's a little utility that provides the async equivalent of enumerate.

git - Creating Pull Request for master

A co-worker asked me for a code review (we're using Stash, but this can work for other systems as well), the problem was that he worked on master (started his own project) and not in development branch. The solution was to create an empty orphan branch and then a pull request from master to that branch (reverse the usual order).

Here's how to create such branch.

Go Tour Exercise Solutions

As a backup plan for the last Go Meetup, I wrote the solutions to the exercises in Go Tour and we discussed some of them.

You can find the solutions here.

re2 available on conda

We're using re2 to get some speed gains on the many regular expressions we're trying to match. So far building it was either a manual step or a script that ran when building docker container. I decided to create a conda package (we're using Miniconda as our Python environment).

I started with conda skeleton pypi re2 (you need to conda install conda-build first). Then after some tweaking to build.sh we were good to go.

The result - you can now conda install -c tebeka re2 (only 64bit linux supported currently).

The project is here, I'll gladly accept any comments/improvements.

Here's build.sh which patches re2 Makefile and added the library and header location to the Python build step.

fastavro moved to github

If you can't beat them ... :)

fastavro is now on github. I still prefer mercurial as an SCM but most of the pull requests I get are on github and it doesn't worth the effort of maintaining two repositories (though hg-git is a big help)

dockermon - A Docker Event Monitor

I'm currently working with the awesome team at CyberInt (and yes, they are hiring).

We're moving to a docker based environment. The old environment used Supervisor to monitor and relaunch daemons. We had an event listener that notified us on our HipChat room every time a daemon crashed and wanted the same feature with our docker containers.

We didn't find a ready solution, so we wrote one and made it open source. The project is called dockermon and is one Python script with no external dependencies and also Python 2 and 3 compatible.

Naming "with open" Variable

Python''s "with" statement is great for resource handling. However I find my self struggling with naming (and naming is important) the context manager variable.

When you write "with open(''/path/to/somethere'') as X", what''s the best name for X? In some cases it''s obvious, but in most cases I find myself using the generic "fo" (stands for "file object").

I decided to run a little script on Python''s 3.4 Lib directory and find out what is the most common name. Here are the results:

Seems like f is the most common, but I really don''t like single letter variables. I''ll go with the 2nd place - fp.

Here''s the script used to generate this chart:

353Solutions - A Year in Review

353Solutions was founded a bit more than a year ago. I wasn't planning on doing consulting, I'm a techie and love the development abstraction layer that companies give you and let you code most of the time. (If this is not the case in the company you're working at - consider finding a better one :)

However as the old saying goes - "Man plans and god laughs". I found myself owning a teaching/consulting company called 353Solutions. So far it's fun and provides for the family - what else can you ask for?

Here are results from a short retrospective we did lately.

The Numbers

• 6 clients
• 204 work days
• 204 hours teaching Python (7 courses)

Thoughts

I like working from home, however most companies I talked to wanted some office time. This is understandable since I don't only code but also do system and process design - these roles require more face to face communication. I'm still looking for something that will allow me to spend most of my time working from home.

Teaching is fun! I did that on and off most of my professional carrier, but now it's a big chunk of my time. I'm grateful to Raymond Hettinger who started me off and showed me what a top-notch class/workshop should look like. So far I'm mostly going to companies and teaching there, but just now we launched our own classes - it will  be awesome!

The downside for teaching is that it takes me away from home. For a limited amount this is great (I spend about a week every month teaching Python in the UK). However I'm looking for opportunities that will let me teach from home - stay tuned.

The social network is by far my biggest source of new jobs. Talking to other people - it's not just me. Investing time in making connections and keeping them will pay off. The main downside for is that people want to hire me and not 353Solutions. This means I need to work harder to market the other people who I work with - I can't do everything.

Learning to say "no" was the hardest thing for me. So many interesting things to do, so many cool companies ... But I like spending time with my family, friends and hobbies. You need to find the things that make you happy and pay enough, going cheap is not a good thing in most cases. What I did in some cases was to take less money and get equity instead. Something like "technical" investing in startups.

The main point I need to improve is marketing. It's not something I like to do but feel the need, especially now that we have our own classes. I'm learning and looking for the best thing that will get maximal impact with minimal amount of time. Or maybe hire someone for that? If you know a good option  - please let me know :)

Use contextlib.closing to Handle "Legacy" Resources

Python''s context managers (with statement) are very handy at handling resources. (You see way less finally in Python code due to them). Maybe objects in Python can be used as context mangers - files, locks, database drivers and more. But some objects still do not.

To handle these "legacy" objects you can use contextlib.closing function which will return a context manager that will call obj.close() once the context manager exists.

Here''s an example of using contextlib.closing with sockets. We''ll be doing a simple HTTP request (Yeah, you should probably use requests or urlopen - this is just an example :)

Note also the user of iter with sentinel to read chunks up to 1K from the socket.

Combining jQuery and Multi Components of React

React is a great library for generating reactive web UI (and mobile). Reacts works well if there are isolated components or a big one with hierarchy. However I wanted to have a page with several isolated react components that are updated from the same data. The solution I found is to use an observer pattern and have each components register a callback to handle data change. See the code below and a live demo here.

Note that I am a React newbie, if you know of a better way - please enlighten me.