Advent of Code 2017 #8 and Python's eval

I'm having fun solving Advent of Code 2017. Problem 8 reminded the power of Python's eval (and before you start commenting the "eval is evil" may I remind you of this :)

You can check The Go implementation that don't have eval and need to work harder.

Python → Go Cheat Sheet

I'm teaching and consulting in Go a lot lately, and I work a lot with Python as well. There's a big trend of rewriting backend services in Go and to help people coming from the Python world I've created a Go →Python cheatsheet.

The code in here, I'd love to hear if you have suggestion (via a PR ;)

Finding Where Most Activity Time Was Spent

View Notebook

Checking for Zero Values in Go

In Go, every type has a zero value. Which is the value a variable of this type get if it's not initialized. I had a configuration object of type map[string]interface{} and I needed to check if value exists and is not a zero value.

Here's a small piece of code that checks for zero values:

Generating Power Set using Bitmap

I was asked to write a function that generate a power set of items. At first I wrote a recursive algorithms but then another approach came to mind. When you calculate how many subsets there are, you can say that each item in the original set can either be or not be in a subset, which means 2^n subsets. This yes/no for including can be seen as a bitmask, and since we know that there are 2^n subsets we can use the number from 0 to 2^n-1 as bitmasks.

Who Touched the Code Last? (git)

Sometimes I'd like to know who to ask about a piece of code. I've developed a little Python script that shows the last people who touch a file/directory and the ones who touched it most.

Example output (on arrow project)
$ owners
Most: Wes McKinney (39.5%), Uwe L. Korn (15.3%), Kouhei Sutou (10.8%)
Last: Kengo Seki (31M), Uwe L. Korn (39M), Max Risuhin (23H)

So ask Wes or Uwe :)

Here's the code:

Go's append vs copy

When we'd like to concatenates slices in Go, most people usually reach out for append. Most of the time this solution is OK, however if you need to squeeze more performance - copy will be better.

EDIT: As Henrik Johansson suggested, if you pre-allocate the slice to append it'll fast as well.

scikit-learn Compatible Pipeline Steps

A client wanted a way to create a pipeline of transformations on DataFrames. Since they already work with scikit-learn, they were familiar with Pipelines. It took very little code to create a base class for a pipeline step that will work with DataFrames.

---

If you find this interesting, you might want to hire me ;)

Color Log Lines

There are several log handler out there the color log lines. However I prefer to leave the log as is and when I want colors I pipe the log via a utility that does that. Coloring log lines by levels can be easily done with a little awk.

Here's an example how it looks

353Solutions - 2016 in Review

Happy new year!

Let's see how 353solutions did in 2016. We'll start with the numbers and then some insights and future goals.

Numbers

• Total of 193 work days (days where customer were billed)
• Up 23 days from 2015
• Out of 261 work days in 2016
• Out of these 61 days are in workshops and the rest consulting
• 18 workshops
• Up from 14 in 2015
• Go workshops
• First docker workshop (teaching with ops specialist)
• First open enrollment class
• In Israel, UK, and Poland
• Only 4 out of the country (vs 8 last year, which is good - I prefer to fly less)
• Workshops range from 1 to 4 days
• Revenue up 10% from 2016
• Several new clients including GE, AOL, Outbrain, EMC and more

Insights

• Social network keeps providing almost all the work
• Doing good work will get you more work
• Workshops pay way more than consulting
• However can't work from home in workshops
• Consulting keeps you updated with latest tech
• Python and data science are big and in high demand
• However Go is getting traction
• Having someone else take care of the "overhead" (billing, contracts ...) is great help.

Last Year's Goals
We've set some goals in 2015, let's see how we did:

• Keep positioning in Python and Scientific Python area
• Done
• Drive more Go projects and workshops
• Done
• Works less days, have same revenue at end of year
• Failed here. Worked more and revenue went up
• Start some "public classes" where we rent a class and people show up
• Did only one this year
• Publish my book
• Failed here, though I'm close. Probably Q1 this year.

Goals for 2017

• Work less while keeping same revenue
• Work more from home
• Publish my book
• Publish a video course
• More open enrollment classes

Automatically Running BigQuery Flows

At a current project we're using Google's BigQuery to crunch some petabyte scale data. We have several SQL scripts that we need to run in specific order. The below script detects the table dependencies and run the SQL scripts in order. As a bonus you can run it with --view and it'll show you the dependency graph.

Using built-in slice for indexing

At one place I consult I saw something like the following code:
This is good and valid Python code, however we can use the slice built-in slice to do the same job.

Also when you're writing your own __getitem__ consider that key might be a slice object.

Weighted Rating

Notebook here.

First Contact with Data

Notebook here.

Simple Object Pools

Sometimes we need object pools to limit the number of resource consumed. The most common example is database connnections.

In Go we sometime use a buffered channel as a simple object pool.

In Python, we can dome something similar with a Queue. Python's context manager makes the resource handing automatic so clients don't need to remember to return the object.


Here's the output of both programs:

$ go run pool.go
worker 7 got resource 0
worker 0 got resource 2
worker 3 got resource 1
worker 8 got resource 2
worker 1 got resource 0
worker 9 got resource 1
worker 5 got resource 1
worker 4 got resource 0
worker 2 got resource 2
worker 6 got resource 1

$ python pool.py
worker 5 got resource 1
worker 8 got resource 2
worker 1 got resource 3
worker 4 got resource 1
worker 0 got resource 2
worker 7 got resource 3
worker 6 got resource 1
worker 3 got resource 2
worker 9 got resource 3
worker 2 got resource 1

"Manual" Breakpoints in Go

When debugging, sometimes you need to set conditional breakpoints. This option is available both in gdb and delve. However sometimes when the condition is complicated, it's hard or even impossible to set it. A way around is to temporary write the condition in Go and set breakpoint "manually".

I Python we do it with pdb.set_trace(), in Go we'll need to work a little harder. The main idea is that breakpoints are special signal called SIGTRAP.

Here's the code to do this:
You'll need tell the go tool not to optimize and keep variable information:

$ go build -gcflags "-N -l" manual-bp

Then run a gdb session

$ gdb manual-bp 

(gdb) run 

 When you hit the breakpoint, you'll be in assembly code. Exit two functions to get to your code

(gdb) fin
(gdb) fin

Then you'll be in your code and can run gdb commands

(gdb) p i
$1 = 3

This scheme also works with delve

$ dlv debug manual-bp.go 
(dlv) c 

Sadly delve don't have "fin" command so you'll need to hit "n" (next) until you reach your code. 

That's it, happy debugging.

Oh - and in the very old days we did about the same trick in C code. There we manually inserted asm("int $3)" to the code. You can do with with cgo but sending a signal seems easier.

Generate Relation Diagram from GAE ndb Model

Working with GAE, we wanted to create relation diagram from out ndb model. By deferring the rendering to dot and using Python's reflection this became an easy task. Some links are still missing since we're using ancestor queries, but this can be handled by some class docstring syntax or just manually editing the resulting dot file.

Testing numpy Code

Notebook here.

Work with AppEngine SDK in the REPL

Working again with AppEngine for Python. Here's a small code snippet that will let you work with your code in the REPL (much better than the previous solution).
What I do in IPython is:

In [1]: %run initgae.py

In [2]: %run app.py

And then I can work with my code and test things out.

Using ImageMagick to Generate Images

One of the exercises we did this week in the Python workshop used the term "bounding box diagonal". I had a hard time to explain it to the students without an image. Google image search didn't find anything great, so I decided to create such an image.

First I tried with drawing programs, but couldn't make the rectangle a square and the circle non-oval. Then I remembered imagemagick, I have it installed and mostly use it to resize images - but it can do much more. A quick look at the examples and some trial and error, and here's the result.

And here's the script that generated it: