PythonWise

If it won't be simple, it simply won't be. [Hire me, source code] by Miki Tebeka, CEO, 353Solutions

Thursday, March 05, 2020

Using __getattr__ for nicer configuration API

Typically, you'll read configuration from files (such as YAML) and get them as a dictionary. However in Python you'd like to write config.httpd.port and not config['httpd']['port']

__getattr__ is a hook method that's called by Python when regular attribute lookup fails (not to be confused with the lower level __getattribute__, which is much harder to work with). You can use it to wrap the configuration dictionary. Here's a small example.


Tuesday, February 04, 2020

Adding a timer to Go's "present" tool

Go has a nice present tool which let's you present code and run it. I wanted to use it in my lightning talk at GopherCon Israel. The talk is a quiz and I wanted a timer to show on each quiz slide.

present syntax supports .html directive which lets you add any HTML code to the slide. From there it easy to add a timer. I wrote timer.html below and added .html timer.html to the first slide.

Here's how it looks (bottom left):

Here's the code:

Sunday, January 05, 2020

Nested structs for HTTP Calls

Some API's return complex structures. Here's for an example from Stocktwits API.
Instead of creating a specific type for every element in the structre, we use use a anonymous structure and define only the fields we're intrested in. In our example we'd like to see the symbols most mentioned for a given symbol (in our case AAPL). And the output:
$ go run stocktwits.go
SPY    10
TSLA    7
AMZN    5
BTC.X   5
TVIX    2

Monday, August 12, 2019

viaenv

These days, I write a lot of Go and a lot of Python (and some other languages in the mix). What's great is that I'm getting ideas from both ecosystems and find myself more productive.

In the Go world, I like Kelsey Hightower's envconfig for a simple application configuration from environment variables. I couldn't find something similar in the Python world, so I wrote viaenv which uses variable annotation.

One design decision I took is not to invent my own serialization format when getting lists & dicts from the environment (where everything is a string). I'm using JSON, there's really no need to invent yet another serialization format (as envconfig does).

The code itself is not that long (less than 150 lines of code). I'm showing it below:


Wednesday, March 20, 2019

Speed: Default value vs checking for None

Python's dict has a get method. It'll either return an existing value for a given key or return a default value if the key is not in the dict. It's very tempting to write code like val = d.get(key, Object()), however you need to think about the performance implications. Since function arguments are evaluated before calling the function, this means the a new Object will be created regardless if it's in the dict or not. Let's see this affects performance.

get_default will create new Point every time and get_none will create only if there's no such object, it works since or evaluate it's arguments lazily and will stop once the first one is True.

First we'll try with a missing key:

In [1]: %run default_vs_none.py                                     
In [2]: locations = {}  # name -> Location 
In [3]: %timeit get_default(locations, 'carmen')
384 ns ± 2.56 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [4]: %timeit get_none(locations, 'carmen')
394 ns ± 1.61 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Not so much difference. However if the key exists:

In [5]: locations['carmen'] = Location(7, 3)
In [6]: %timeit get_default(locations, 'carmen')                 
377 ns ± 1.84 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [7]: %timeit get_none(locations, 'carmen')
135 ns ± 0.108 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

We get much faster results.

Monday, March 04, 2019

CPU Affinity in Go

Go's concurrency unit is a goroutine, the Go runtime multiplexes goroutines to operating system (OS) threads. At an upper level, the OS maps threads to CPUs (or cores). To see this goroutine/thread migration, you'll need to use some C code (note that this is Linux specific).


If you run this code and sort the output, you'll see the workers moving between cores:
$ go run affinity.go | sort -u
worker: 0, CPU: 0
worker: 0, CPU: 1
worker: 0, CPU: 2
worker: 0, CPU: 3
worker: 1, CPU: 0
worker: 1, CPU: 1
worker: 1, CPU: 2
worker: 1, CPU: 3
worker: 2, CPU: 0
worker: 2, CPU: 1
worker: 2, CPU: 2
worker: 2, CPU: 3
worker: 3, CPU: 0
worker: 3, CPU: 1
worker: 3, CPU: 2
worker: 3, CPU: 3

There is a cost for moving a thread from one core to another (see more here). In some cases this cost might be unacceptable and you'd like to pin a goroutine to a specific CPU.

Go has runtime.LockOSThread which pins the current goroutine to the current thread it's running on. For the rest of the way - pinning the thread to a CPU, you'll need to use C again.

Now if you run our code with the LOCK environment variable set, you can see each worker is pinned to a specific core.
$ LOCK=1 go run affinity.go | sort -u
worker: 0, CPU: 0
worker: 1, CPU: 1
worker: 2, CPU: 2
worker: 3, CPU: 3

Thursday, January 17, 2019

Place a string in the middle of the screen - in bash

For ages my shell is greeting me with "Let's boogie, Mr Swine". I had it printed in fixed offset but following a discussion in comp.lang.python I decided to get the screen size and calculate the exact location.

I'm using tput to get the size and the fact that printf can get the width of the string as a parameter if you specify a * there (BTW - Python supports that as well)

Here's the code.

Blog Archive