PythonWise

If it won't be simple, it simply won't be. [Hire me, source code] by Miki Tebeka, CEO, 353Solutions

Tuesday, November 13, 2018

direnv

I use the command line a lot. Some projects require different settings, say Python virtual environment, GOPATH for installing go packages and more.

I'm using direnv to help with settings per project in the terminal. For every project I have a .envrc file which specifies required settings, this file is automatically loaded once I change directory to the project directory or any of it's sub directories.

You'll need the following in your .zshrc

if whence direnv > /dev/null; then
    eval "$(direnv hook zsh)"
fi

Every time you create or change your .envrc, you'll need to run direnv allow to validate it and make sure it's loaded. (If you did some changes and want to check them, run "cd .")

Here are some .envrc examples for various scenarios:

Python + pipenv

source $(pipenv --venv)/bin/activate

Go

GOPATH=$(pwd | sed s"#/src/.*##")
PATH=${GOPATH}/bin:${PATH}

This assumes your project's path that looks like /path/to/project/src/github.com/project

If you're using the new go modules (in 1.11+), you probably don't need this.

Python + virtualenv

source venv/bin/activate

Python + conda

source activate env-name

Replace env-name with the name of your conda environment.

Wednesday, November 07, 2018

Go, protobuf & JSON

Sometimes you'd like more than one way to serve an API. In my case I'm currently working on serving both gRPC and HTTP. I'd like to have one place where objects are defined and have a nice way to serialize both from protobuf (which is the serialization gRPC uses) and JSON .

When producing Go code, protobuf adds JSON struct tags. However since JSON comes from dynamic languages, fields can have any type. In Go we can use map[string]interface{} but in protobuf this is a bit more complicated and we need to use oneof. The struct generated by oneof does not look like regular JSON and will make users of the API write complicated JSON structures.

What's nice about Go, is that we can have any type implement json.Marshaler and json.Unmarshaler. What's extra nice is that in Go, you can add these methods to the generated structs in another file (in Python, we'd have to change the generated source code since methods need to be inside the class definition).

Let's have a look at a simple Job definition


And now we can add some helper methods to aid with JSON serialization (protoc generates code to pb directory)


As a bonus, we added job.Properties that returns a "native" map[string]interface{}

Let's look at a simple example on how we can use it

And its output:
$ go run job.go
[j1]  user:"Saitama" count:1 properties: > properties:
[json]  {"user":"Saitama","count":1,"properties":{"retries":3,"target":"Metal Knight"}}

[j2]  user:"Saitama" count:1 properties: > properties:

Thursday, July 26, 2018

Specifying test cases for pytest using TOML

Say you have a function that converts text and you'd like to test it. You can write a directory with input and output and use pytest.parameterize to iterate over the cases. The problem is that the input and the output are in different files and it's not obvious to see them next to each other.

If the text for testing is not that long, you can place all the cases in a configuration file. In this example I'll be using TOML format to hold the cases and each case will be in a table in array of tables. You can probably do the same with multi document YAML.

Here's the are the test cases
And here's the testing code (mask removes passwords from the text)

When running pytest, you'll see the following:

$ python -m pytest -v
========================================= test session starts =========================================
platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0 -- /home/miki/.local/share/virtualenvs/pytest-cmp-F3l45TQF/bin/python
cachedir: .pytest_cache
rootdir: /home/miki/Projects/pythonwise/pytest-cmp, inifile:
collected 3 items                                                                                     

test_mask.py::test_mask[passwd] PASSED                                                          [ 33%]
test_mask.py::test_mask[password] PASSED                                                        [ 66%]
test_mask.py::test_mask[no change] PASSED                                                       [100%]

====================================== 3 passed in 0.01 seconds =======================================

Saturday, June 09, 2018

pexify - Package Python scripts using PEX

Sometimes you'd like to publish a simple Python script, but it depends on some external packages (e.g. requests). The usual solution is to create a package, which is a great solution but requires some work and might not be suitable for internal packages.

Another solution is to use PEX, which creates an executable virtual environment. The user running the script just needs a Python interpreter from the same version installed on their machine (which is usually the case).

Converting a script to PEX requires some work, to make this simpler I wrote pexify. which automates the process of creating a PEX from a single Python script.

Wednesday, April 18, 2018

Creating a Book with pandoc

One of my clients asked me for a printable book to accompany one of my workshops. My teaching style is more fluid and we write a lot of code in class. I wanted a quick way to take all the source code files and some images and make a book out of them.

Since I already work with markdown, I looked for a solution that can convert markdown to PDF. The winner was Pandoc (which can be installed with conda). Of course the out-of-the-box results weren't satisfactory and some tweaking was required. Mostly to make images appear where they are define in the markdown and not where pandoc/LaTex thinks is the optimal location.

The solution is composed of an awk script to add an include directive to markdown, a custom LaTex header to inline images and a Makefile to bind them all.

You can view the whole project here, including a example book. You can view the output here (decorators anyone?).

Tuesday, April 10, 2018

Installing Arch Linux on Laptop with UEFI

After some time with Xubuntu I decided to get back to Arch Linux.

Arch have a command line based installer, the installation instructions are pretty good but for a laptop with UEFI I had to do some extra steps. Here's what I came up with, hope you'll find it useful as well.

Most of my files are backed on "the cloud", my home directory with all the RC files is on a private git repository. Getting up and running after the initial install was pretty easy.

Tuesday, February 27, 2018

Python's iter & functools.partial

Python's built-in iter function is mostly used to extract an iterator from an iterable (if you're confused by this, see Raymond's excellent answer on StackOverflow).

iter has a second form where it takes a function with not arguments and a sentinel value. This doesn't see that useful but check out the readline example in the documentation. iter will call the function repeatably until it the function return the sentinel value and then will stop.

What happens when the function  producing the values require some arguments? I this case we can use functools.partial to create a zero arguments function.

Here's an example of a simple HTTP client using raw sockets, we can directly use join on the iterator.

Blog Archive