tag:blogger.com,1999:blog-278540512024-03-16T20:52:52.214+02:00PythonWiseIf it won't be simple, it simply won't be. [<b><a href="http://353solutions.com">Hire me</a></b>, <a href="http://github.com/tebeka/pythonwise/">source code</a>]Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.comBlogger323125tag:blogger.com,1999:blog-27854051.post-75370293054483360472021-08-24T13:12:00.002+03:002021-08-24T13:12:31.012+03:00Using gpg for SecretsI'm using <a href="https://direnv.net/" target="_blank">direnv</a> in my project, and sometimes want to set secrets such as <code>AWS_ACCESS_KEY_ID<code> and friends. I usually have a file like:
<code><pre>
export AWS_ACCESS_KEY_ID=ABCDEFGHIJK
export AWS_SECRET_ACCESS_KEY=ABCDEFGHIJK/lmnopqrstuvwxyz
export AWS_DEFAULT_REGION=us-east-1
</pre></code>
However I don't want to store these values in the clear. The solution I found was to encrypt the file with gpg and then have the following line in the <code>.envrc</code>:
<code><pre>
eval $(gpg -qdo- ../../../aws.sh.gpg)
</pre></code>Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-39880805912844231782021-07-20T17:03:00.001+03:002021-08-24T13:13:13.841+03:00Go's "defer" in pytest (and Python)<p> I'm writing a lot of Go & Python these days, and both affect the way I write code.</p><p>One feature I really like in Go is <a href="https://tour.golang.org/flowcontrol/12" target="_blank">defer</a>. Lately, after writing some complicated <a href="https://docs.pytest.org/en/6.2.x/fixture.html" target="_blank">pytest fixtures</a>, I tried to make simpler and use a "defer" like fixture. Here's what I came with:</p>
<script src="https://gist-it.appspot.com//github/tebeka/pythonwise/blob/master/py_defer/test_defer.py"></script>
<p>Out of the context of pytest, the same idea can become a context manager:
<script src="https://gist-it.appspot.com//github/tebeka/pythonwise/blob/master/py_defer/defer.py"></script>
</p><p><br /></p>
Update: I've published <a href="https://github.com/tebeka/pytest-defer">pytest-defer</a> to PyPI.Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com3tag:blogger.com,1999:blog-27854051.post-64629127477958958962020-07-09T07:52:00.000+03:002020-07-09T07:52:41.965+03:00Using module __dir__ and __getattr__ for configuration<a href="https://www.python.org/dev/peps/pep-0562/" target="_blank">PEP 562</a> added support for module level <font face="courier">__dir__</font> and <font face="courier">__getitem__</font>. <div><ul style="text-align: left;"><li><font face="courier">__dir__</font> is called when the built-in <a href="https://docs.python.org/3/library/functions.html#dir" target="_blank">dir</a> function is called on the module</li><li><font face="courier">__getattr__</font> is called when an attribute is not found via the regular attribute lookup</li></ul><div>Let's use this to build an environment based configuration module. </div><div><ul style="text-align: left;"><li>Conviruation values has a value, environment key and a function to convert from str to right type</li><ul><li>I'm going to use <a href="https://docs.python.org/3/library/dataclasses.html" target="_blank">dataclasses</a> and populate values from environment in <a href="https://docs.python.org/3/library/dataclasses.html#post-init-processing" target="_blank">__post_init__</a></li><li>Complex data types (such as list) should be JSON encoded in the environment variables</li></ul><li>All configuration values with start with the <font face="courier">c_</font> prefix</li><li><font face="courier">__dir__</font> will return a list configuration variables without the <font face="courier">c_</font> prefix</li><li><font face="courier">__getattr__</font> will add the <font face="courier">c_</font> prefix and will look for the varialbes in <a href="https://docs.python.org/3/library/functions.html#globals" target="_blank">globals</a></li></ul><div>We're adding c_ prefix and removing it to bypass the regular attribute lookup mechanism. If we'll call a variable <font face="courier">http_port</font> and user will write <font face="courier">config.http_port</font>, our <font face="courier">__dir__</font> function won't be called.</div></div></div><div><br /></div><div>Here's the code</div><div><br /></div>
<script src="https://gist-it.appspot.com//github/tebeka/pythonwise/blob/master/config/config.py"></script>Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com13tag:blogger.com,1999:blog-27854051.post-87957561732076253542020-04-09T14:01:00.000+03:002020-04-09T14:01:35.949+03:00Functions as State Machine (Go)Let's say you'd like to check that a string is a valid floating point (e.g. "3.14", ".2", "-2.718" ...).<br />
One common techinque to solve these kind of problems is to write a <a href="https://en.wikipedia.org/wiki/Finite-state_machine" target="_blank">state machine</a>.<br />
<br />
In our case, the state machine is:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Y35EJw8FjU8/Xo782ZCipjI/AAAAAAAAkv8/WlxFKsou-LUqy0wMzmRS1aH-ldH29qN5gCLcBGAsYHQ/s1600/state-machine.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="221" data-original-width="816" height="86" src="https://1.bp.blogspot.com/-Y35EJw8FjU8/Xo782ZCipjI/AAAAAAAAkv8/WlxFKsou-LUqy0wMzmRS1aH-ldH29qN5gCLcBGAsYHQ/s320/state-machine.png" width="320" /></a></div>
<br />
Instead of writng the state machine a one big function with a lot of state and a bunch of "if" statements, we're going to write states as functions.<br />
<br />
<script src="https://gist-it.appspot.com//github/tebeka/pythonwise/blob/master/state-machine/float.go?slice=9:12"></script>
<br />
<div>
To simplify things, the end state is going to be <span style="font-family: "courier new" , "courier" , monospace;">nil</span> for easy comparison.</div>
<div>
Then our main function will look like:</div>
<div>
<script src="https://gist-it.appspot.com//github/tebeka/pythonwise/blob/master/state-machine/float.go?slice=76:94"></script>
<div>
<br /></div>
</div>
<div>
An here's how startState looks like</div>
<div>
<br /></div>
<div>
<script src="https://gist-it.appspot.com//github/tebeka/pythonwise/blob/master/state-machine/float.go?slice=13:27"></script>
<br /></div>
<div>
When you write this style of code, states are small and contained function. Easier to test and reason about.</div>
<div>
<br /></div>
<div>
Final comments:</div>
<div>
<ul>
<li>You can see the full code <a href="https://github.com/tebeka/pythonwise/tree/master/state-machine" target="_blank">here</a></li>
<li>This blog was inspired by the <a href="https://leetcode.com/problems/valid-number/" target="_blank">Valid Number</a> problem on leetcode</li>
<li>Rob Pike has an excellent "Lexcical Scanning in Go" talk that demonstrates this methon</li>
<ul>
<li><a href="https://www.youtube.com/watch?v=HxaD_trXwRE" target="_blank">Video</a></li>
<li><a href="https://talks.golang.org/2011/lex.slide" target="_blank">Slides</a></li>
</ul>
</ul>
</div>
<div>
<br /></div>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-10192400048703982952020-04-01T16:22:00.003+03:002020-04-01T16:23:31.081+03:00A Twitter Quote Bot on GAEI like quotes and have an extensive list stored in a text file. I was toying with the idea of making a twitter bot that will post a quote per day. Reading that Google cloud is now offering a <a href="https://cloud.google.com/blog/products/identity-security/introducing-google-clouds-secret-manager" target="_blank">Secret Manager</a> rekindled this idea.<br />
<br />
The result is <a href="https://twitter.com/quotamiki" target="_blank">@quotamiki</a> (which you should follow ;) It posts one quote per day (9am Asia/Jerusalem). It even has a simple <a href="https://quotamiki.appspot.com/" target="_blank">web site</a>.<br />
<br />
The code was simple to write, I like how<a href="https://cloud.google.com/appengine/docs/go" target="_blank"> Google App Engine</a> let you focus on writing code and not futz with operations.<br />
<br />
You can view the code at <a href="https://gitlab.com/tebeka/quotamiki">https://gitlab.com/tebeka/quotamiki</a>Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-78762394327848730472020-03-05T22:19:00.003+02:002020-03-05T22:21:32.053+02:00Using __getattr__ for nicer configuration APITypically, you'll read configuration from files (such as YAML) and get them as a dictionary. However in Python you'd like to write <span style="font-family: Courier New, Courier, monospace;">config.httpd.port</span> and not <span style="font-family: Courier New, Courier, monospace;">config['httpd']['port']</span><br />
<br />
<a href="https://docs.python.org/3/reference/datamodel.html#object.__getattr__" target="_blank">__getattr__</a> is a hook method that's called by Python when regular attribute lookup fails (not to be confused with the lower level <a href="https://docs.python.org/3/reference/datamodel.html#object.__getattribute__" target="_blank">__getattribute__</a>, which is much harder to work with). You can use it to wrap the configuration dictionary. Here's a small example.<br />
<br />
<br />
<script src="https://gist-it.appspot.com//github/tebeka/pythonwise/blob/master/config_wrapper.py"></script>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com4tag:blogger.com,1999:blog-27854051.post-63740573466669242502020-02-04T15:02:00.002+02:002020-02-04T15:03:07.196+02:00Adding a timer to Go's "present" toolGo has a nice <a href="https://godoc.org/golang.org/x/tools/present">present</a> tool which let's you present code <i>and</i> run it. I wanted to use it in my lightning talk at <a href="https://www.gophercon.org.il/">GopherCon Israel</a>. The talk is a quiz and I wanted a timer to show on each quiz slide.<br />
<br />
present syntax supports <span style="font-family: "courier new" , "courier" , monospace;">.html</span> directive which lets you add any HTML code to the slide. From there it easy to add a timer. I wrote <span style="font-family: "courier new" , "courier" , monospace;">timer.html</span> below and added <span style="font-family: "courier new" , "courier" , monospace;">.html timer.html</span> to the first slide.<br />
<br />
Here's how it looks (bottom left):<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-qrLcSRgDpKE/XjlqvydUTuI/AAAAAAAAjiI/WPLP8QBsHoIAeG7k4gX1G4_94-uPw3UbQCLcBGAsYHQ/s1600/present-timer.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="616" data-original-width="975" height="202" src="https://1.bp.blogspot.com/-qrLcSRgDpKE/XjlqvydUTuI/AAAAAAAAjiI/WPLP8QBsHoIAeG7k4gX1G4_94-uPw3UbQCLcBGAsYHQ/s320/present-timer.png" width="320" /></a></div>
<br />
Here's the code:<br />
<script src="https://gist-it.appspot.com//github/tebeka/pythonwise/blob/master/present-timer/timer.html"></script>Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com1tag:blogger.com,1999:blog-27854051.post-39899401011779785182020-01-05T22:51:00.001+02:002020-01-05T22:51:22.746+02:00Nested structs for HTTP CallsSome API's return complex structures. Here's for an example from <a href="https://api.stocktwits.com/developers/docs">Stocktwits API</a>.<br />
<script src="https://gist-it.appspot.com//github/tebeka/pythonwise/blob/master/stocktwits.json"></script>
Instead of creating a specific type for every element in the structre, we use use a anonymous structure and define only the fields we're intrested in. In our example we'd like to see the symbols most mentioned for a given symbol (in our case AAPL).
<script src="https://gist-it.appspot.com//github/tebeka/pythonwise/blob/master/stocktwits.go"></script>
And the output:
<code><pre>
$ go run stocktwits.go
SPY 10
TSLA 7
AMZN 5
BTC.X 5
TVIX 2
</pre></code>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-1632774749930658222019-08-12T18:52:00.002+03:002019-08-12T18:52:59.334+03:00viaenvThese days, I write a lot of Go and a lot of Python (and some other languages in the mix). What's great is that I'm getting ideas from both ecosystems and find myself more productive.<br />
<br />
In the Go world, I like Kelsey Hightower's <a href="https://github.com/kelseyhightower/envconfig">envconfig</a> for a simple application configuration from environment variables. I couldn't find something similar in the Python world, so I wrote <a href="https://github.com/tebeka/viaenv">viaenv</a> which uses <a href="https://www.python.org/dev/peps/pep-0526/">variable annotation</a>.<br />
<br />
One design decision I took is not to invent my own serialization format when getting lists & dicts from the environment (where everything is a string). I'm using JSON, there's really no need to invent yet another serialization format (<a href="https://github.com/kelseyhightower/envconfig/commit/f611eb38b3875cc3bd991ca91c51d06446afa14c#diff-04c6e90faac2675aa89e2176d2eec7d8">as envconfig does</a>).<br />
<br />
The code itself is not that long (less than 150 lines of code). I'm showing it below:<br />
<br />
<hr />
<script src="https://gist-it.appspot.com/http://github.com/tebeka/viaenv/blob/master/viaenv.py"></script>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com6tag:blogger.com,1999:blog-27854051.post-28071701940770411562019-03-20T11:50:00.000+02:002019-03-20T11:50:41.526+02:00Speed: Default value vs checking for NonePython's dict has a <a href="https://docs.python.org/3/library/stdtypes.html#dict.get">get method</a>. It'll either return an existing value for a given key or return a default value if the key is not in the dict. It's very tempting to write code like <span style="background-color: #f3f3f3; font-family: "courier new" , "courier" , monospace;">val = d.get(key, Object())</span>, however you need to think about the performance implications. Since function arguments are evaluated before calling the function, this means the a new <span style="background-color: #f3f3f3; font-family: "courier new" , "courier" , monospace;">Object</span> will be created regardless if it's in the dict or not. Let's see this affects performance.<br />
<br />
<script src="https://gist-it.appspot.com//github/tebeka/pythonwise/blob/master/default_vs_none.py"></script><span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace;">
get_default</span> will create new <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace;">Point</span></span> every time and <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace;">get_none</span> will create only if there's no such object, it works since <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace;">or</span> evaluate it's arguments lazily and will stop once the first one is <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace;">True</span>.<br />
<br />
First we'll try with a missing key:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">In [1]: %run default_vs_none.py </span><br />
<span style="font-family: Courier New, Courier, monospace;">In [2]: locations = {} # name -> Location </span><br />
<span style="font-family: Courier New, Courier, monospace;">In [3]: %timeit get_default(locations, 'carmen')</span><br />
<span style="font-family: Courier New, Courier, monospace;">384 ns ± 2.56 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)</span><br />
<span style="font-family: Courier New, Courier, monospace;">In [4]: %timeit get_none(locations, 'carmen')</span><br />
<span style="font-family: Courier New, Courier, monospace;">394 ns ± 1.61 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)</span><br />
<div>
<br /></div>
<div>
Not so much difference. However if the key exists:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">In [5]: locations['carmen'] = Location(7, 3)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">In [6]: %timeit get_default(locations, 'carmen') </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">377 ns ± 1.84 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">In [7]: %timeit get_none(locations, 'carmen')</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">135 ns ± 0.108 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)</span></div>
</div>
<div>
<br /></div>
<div>
We get much faster results.</div>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com3tag:blogger.com,1999:blog-27854051.post-29704461149890170232019-03-04T20:52:00.000+02:002019-03-04T20:58:03.398+02:00CPU Affinity in GoGo's concurrency unit is a goroutine, the Go runtime multiplexes goroutines to operating system (OS) threads. At an upper level, the OS maps threads to CPUs (or cores). To see this goroutine/thread migration, you'll need to use some C code (note that this is Linux specific).<br />
<br />
<br />
<script src="https://gist-it.appspot.com/github/tebeka/pythonwise/blob/master/affinity_1.go"></script>
If you run this code and sort the output, you'll see the workers moving between cores:<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ go run affinity.go | sort -u</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 0, CPU: 0</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 0, CPU: 1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 0, CPU: 2</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 0, CPU: 3</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 1, CPU: 0</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 1, CPU: 1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 1, CPU: 2</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 1, CPU: 3</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 2, CPU: 0</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 2, CPU: 1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 2, CPU: 2</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 2, CPU: 3</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 3, CPU: 0</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 3, CPU: 1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 3, CPU: 2</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 3, CPU: 3</span><br />
<div>
<br /></div>
<div>
There is a cost for moving a thread from one core to another (see more <a href="https://en.wikipedia.org/wiki/Processor_affinity#Usage">here</a>). In some cases this cost might be unacceptable and you'd like to pin a goroutine to a specific CPU.</div>
<div>
<br /></div>
<div>
Go has <a href="https://golang.org/pkg/runtime/#LockOSThread">runtime.LockOSThread</a> which pins the current goroutine to the current thread it's running on. For the rest of the way - pinning the thread to a CPU, you'll need to use C again.</div>
<div>
<br /></div>
<script src="https://gist-it.appspot.com/github/tebeka/pythonwise/blob/master/affinity.go"></script>
Now if you run our code with the LOCK environment variable set, you can see each worker is pinned to a specific core.<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ LOCK=1 go run affinity.go | sort -u</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 0, CPU: 0</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 1, CPU: 1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 2, CPU: 2</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">worker: 3, CPU: 3</span><br />
<div>
<br /></div>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-26656882099478274792019-01-17T07:19:00.001+02:002019-01-17T07:19:08.436+02:00Place a string in the middle of the screen - in bashFor ages my shell is greeting me with "<span style="font-family: "courier new" , "courier" , monospace;">Let's boogie, Mr Swine</span>". I had it printed in fixed offset but following a <a href="https://groups.google.com/d/topic/comp.lang.python/LTK_zgw41tw/discussion">discussion in comp.lang.python</a> I decided to get the screen size and calculate the exact location.<br />
<br />
I'm using tput to get the size and the fact that printf can get the width of the string as a parameter if you specify a * there (BTW - <a href="https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting">Python supports that as well</a>)<br />
<br />
Here's the code.<br />
<br />
<script src="https://gist-it.appspot.com/https://github.com/tebeka/pythonwise/blob/master/greet.sh"></script>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-73961504466732913282019-01-13T17:12:00.001+02:002019-01-13T17:12:57.086+02:00353solutions - 2018 in Review<h1 id="solutions---2018-in-review">
353Solutions - 2018 in Review</h1>
Happy new year! Here’s a summary of 353solution’s 2018.<br />
<h2 id="numbers">
Numbers</h2>
<ul>
<li>Total of 242 calendar days where worked for a customer (including partial days)
<ul>
<li>Down 11 days from 2017</li>
<li>Out of 261 work days in 2018</li>
</ul>
</li>
<li>Median consulting work day is 3:36h
<ul>
<li>Workshops are usually a full day</li>
</ul>
</li>
<li>Normalized work days (total divided by 8) is 203.3
<ul>
<li>Up from 171.6 normalized days in 2017</li>
</ul>
</li>
<li>Of these 53.5 days were in workshops and the rest in consulting
<ul>
<li>Up from 30 in 2017</li>
</ul>
</li>
<li>13 workshops
<ul>
<li>Up from 10 last year</li>
<li>4 more video courses on LinkedIn Learning/Lynda (over 300K viewers already)</li>
<li>Teaching in Israel, UK and the US and Germany</li>
</ul>
</li>
<li>Several new client including MGT, Gett, Actiview and others</li>
<li>Revenue up by 20%
<ul>
<li>40% from workshops</li>
</ul>
</li>
</ul>
<h2 id="insights">
Insights</h2>
<ul>
<li>Personal social network keep bringing all the work</li>
<li>Go is exploding
<ul>
<li>GopherCon Israel brought many connections</li>
<li>Much more Go in consulting</li>
</ul>
</li>
<li>Python & Data Science in demand for workshops
<ul>
<li>As well as Go</li>
</ul>
</li>
<li>We need to get better on marketing open classes
<ul>
<li>Might have an alliance for that</li>
</ul>
</li>
</ul>
<h1 id="last-years-goals">
Last Year’s Goals</h1>
<ul>
<li>Work less while increasing revenue
<ul>
<li>Mixed results here. Worked more but revenue is up ☺</li>
</ul>
</li>
<li>Publish my book
<ul>
<li>Done: https://forging-python.com</li>
</ul>
</li>
<li>More workshops and less consulting
<ul>
<li>Done</li>
</ul>
</li>
<li>Two open enrollment workshops
<ul>
<li>Nope ☹</li>
</ul>
</li>
<li>Two free 1/2 day workshops
<ul>
<li>Nope ☹</li>
</ul>
</li>
<li>Keep working from home
<ul>
<li>Done</li>
</ul>
</li>
<li>Attend at least 3 conferences
<ul>
<li>Only PyCon Israel</li>
</ul>
</li>
<li>Give at least 4 talks in meetups or conferences
<ul>
<li>Done (2 PyWeb IL, 1 Go Israel, 1 PyCon Israel, 1 Big Data)</li>
</ul>
</li>
<li>Get better at marketing
<ul>
<li>Decided marketing is the open source work I’m doing.</li>
</ul>
</li>
</ul>
<h2 id="goals-for-2019">
Goals for 2019</h2>
<ul>
<li>GopherCon Israel (February 11, 2019)</li>
<li>Work less while increasing revenue</li>
<li>Attend at least 3 conferences</li>
<li>Give at least 4 talks in meetups or conferences</li>
<li>Two free 1/2 day workshops</li>
<li>Start another book</li>
</ul>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-34995747383666140582018-11-13T11:09:00.002+02:002018-11-13T11:09:22.243+02:00direnvI use the command line a lot. Some projects require different settings, say Python virtual environment, GOPATH for installing go packages and more.<br />
<br />
I'm using <a href="https://direnv.net/">direnv</a> to help with settings per project in the terminal. For every project I have a <span style="font-family: Courier New, Courier, monospace;">.envrc</span> file which specifies required settings, this file is automatically loaded once I change directory to the project directory or any of it's sub directories.<br />
<br />
You'll need the following in your <span style="font-family: Courier New, Courier, monospace;">.zshrc</span><br />
<br />
<span style="font-family: Courier New, Courier, monospace;">if whence direnv > /dev/null; then</span><br />
<span style="font-family: Courier New, Courier, monospace;"> eval "$(direnv hook zsh)"</span><br />
<span style="font-family: Courier New, Courier, monospace;">fi</span><br />
<div>
<br /></div>
<div>
Every time you create or change your <span style="font-family: Courier New, Courier, monospace;">.envrc</span>, you'll need to run <span style="font-family: Courier New, Courier, monospace;">direnv allow</span> to validate it and make sure it's loaded. (If you did some changes and want to check them, run "<span style="font-family: Courier New, Courier, monospace;">cd ."</span><span style="font-family: inherit;">)</span></div>
<div>
<br /></div>
<div>
Here are some <span style="font-family: Courier New, Courier, monospace;">.envrc</span> examples for various scenarios:</div>
<h4>
Python + <a href="https://pipenv.readthedocs.io/en/latest/">pipenv</a></h4>
<div>
<div>
source $(pipenv --venv)/bin/activate</div>
</div>
<h4>
Go</h4>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">GOPATH=$(pwd | sed s"#/src/.*##")</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">PATH=${GOPATH}/bin:${PATH}</span></div>
</div>
<div>
<br /></div>
<div>
This assumes your project's path that looks like <span style="font-family: Courier New, Courier, monospace;">/path/to/project/src/github.com/project</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: inherit;">If you're using the new <a href="https://github.com/golang/go/wiki/Modules">go modules</a> (in 1.11+), you probably don't need this.</span></div>
<h4>
Python + virtualenv</h4>
<div>
<span style="font-family: Courier New, Courier, monospace;">source venv/bin/activate</span></div>
<h4>
<span style="font-family: inherit;">Python + <a href="https://conda.io/docs/">conda</a></span></h4>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">source activate env-name</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Replace </span><span style="font-family: Courier New, Courier, monospace;">env-name</span><span style="font-family: inherit;"> with the name of your conda environment.</span></div>
</div>
<div>
<br /></div>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-63278398343170733042018-11-07T09:28:00.000+02:002018-11-07T09:28:53.482+02:00Go, protobuf & JSONSometimes you'd like more than one way to serve an API. In my case I'm currently working on serving both <a href="https://grpc.io/">gRPC</a> and HTTP. I'd like to have one place where objects are defined and have a nice way to serialize both from <a href="https://developers.google.com/protocol-buffers/">protobuf</a> (which is the serialization gRPC uses) and JSON .<br />
<br />
When producing Go code, protobuf adds JSON struct tags. However since JSON comes from dynamic languages, fields can have any type. In Go we can use <span style="font-family: "courier new" , "courier" , monospace;">map[string]interface{}</span> but in protobuf this is a bit more complicated and we need to use <a href="https://developers.google.com/protocol-buffers/docs/proto#using-oneof">oneof</a>. The struct generated by <span style="font-family: "courier new" , "courier" , monospace;">oneof</span> does not look like regular JSON and will make users of the API write complicated JSON structures.<br />
<br />
What's nice about Go, is that we can have any type implement <a href="https://golang.org/pkg/encoding/json/#Marshaler">json.Marshaler</a> and <a href="https://golang.org/pkg/encoding/json/#Unmarshaler">json.Unmarshaler</a>. What's extra nice is that in Go, you can add these methods to the generated structs in another file (in Python, we'd have to change the generated source code since methods need to be inside the class definition).<br />
<br />
Let's have a look at a simple Job definition<br />
<br />
<br />
<script src="https://gist-it.appspot.com/https://github.com/tebeka/pythonwise/blob/master/go-protobuf-json/job.proto"></script>
And now we can add some helper methods to aid with JSON serialization (protoc generates code to pb directory)<br />
<br />
<br />
<script src="https://gist-it.appspot.com/https://github.com/tebeka/pythonwise/blob/master/go-protobuf-json/pb/methods.go"></script>
As a bonus, we added <a href="https://github.com/tebeka/pythonwise/blob/master/go-protobuf-json/pb/methods.go#L12">job.Properties</a> that returns a "native" <span style="font-family: "courier new" , "courier" , monospace;">map[string]interface{}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: inherit;">Let's look at a simple example on how we can use it</span><br />
<script src="https://gist-it.appspot.com/https://github.com/tebeka/pythonwise/blob/master/go-protobuf-json/job.go"></script>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">And its output:</span><br />
<span style="font-family: Courier New, Courier, monospace;">$ go run job.go</span><br />
<span style="font-family: Courier New, Courier, monospace;">[j1] user:"Saitama" count:1 properties:<key: int:3="" retries="" value:=""> > properties:<key: etal="" knight="" str:="" target="" value:=""> > </key:></key:></span><br />
<span style="font-family: Courier New, Courier, monospace;">[json] {"user":"Saitama","count":1,"properties":{"retries":3,"target":"Metal Knight"}}</span><br />
<span style="font-family: Courier New, Courier, monospace;"></span><br />
<span style="font-family: Courier New, Courier, monospace;">[j2] user:"Saitama" count:1 properties:<key: int:3="" retries="" value:=""> > properties:<key: etal="" knight="" str:="" target="" value:=""> > </key:></key:></span><br />
<div>
<br /></div>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-2145615823406887372018-07-26T12:18:00.000+03:002018-07-26T12:18:49.234+03:00Specifying test cases for pytest using TOMLSay you have a function that converts text and you'd like to test it. You can write a directory with input and output and use<a href="https://docs.pytest.org/en/latest/parametrize.html"> pytest.parameterize</a> to iterate over the cases. The problem is that the input and the output are in different files and it's not obvious to see them next to each other.<br />
<br />
If the text for testing is not that long, you can place all the cases in a configuration file. In this example I'll be using <a href="https://github.com/toml-lang/toml">TOML</a> format to hold the cases and each case will be in a table in <a href="https://docs.pytest.org/en/latest/parametrize.html">array of tables</a>. You can probably do the same with <a href="http://yaml.org/spec/1.0/#id2561559">multi document YAML</a>.<br />
<br />
Here's the are the test cases<br />
<script src="https://gist-it.appspot.com/https://github.com/tebeka/pythonwise/blob/master/pytest-cmp/mask_cases.toml"></script>
And here's the testing code (mask removes passwords from the text)<br />
<script src="https://gist-it.appspot.com/https://github.com/tebeka/pythonwise/blob/master/pytest-cmp/test_mask.py"></script>
<br />
When running pytest, you'll see the following:
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ python -m pytest -v</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">========================================= test session starts =========================================</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0 -- /home/miki/.local/share/virtualenvs/pytest-cmp-F3l45TQF/bin/python</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">cachedir: .pytest_cache</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">rootdir: /home/miki/Projects/pythonwise/pytest-cmp, inifile:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">collected 3 items </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">test_mask.py::test_mask[passwd] PASSED [ 33%]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">test_mask.py::test_mask[password] PASSED [ 66%]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">test_mask.py::test_mask[no change] PASSED [100%]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">====================================== 3 passed in 0.01 seconds =======================================</span><br />
<div>
<br /></div>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-37466635118901097892018-06-09T07:29:00.000+03:002018-06-09T07:29:09.998+03:00pexify - Package Python scripts using PEXSometimes you'd like to publish a simple Python script, but it depends on some external packages (e.g. <a href="http://docs.python-requests.org/en/master/">requests</a>). The usual solution is to create a package, which is a great solution but requires some work and might not be suitable for internal packages.<br />
<br />
Another solution is to use <a href="https://pex.readthedocs.io/">PEX</a>, which creates an executable virtual environment. The user running the script just needs a Python interpreter from the same version installed on their machine (which is usually the case).<br />
<br />
Converting a script to PEX requires some work, to make this simpler I wrote <span style="font-family: "courier new" , "courier" , monospace;">pexify</span>. which automates the process of creating a PEX from a single Python script.<br />
<br />
<script src="https://gist-it.appspot.com/https://github.com/tebeka/pythonwise/blob/master/pexify.py"></script>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-71547710066534858052018-04-18T15:24:00.000+03:002018-04-18T15:25:22.236+03:00Creating a Book with pandocOne of my clients asked me for a printable book to accompany one of my workshops. My teaching style is more fluid and we write a lot of code in class. I wanted a quick way to take all the source code files and some images and make a book out of them.<br />
<br />
Since I already work with markdown, I looked for a solution that can convert markdown to PDF. The winner was <a href="https://pandoc.org/">Pandoc</a> (which can be installed with conda). Of course the out-of-the-box results weren't satisfactory and some tweaking was required. Mostly to make images appear where they are define in the markdown and not where pandoc/LaTex thinks is the optimal location.<br />
<br />
The solution is composed of an <a href="https://github.com/tebeka/pythonwise/blob/master/book/flatten.awk">awk script</a> to add an include directive to markdown, a <a href="https://github.com/tebeka/pythonwise/blob/master/book/header.tex">custom LaTex header</a> to inline images and a <a href="https://github.com/tebeka/pythonwise/blob/master/book/Makefile">Makefile</a> to bind them all.<br />
<br />
You can view the whole project <a href="https://github.com/tebeka/pythonwise/tree/master/book">here</a>, including a <a href="https://github.com/tebeka/pythonwise/blob/master/book/book.md">example book</a>. You can view the output <a href="https://github.com/tebeka/pythonwise/raw/master/book/book.pdf">here</a> (decorators anyone?).Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-63310978052102923032018-04-10T13:38:00.001+03:002018-04-10T13:38:32.449+03:00Installing Arch Linux on Laptop with UEFIAfter some time with <a href="https://xubuntu.org/">Xubuntu</a> I decided to get back to <a href="https://www.archlinux.org/">Arch Linux</a>.<br />
<br />
Arch have a command line based installer, the <a href="https://wiki.archlinux.org/index.php/Installation_guide">installation instructions</a> are pretty good but for a laptop with UEFI I had to do some extra steps. Here's what I came up with, hope you'll find it useful as well.<br />
<br />
Most of my files are backed on "the cloud", my home directory with all the RC files is on a private git repository. Getting up and running after the initial install was pretty easy.<br />
<br />
<script src="https://gist.github.com/tebeka/8106b5aef331f32cce84677c5f47f538.js"></script>Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-34734450218084371062018-02-27T10:20:00.000+02:002018-02-28T12:17:07.237+02:00Python's iter & functools.partialPython's built-in <a href="https://docs.python.org/3/library/functions.html#iter">iter</a> function is mostly used to extract an iterator from an iterable (if you're confused by this, see <a href="https://twitter.com/raymondh">Raymond's</a> <a href="https://stackoverflow.com/a/9884501/7650">excellent answer</a> on StackOverflow).<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">iter</span> has a second form where it takes a function with not arguments and a sentinel value. This doesn't see that useful but check out the readline example in the documentation. <span style="font-family: Courier New, Courier, monospace;">iter</span> will call the function repeatably until it the function return the sentinel value and then will stop.<br />
<br />
What happens when the function producing the values require some arguments? I this case we can use <a href="https://docs.python.org/3/library/functools.html#functools.partial">functools.partial</a> to create a zero arguments function.<br /><br />Here's an example of a simple HTTP client using raw sockets, we can directly use join on the iterator.<br />
<br />
<script src="https://gist-it.appspot.com/https://github.com/tebeka/pythonwise/blob/master/httpc.py"></script>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com2tag:blogger.com,1999:blog-27854051.post-29586479211004084942018-02-11T09:29:00.001+02:002018-02-11T09:29:21.216+02:00353Solutions - 2017 in ReviewA little late, but here's a summary of 2017.<br />
<h2>
Numbers</h2>
<div>
<ul>
<li>Total of 255 calendar days where worked for a customer (including partial days)</li>
<ul>
<li>Up 62 days from 2016</li>
<li>Out of 260 work days in 2016</li>
</ul>
<li>Median work day is 6:53h</li>
<li>Normalized work days (total divided by 8) is 171.6</li>
<ul>
<li>Up from 158.1 normalized days in 2016</li>
</ul>
<li>Of these 30 days were in workshops and the rest in consulting</li>
<li>10 workshops</li>
<ul>
<li>Down from 18 last year</li>
<li>First <a href="https://lynda.com/Miki-Tebeka/7418954-1.html">video course on Lynda</a> (over 100K viewers already)</li>
<li>Teaching in Israel, UK, US and Poland</li>
</ul>
<li>Several new client including PayPal, CommonSense Robotics, Iguazio and others</li>
<li>Revenue down by 5%</li>
<ul>
<li>But earnings are up :)</li>
</ul>
<li><a href="https://mailchi.mp/2aff0ed15155/353solution-newsletter-january-2018">First newsletter</a> went out</li>
<ul>
<li>Why not <a href="http://eepurl.com/c0u9ob">subscribe to it</a>?</li>
</ul>
</ul>
<h2>
Insights</h2>
</div>
<div>
<ul>
<li>Personal social network keep bringing all the work</li>
<li>Very big workshops customer cut down a lot - picked up the difference with more consulting with is less lucrative</li>
<li>Python & Data Science in demand for workshops</li>
<li>Much more Go in consulting</li>
<li>Free 1/2 day open class did not bring any work</li>
<ul>
<li>However will do some more - it was fun</li>
</ul>
<li>We need to get better on marketing open classes</li>
</ul>
<h2>
Last Year's Goals</h2>
</div>
<div>
<ul>
<li>Work less while keeping same revenue</li>
<ul>
<li>Failed here. Worked more to keep about the same revenue</li>
</ul>
<li>Work more from home</li>
<ul>
<li>Success here</li>
</ul>
<li>Publish my book</li>
<ul>
<li>I can't believe this is not done yet.</li>
</ul>
<li>Publish a video course</li>
<ul>
<li>Done (with two more to come)</li>
</ul>
<li>More open enrollment classes</li>
<ul>
<li>Tried that but not enough enrollment, need to get better at marketing</li>
</ul>
</ul>
<h2>
Goals for 2018</h2>
</div>
<div>
<ul>
<li>Work less while increasing revenue</li>
<li>Publish my book</li>
<ul>
<li>Learned from last year and reserved serveral days this quarter to finish it</li>
</ul>
<li>More workshops and less consulting</li>
<ul>
<li>Two open enrollment workshops</li>
<li>Two free 1/2 workshops</li>
</ul>
<li>Keep working from home</li>
<ul>
<li>Attend at least 3 conferences</li>
</ul>
<li>Give at least 4 talks in meetups or conferences</li>
<ul>
<li>First talk was Jan 1 on <a href="https://www.meetup.com/PyWeb-IL/">PyWeb-IL</a>, have another one slated for March</li>
</ul>
<li>Get better at marketing</li>
</ul>
</div>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-27970632379996952692017-12-08T19:28:00.000+02:002017-12-08T19:37:47.628+02:00Advent of Code 2017 #8 and Python's evalI'm having fun <a href="https://github.com/tebeka/adventofcode">solving</a> <a href="http://adventofcode.com/2017">Advent of Code 2017</a>. Problem 8 reminded the power of Python's <a href="https://docs.python.org/3/library/functions.html#eval">eval</a> (and before you start commenting the "eval is evil" may I remind you of <a href="https://xkcd.com/327/">this</a> :)<br />
<br />
<script src="https://gist-it.appspot.com/https://github.com/tebeka/adventofcode/blob/master/day8.py"></script>
You can check <a href="https://github.com/tebeka/adventofcode/blob/master/day8.go">The Go implementation</a> that don't have eval and need to work harder.Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-46173209918631013442017-11-24T15:59:00.002+02:002017-11-24T15:59:53.770+02:00Python → Go Cheat Sheet<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-yDzUANi6k9I/WhglhAKxsbI/AAAAAAAARNg/0FCDbvGLRdseOX4gHVjF1kD69-z8q-iTgCLcBGAs/s1600/py2go-screenshot.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="670" data-original-width="1147" height="233" src="https://2.bp.blogspot.com/-yDzUANi6k9I/WhglhAKxsbI/AAAAAAAARNg/0FCDbvGLRdseOX4gHVjF1kD69-z8q-iTgCLcBGAs/s400/py2go-screenshot.png" width="400" /></a></div>
<br />
I'm teaching and consulting in <a href="https://golang.org/">Go</a> a lot lately, and I work a lot with Python as well. There's a big trend of rewriting backend services in Go and to help people coming from the Python world I've created a <a href="https://www.353.solutions/py2go/index.html">Go →Python cheatsheet</a>.<br />
<br />
The code in <a href="https://github.com/tebeka/py2go-cheatsheet">here</a>, I'd love to hear if you have suggestion (via a PR ;)Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-13335273435614740512017-09-24T21:21:00.001+03:002017-09-24T21:21:08.766+03:00Finding Where Most Activity Time Was Spent<a href="https://nbviewer.jupyter.org/github/tebeka/pythonwise/blob/master/Most-Time-Spent.ipynb" target="_blank">View Notebook</a>
<iframe onload="this.width=screen.width;this.height=screen.height;" src="https://nbviewer.jupyter.org/github/tebeka/pythonwise/blob/master/Most-Time-Spent.ipynb">
</iframe>Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0tag:blogger.com,1999:blog-27854051.post-88002446682373994892017-09-14T20:37:00.000+03:002017-09-14T20:37:56.192+03:00Checking for Zero Values in GoIn <a href="https://golang.org/">Go</a>, every type has a zero value. Which is the value a variable of this type get if it's not initialized. I had a configuration object of type <span style="font-family: Courier New, Courier, monospace;">map[string]interface{}</span> and I needed to check if value exists and is not a zero value.<br />
<br />
Here's a small piece of code that checks for zero values:<br />
<br />
<script src="https://gist-it.appspot.com/https://github.com/tebeka/pythonwise/blob/master/is_zero.go"></script>
Miki Tebekahttp://www.blogger.com/profile/03296782049507092947noreply@blogger.com0