PythonWise

Friday, November 14, 2008

The Code You Don't Write


Act without doing, work without effort.
Think of the small as large and the few as many.
Confront the difficult while it is still easy;
accomplish the great task by a series of small steps.
    - Lao-Tze

Sometimes, the code you don't write is more important than the one you write.
Whenever I start on a new task, my first question is "how can I do this without coding?".

Here's a small (true) exmaple:

We had a problem that serving files on an NFS mounted volume was slow for the first request and them it was good. Probably the mount went "stale" after a while.

First option of learning the inner working of NFS mounts was dropped immediately - you can never know how much time this tinkering will take and if it'll work eventually.

So I decided to keep the NFS mount "warm" by periodically accessing it.

First Version:

from time import sleep
from os import listdir

while 1:
    listdir("/path/to/nfs")
    sleep(10 * 60 * 60)

and in crontab


@reboot /usr/bin/python /path/to/script.py

Then I thought - "cron", and second version came:

from os import listdir

listdir("/path/to/nfs")</code></pre>

and in crontab:


*/10 * * * * /usr/bin/python /path/to/script.py

And then last version (just crontab):


*/10 * * * * /bin/ls /path/to/nfs

So down from 6 LOC to 3 LOC to 1 LOC - that's what I call produtivity :)

Saturday, November 08, 2008

Where is Miki?

A little CGI script that show where I am (gets data from my google calendar).

Using Google Data Python API

#!/usr/bin/env python
'''Where where am I? (data from Google calendar)

Get gdata from http://code.google.com/p/gdata-python-client/
'''

__author__ = "Miki Tebeka <miki.tebeka@gmail.com>"

import gdata.calendar.service as cal_service
from time import localtime, strptime, strftime, mktime, timezone

DAY = 24 * 60 * 60

def caltime_to_local(caltime):
    # 2008-11-07T23:30:00.000+02:00
    t = mktime(strptime(caltime[:16], "%Y-%m-%dT%H:%M"))
    tz_h, tz_m = map(int, caltime[-5:].split(":"))
    cal_tz = (tz_h * 60 * 60) + (tz_m * 60)
    if caltime[-6] == "-":
        cal_tz = -cal_tz

    # See timezone documentation, the sign is reversed
    diff = -timezone - cal_tz 

    return localtime(t + diff)

def iter_meetings():
    client = cal_service.CalendarService()
    client.email = "your-google-user-name"
    client.password = "your-google-password"
    client.source = "Where-is-Miki"
    client.ProgrammaticLogin()

    query = cal_service.CalendarEventQuery("default", "private", "full")
    query.start_min = strftime("%Y-%m-%d")
    tomorrow = localtime(mktime(localtime()) + DAY)
    query.start_max = strftime("%Y-%m-%d", tomorrow)
    feed = client.CalendarQuery(query)
    for event in feed.entry:
        title = event.title.text
        when = event.when[0]
        start = caltime_to_local(when.start_time)
        end = caltime_to_local(when.end_time)

        yield title, start, end

def find_meeting(meetings, now):
    for title, start, end in meetings:
        print title, start, end
        if start <= now <= end:
            return title, end

    return None, None

def meetings_html(meetings):
    if not meetings:
        return "No meetings today"

    trs = []
    tr = "<tr><td>%s</td><td>%s</td><td>%s</td></tr>"
    for title, start, end in meetings:
        start = strftime("%H:%M", start)
        end = strftime("%H:%M", end)
        trs.append(tr % (title, start, end))

    return "Today's meetings: <table border='1'>" + \
           "<tr><th>Title</th><th>Start</th><th>End</th></tr>" + \
            "\n".join(trs) + \
            "</table>"

HTML = '''
<html>
    <head>
        <title>Where is Miki?</title>
        <style>
            body, td, th {
                font-family: Monospace;
                font-size: 22px;
            }
        </style>
    </head>
    <body>
        <h1>Where is Miki?</h1>
        <p>
            Seems that he is <b>%s</b>.
        </p>
        <p>
            %s
        </p>
    </body>
</html>
'''

if __name__ == "__main__":
    import cgitb; cgitb.enable()
    from operator import itemgetter

    days = ["Mon","Tue","Wed", "Thu", "Fri", "Sat", "Sun"]
    now = localtime()

    day = days[now.tm_wday]
    meetings = sorted(iter_meetings(), key=itemgetter(-1))

    # Yeah, yeah - I get in early
    if (now.tm_hour < 6) or (now.tm_hour > 17):
        where = "at home"
    elif day in ["Sat", "Sun"]:
        where = "at home"
    else:
        title, end = find_meeting(now, meetings)
        if end:
            where = "meeting %s (until %s)" % (title, strftime("%H:%M", end))
        else:
            where = "at work"

    print "Content-Type: text/html\n"
    print HTML % (where, meetings_html(meetings))

Wednesday, November 05, 2008

Document With Examples

I've found out that a lot of times when I have a parsing code, it's best to document the methods with examples of the input.

A small example:

#!/usr/bin/env python
'''Simple script showing how to document with examples'''

__author__ = "Miki Tebeka <miki.tebeka@gmail.com>"

import re

# HTTP/1.1 200 OK
# HTTP/1.1 301 Moved Permanently
def http_code(line):
    return line.split()[1]

if __name__ == "__main__":
    print http_code("HTTP/1.1 301 Moved Permanently")

Monday, October 13, 2008

JavaScript sound player

It's very easy to connect Adobe FLEX to JavaScript.

We'll create a simple sound player that exposes two functions: play and stop. It'll also call the JavaScript function on_play_complete when the current sound has finished playing.

soundplayer.mxml

Compile with mxmlc soundplayer.mxml

soundplayer.html

Tuesday, September 23, 2008

"Disabling" an image

Sometimes you want to mark a button image as "disabled". The usual method is to have two images and display the "disabled" state image when disabled.

However you can use the image opacity the mark is as disabled as well:


<html>
    <head>
        <title>Dimmer</title>
        <style>
            .disabled {
                filter: alpha(opacity=50);
                -moz-opacity: 0.50;
                opacity: 0.50;
            }
        </style>
    </head>
    <body>
        <center>
            <p>Show how to "dim" an image, marking it disabled</p>
            <img src="image.png" id="image" /> <br />
            <button onclick="disable();">Disable</button>
            <button onclick="enable();">Enable</button>
        </center>
    </body>
    <script src="jquery.js"></script>
    <script>
        function disable() {
            $('#image').addClass('disabled');
        }

        function enable() {
            $('#image').removeClass('disabled');
        }
    </script>
</html>

Thursday, September 18, 2008

Destktop Web Application

It's very easy have the browser window as your user interface, even on desktop applications.

Below is a small phone book demo.

General Design:

Have a web server serve the main search page

I'm using BaseHTTPServer but you can use any other framework

Run an AJAX query and display the results

I'm using jQuery

fork the server so the program will return

webphone.py (AKA the web server)

index.html (AKA the GUI)

pymodver

Which version of module do I have?

#!/usr/bin/env python
'''Find python module version'''

__author__ = "Miki Tebeka <miki.tebeka@gmail.com>"

def valueof(v):
    if callable(v):
        try:
            return v()
        except Exception:
            return None
    return v

def load_module(module_name):
    module = __import__(module_name)

    # __import__("a.b") will give us a
    if ("." in module_name):
        names = module_name.split(".")[1:]
        while names:
            name = names.pop(0)
            module = getattr(module, name)

    return module

def find_module_version(module_name):
    module = load_module(module_name)
    attrs = set(dir(module))

    for known in ("__version__", "version"):
        if known in attrs:
            v = valueof(getattr(module, known))
            if v:
                return v

    for attr in attrs:
        if "version" in attr.lower():
            v = getattr(module, attr)
            if not v:
                continue
            v = valueof(v)
            if v:
                return v

def main(argv=None):
    if argv is None:
        import sys
        argv = sys.argv

    from optparse import OptionParser

    parser = OptionParser("usage: %prog MODULE_NAME")

    opts, args = parser.parse_args(argv[1:])
    if len(args) != 1:
        parser.error("wrong number of arguments") # Will exit

    module_name = args[0]

    try:
        version = find_module_version(module_name)
    except ImportError, e:
        raise SystemExit("error: can't import %s (%s)" % (module_name, e))

    if version:
        print version
    else:
        raise SystemExit("error: can't find version for %s" % module_name)

if __name__ == "__main__":
    main()

Tuesday, September 16, 2008

Exit Gracefully

When your program is terminated by a signal, the atexit handlers are not called.

A short solution:

Sunday, September 07, 2008

"unpack" updated

Updated the code to unpack and added view functionality to it.

Thursday, September 04, 2008

putclip

A "cross platform" command line utility to place things in the clipboard.
(On linux uses xsel)

#!/usr/bin/env python
'''Place stuff in clipboard - multi platform'''

__author__ = "Miki Tebeka <miki.tebeka@gmail.com>"

from os import popen
from sys import platform

COMMDANDS = { # platform -> command
    "darwin" : "pbcopy",
    "linux2" : "xsel -i",
    "cygwin" : "/bin/putclip",
}

def putclip(text):
    command = COMMDANDS[platform]
    popen(command, "w").write(text)

def main(argv=None):
    if argv is None:
        import sys
        argv = sys.argv

    from optparse import OptionParser
    from sys import stdin

    parser = OptionParser("%prog [PATH]")

    opts, args = parser.parse_args(argv[1:])

    if len(args) not in (0, 1):
        parser.error("wrong number of arguments") # Will exit

    if platform not in COMMDANDS:
        message = "error: don't know how to handle clipboard on %s" % platform
        raise SystemExit(message)

    if (not args) or (args[0] == "-"):
        info = stdin
    else:
        try:
            infile = args[0]
            info = open(infile)
        except IOError, e:
            raise SystemExit("error: can't open %s - %s" % (infile, e))

    try:
        putclip(info.read())
    except OSError, e:
        raise SystemExit("error: %s" % e)

if __name__ == "__main__":
    main()

Friday, August 15, 2008

flatten


#!/usr/bin/env python

__author__ = "Miki Tebeka <miki.tebeka@gmail.com>"

def flatten(items):
    '''Flatten a nested list.

    >>> a = [[1], 2, [[[3]], 4]]
    >>> list(flatten(a))
    [1, 2, 3, 4]
    >>>
    '''
    for item in items:
        if getattr(item, "__iter__", None):
            for subitem in flatten(item):
                yield subitem
        else:
            yield item

if __name__ == "__main__":
    from doctest import testmod
    testmod()

pipe

Thursday, August 07, 2008

printobj


'''Quick and dirty object "repr"'''

__author__ = "Miki Tebeka "
# FIXME: Find how to make doctest play with "regular" class definition

def printobj(obj):
    '''
    Quick and dirty object "repr"

    >>> class Point: pass
    >>> p = Point()
    >>> p.x, p.y = 1, 2
    >>> printobj(p)
    ('y', 2)
    ('x', 1)
    >>>
    '''
    print "\n".join(map(str, obj.__dict__.items()))

if __name__ == "__main__":
    from doctest import testmod
    testmod()

Thursday, July 24, 2008

CGI trampoline for cross site AJAX

Most (all?) browsers won't let you do cross site AJAX calls.

One solution is JSONP (which is supported by jQuery). However not all servers support it.

The other solution is to create a "trampoline" in your site that returns the data from the remote site:

Sunday, July 20, 2008

whichscm


#!/usr/bin/env python
'''Find under which SCM directory is'''

__author__ = "Miki Tebeka <miki.tebeka@gmail.com>"

from os import sep
from os.path import join, isdir, abspath
from itertools import ifilter, imap

def updirs(path):
    parts = path.split(sep)
    if not parts[0]:
        parts[0] = sep # FIXME: Windows
    while parts:
        yield join(*parts)
        parts.pop()

def scmdirs(path, scms):
    for scmext in scms:
        yield join(path, scmext)

def scm(dirname):
    return dirname[-3:].lower()

def scms(path, scms):
    return imap(scm, ifilter(isdir, scmdirs(path, scms)))


def whichscm(path):
    path = abspath(path)

    for scm in scms(path, (".svn", "CVS")):
        return scm

    scmdirs = (".bzr", ".hg", ".git")
    for dirname in updirs(path):
        for scm in scms(dirname, (".bzr", ".hg", ".git")):
            return scm

def main(argv=None):
    if argv is None:
        import sys
        argv = sys.argv

    from optparse import OptionParser

    parser = OptionParser("usage: %prog [DIRNAME]")

    opts, args = parser.parse_args(argv[1:])
    if len(args) not in (0, 1):
        parser.error("wrong number of arguments") # Will exit

    dirname = args[0] if args else "."
    if not isdir(dirname):
        raise SystemExit("error: %s is not a directory" % dirname)

    scm = whichscm(dirname)
    if not scm:
        raise SystemExit("error: can't find scm for %s" % dirname)

    print scm

if __name__ == "__main__":
    main()

Thursday, July 17, 2008

wholistens


#!/usr/bin/env python
'''Find out who is listening on a port'''

from os import popen
from os.path import isdir
import re

is_int = re.compile("\d+").match

def find_pid(port):
    for line in popen("netstat -nlp 2>&1"):
        match = re.search(":(%s)\\s+" % port, line)
        if not match:
            continue

        pidname = line.split()[-1].strip()
        return pidname.split("/")[0]

    return None

def find_cmdline(pid):
    cmd = open("/proc/%s/cmdline" % pid, "rb").read()

    return " ".join(cmd.split(chr(0)))

def find_pwd(pid):
    data = open("/proc/%s/environ" % pid, "rb").read()
    for line in data.split(chr(0)):
        if line.startswith("PWD"):
            return line.split("=")[1]

    return None

def main(argv=None):
    if argv is None:
        import sys
        argv = sys.argv

    from optparse import OptionParser

    parser = OptionParser("usage: %prog PORT")
    opts, args = parser.parse_args(argv[1:])
    if len(args) != 1:
        parser.error("wrong number of arguments") # Will exit

    port = args[0]
    pid = find_pid(port)
    if not (pid and is_int(pid)):
        raise SystemExit(
            "error: can't find who listens on port %s"
            " [try again with sudo?] " % port)

    if not isdir("/proc/%s" % pid):
        raise SystemExit("error: can't find information on pid %s" % pid)

    pwd = find_pwd(pid) or "<unknown>"
    print "%s (pid=%s, pwd=%s)" % (find_cmdline(pid), pid, pwd)

if __name__ == "__main__":
    main()

Note: This does not work on OSX (no /proc and different netstat api)

Monday, July 14, 2008

Code in googlecode

I'll post all the code shown here in http://pythonwise.googlecode.com/.

I've uploaded most of the code from 2008 to 2006, will add the other stuff bit by bit.

Friday, July 11, 2008

Computer Load - The AJAX Way

Show computer load using jquery, flot and Python's BaseHTTPServer (all is less that 70 lines of code).


#!/usr/bin/env python
'''Server to show computer load'''

import re
from os import popen
from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
from socket import gethostname

def load():
    '''Very fancy computer load :)'''
    output = popen("uptime").read()
    match = re.search("load average(s)?:\\s+(\\d+\\.\\d+)", output)
    return float(match.groups()[1]) * 100

HTML = '''
<html>
    <head>
        <script src="jquery.js"></script>
        <script src="jquery.flot.js"></script>
        <title>%s load</title>
    </head>
    <body>
        <center>
            <h1>%s load</h1>
            <div id="chart" style="width:600px;height:400px;">
            Loading ...
            </div>
        </center>
    </body>
    <script>
    var samples = [];
    var options = {
        yaxis: {
            min: 0,
            max: 100
        },
        xaxis: {
            ticks: []
        }
    };

    function get_data() {
        $.getJSON("/data", function(data) {
            samples.push(data);
            if (samples.length > 120) {
                samples.shift();
            }

            var xy = [];
            for (var i = 0; i < samples.length; ++i) {
                xy.push([i, samples[i]]);
            }
            $.plot($('#chart'), [xy], options);
        });
    }

    $(document).ready(function() {
        setInterval(get_data, 1000);
    });
    </script>
</html>
''' % (gethostname(), gethostname())

class RequestHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/":
            self.wfile.write(HTML)
        elif self.path.endswith(".js"):
            self.wfile.write(open(".%s" % self.path).read())
        else:
            self.wfile.write("%.2f" % load())

if __name__ == "__main__":
    server = HTTPServer(("", 8888), RequestHandler)
    server.serve_forever()

Wednesday, July 09, 2008

bazaar is slow - who cares?

BS: git is faster than mercurial is faster than bazaar!
ME: Frankly dear, I don't give a damn.
BS: But speed is important!
ME: It is. However when you choose a source control system (if you have the
privilege of doing so), there are many more things to consider:

Does it fit my work model?
Is it stable?
Will it stay for long?
What's the community like?
Is development active?
...
Is it fast enough?

BS: But bazaar is the slowest
ME: For many, many projects, it's fast enough
BS: So who too choose?
ME: You do your own math. I chose `bazaar` because it has two features that
the others (to my knowledge) don't have:

It knows about directories (I like to check-in empty logs directory - it simplifies the logging code)
You can check-in files from another directory (see here)

And, it's fast enough for me (about 1sec for bzr st on ~200K LOC)

BS: OK! ... but git storage is better than mercurial is better than bazaar!
ME: <sigh> Why do I even bother? </sigh> ...

Next week - LISP is faster than Python ;)

-------
BS = blogosphere
ME = me

UPDATE (2009/01/27)
Bazaar slowness started to annoy me too much. I felt that every "bzr st" was taking way to much time (not to mention the updated). So I switched to mercurial.

The difference in timing is even noticeable in the most trivial operations:


[09:19] $time hg --version > /dev/null

real 0m0.060s
user 0m0.048s
sys 0m0.012s
[09:20] fattoc $time bzr --version > /dev/null

real 0m0.191s
user 0m0.144s
sys 0m0.048s
[09:21] $

You feel this 0.13 seconds. It seems that hg --version return immediately but bzr --version takes it's time.

Sometimes speed *does* matter.

Monday, June 23, 2008

smokejs

SmokeJS is a discovery based unittest framework for JavaScript (like nose and py.test)

You can run tests either in the command line (with SpiderMonkey or Rhino) or in the browser.

Go, check it out and fill in bugs...