PythonWise

Friday, June 11, 2010

import_any

Import any file as a Python module. This is a nice big security risk, so beware ...

from types import ModuleType

def import_any(path, name=None):
    module = ModuleType(name or "dummy")
    execfile(path, globals(), module.__dict__)

    return module

Wednesday, June 09, 2010

Thursday, June 03, 2010

Tagging last good build in git

git (and mercurial) has a nice feature that you can "move" tags (using -f). At Sauce Labs we use that to have a track of our last good build and the last deployed revision.

In our continuous integration system, we mark the last good build in the last step which is executed only if the test suite passed.

tag-ok-build.sh

#!/bin/bash

# Error on 1'st failure
set -e

tag=last-green-build
revision=$(git rev-parse HEAD)

echo "Tagging $commit as $tag"
git tag -f $tag $commit
git pull origin master
git push --tags origin master

To deploy, we use the following script who also tags the last deployed version.

deploy.sh

#!/bin/bash
# Deploy, meaning sync from last successful build

tag=last-green-build

# Fetch get the latest changes from remote but does not update working
# directory, just the index
git fetch --tags

# Merge to the last successful bulid
git merge ${tag}

# Tag currently deployed revision
git tag -f deployed ${tag}
git push --tags

Saturday, May 15, 2010

couchnuke

A simple script to "nuke" a couchdb server, use with care!

#!/bin/bash

# Delete all databases on a couchdb server

# Exit on 1'st error
set -e

if [ $# -gt 1 ]; then
    echo "usage: $(basename $0) [DB_URL]"
    exit 1
fi

URL=${1-http://localhost:5984}

echo -n "Delete *ALL* databases on $URL? [y/n] "
read ans
if [ "$ans" != "y" ]; then
    exit
fi

for db in $(curl -s $URL/_all_dbs | tr '",[]' ' \n  ');
do
    echo "Deleting $db"
    curl -X DELETE $URL/$db
done

Monday, April 12, 2010

Run a temporary CouchDB

One nice solution to isolate tests, it to point the code to a temporary database. Below is a script to launch a new instance of CouchDB on a given port.

#!/bin/bash
# A script to run a temporary couchdb instance

if [ $# -ne 1 ]; then
    echo "usage: $(basename $0) PORT"
    exit 1
fi

port=$1

base=$(mktemp -d)
dbdir=$base/data
config=$base/config.ini
mkdir -p $dbdir

cat <<EOF > $config
[httpd]
port = $port

[couchdb]
database_dir = $dbdir
view_index_dir = $dbdir

[log]
file = ${base}/couch.log
EOF

trap "rm -fr $base" SIGINT SIGTERM

echo "couchdb directory is $base"
couchdb -a $config

Friday, April 09, 2010

Sourcing a shell script

Sometime you want to emulate the action of "source" in bash, settings some environment variables.
Here's one way to do it:

from subprocess import Popen, PIPE
from os import environ

def source(script, update=1):
    pipe = Popen(". %s; env" % script, stdout=PIPE, shell=True)
    data = pipe.communicate()[0]

    env = dict((line.split("=", 1) for line in data.splitlines()))
    if update:
        environ.update(env)

    return env

Thursday, March 25, 2010

Showing Solr Search Results Usign jQuery

Solr is a wonderful product. However it does not have a integrated search page (there's one in the admin section, but it's nothing you want to expose).

Luckily, Solr supports JSON results and even JSOP, which means we can do all the code for searching in our web page.

Note that I'm using the Solr demo that comes with the distribution, your search results might have different attributes depending on the schema.

<html>
    <head>
        <title>Solr Search</title>
    </head>
    <body>
        <h3>Solr Search</h3>

        Query: <input id="query" /> 
        <button id="search">Search</button>
        <hr />
        <div id="results">
        </div>
    </body>
    <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
    <script>
        function on_data(data) {
            $('#results').empty();

            var docs = data.response.docs;
            $.each(docs, function(i, item) {
                $('#results').prepend($('<div>' + item.name + '</div>'));
            });

            var total = 'Found ' + docs.length + ' results';
            $('#results').prepend('<div>' + total + '</div>');
        }

        function on_search() {
            var query = $('#query').val();
            if (query.length == 0) {
                return;
            }

            var url='http://localhost:8983/solr/select/?wt=json&json.wrf=?&' +
                    'q=' + query;
            $.getJSON(url, on_data);
        }

        function on_ready() {
            $('#search').click(on_search);
            /* Hook enter to search */
            $('body').keypress(function(e) {
                if (e.keyCode == '13') {
                    on_search();
                }
            });
        }

        $(document).ready(on_ready);
    </script>
</html>

To run the Solr demo do extract the distribution, then

cd example 
java -jar start.jar > /dev/null &
cd exampledocs 
./post.sh *.xml

Friday, March 12, 2010

Summarizer

Just a quick note that I've uploaded the code to "Text Summarizer" here.

Thursday, February 11, 2010

Parse HTTP Response

Steve commented (justly) that there is no Python library function to parse HTTP transportation. Here is one way to do it:

Thursday, February 04, 2010

Publish Buildbot builds to Twitter

Buildbot exposes a simple XMLRPC protocol. Using this we can write a small daemon that pools the build status and publish it somewhere (or do other interesting stuff with it). In our case we'll publish the results to twitter using its publishing API.

#!/usr/bin/env python
# Watch builds on buildbot and publish to twitter

from time import time, sleep
from xmlrpclib import ServerProxy
from urllib import urlopen

user, password = "tebeka", "SECRET_PASSWORD"
bbot_url = "http://buildbot.example.com/xmlrpc"
tweet_url = "http://%s:%s@twitter.com/statuses/update.xml" % (user, password)

def main():
    proxy = ServerProxy(bbot_url)
    last_time = time()

    while 1:
        now = time()
        builds = proxy.getAllBuildsInInterval(last_time, now)
        for build in builds:
            builder, build, status = build[0], build[1], build[5]
            status = "OK" if status == "success" else "BROKEN"
            message = "[%s] build %s is %s" % (builder, build, status)
            urlopen(tweet_url, "status=%s" % message)

        last_time = now if builds else last_time
        sleep(10)

if __name__ == "__main__":
    main()

Friday, January 15, 2010

github-url

#!/bin/bash
# Show github URL for a given file

if [ $# -ne 1 ]; then
    echo "usage: $(basename $0) FILENAME"
    exit 1
fi

if [ ! -e $1 ]; then
    echo "error: can't find $1"
    exit 1
fi

if [ -d $1 ]; then
    cd $1
    dest=.
else
    cd $(dirname $1)
    dest=$(basename $1)
fi

url=$(git config remote.origin.url | cut -d: -f2)
if [ $? -ne 0 ]; then
    echo "error: can't get remote url"
    exit 1
fi
url=${url/.git/}

path=$(git ls-files --full-name $dest | head -1)
if [ $? -ne 0 ]; then
    echo "error: git does not know about $1"
    exit 1
fi

branch=$(git branch --no-color 2>/dev/null | grep '^* ' | sed 's/* //')
prefix="https://github.com/$url"
if [ "$dest" == "." ]; then
    path=$(dirname $path)
    url="${prefix}/tree/${branch}/$path"
else
    url="${prefix}/blob/${branch}/$path"
fi

echo $url

Wednesday, December 30, 2009

Making TeamSpeak 3 work on Ubuntu

TeamSpeak 3, you can download the Linux client from their site. The only problem is that currently (beta8) the client don't play nice with pulseaudio.

EDIT: Seems like beta-20 is working well with Ubuntu 10.4. Use the ts3client_runscript.sh (make sure you change directory to the TeamSpeak directory first).

After digging in the Ubuntu forums (can't find the exact post link), I came with the following solution:

Extract the client (cd /opt && sh ~/Downloads/TeamSpeak3-Client-linux_amd64-3.0.0-beta8.run)
Create the following script

#!/bin/bash

# Make teamspeak work on Ubuntu (basically disable pulseaudio before launching)

conf=$HOME/.pulse/client.conf
backup="${conf}.bak"

if [ -f "$conf" ]; then
    mv "$conf" "$backup"
fi
echo "autospawn = no" > "$conf"
pulseaudio --kill

cleanup() {
rm "$conf"
if [ -f "$backup" ]; then
        mv "$backup" "$conf"
fi
    pulseaudio -D
}

cd /opt/TeamSpeak3-Client-linux_amd64/
./ts3client_linux_amd64&
pid=$!
trap "kill -9 $pid; cleanup" SIGINT SIGTERM
wait $pid
cleanup

Good luck! (The other option is to use mumble.)

Tuesday, December 29, 2009

Multi VCS status

Since I use mercurial, git, subversion and sometimes bzr. I use the following script (called st) to find the status of the current directory.

#!/bin/bash

# Repository status

root=$PWD
while [ "$root" != "/" ];
do
    if [ -d "$root/.hg" ]; then
        hg status 
        break
    fi
    if [ -d "$root/.git" ]; then
        git status --untracked-files=no
        break
    fi
    if [ -d "$root/.svn" ]; then
        svn status
        break
    fi
    if [ -d "$root/.bzr" ]; then
        bzr status
        break
    fi
    root=$(dirname "$root")
done

if [ "$root" == "/" ]; then
    echo "error: can't find a repository here"
    exit 1
fi

Friday, December 11, 2009

Add Subversion files to git

I you need to start a git repository from a Subversion one, and you don't care about history. Then the following should be helpful.

#!/usr/bin/env python

# Add subversion repsotory to a git one
# Run (at top of svn repository)
#   git init
#   git-add-svn
#   git ci -a -m "Initial import from svn $(svn info | grep Revision)"

__author__ = "Miki Tebeka <miki@mikitebeka.com>"

from os import walk
from os.path import join
from fnmatch import fnmatch
from itertools import ifilter
from subprocess import call
from sys import stdout
import re

is_vcs = re.compile("\.(git|svn)", re.I).search

def make_pred(exclude):
    def is_ok(name):
        return not any((fnmatch(name, ext) for ext in exclude))
    return is_ok

def all_files():
    for root, dirs, files in walk("."):
        if is_vcs(root):
            continue
        for name in files:
            yield join(root, name)

def git_add_svn(exclude):
    pred = make_pred(exclude)
    for filename in ifilter(pred, all_files()):
        print filename
        stdout.flush()
        call(["git", "add", "-f", filename])

def main(argv=None):
    import sys
    from optparse import OptionParser

    argv = argv or sys.argv

    parser = OptionParser("%prog [options]")
    parser.add_option("-e", "--exclude", dest="exclude", action="append",
            help="extension to exclude")

    opts, args = parser.parse_args(argv[1:])
    if args:
        parser.error("wrong number of arguments") # Will exit

    git_add_svn(opts.exclude or [])


if __name__ == "__main__":
    main()

Friday, December 04, 2009

Making GMail The Default Email Application On Ubuntu

Open System->Preferences->Preferred Applications
Change "Mail Reader" to "Custom"
Write bash -c 'to="%s"; gnome-open "https://mail.google.com/mail?view=cm&tf=0&to=${to#mailto:}"' in the "Command:" box

Saturday, November 07, 2009

Find Cell Number Carrier

#!/bin/bash

# Find the carrier for a cell number using whitepages

# Miki Tebeka <miki@mikitebeka.com>

if [ $# -ne 1 ]; then
    echo "usage: $(basename $0) NUMBER"
    exit 1
fi

# Fake Firefox
agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) "
agent="$agent Gecko/20061204 Firefox/2.0.0.1"

url="http://www.whitepages.com/carrier_lookup?"
url="${url}carrier=other&number_0=${1}&name_1=&number_1=&name_2=&number_2="
url="${url}&name_3=&number_3=&response=1"

curl -A "$agent" -L -s "$url" | \
    grep -A1 carrier_result | tail -1 | \
    sed -e 's/.*(\(.*\))/\1/' -e 's/^ \+//'

Tuesday, October 27, 2009

EnSalvage

I wrote a little utility to help a fried whose Entourage database got corrupted.

If anybody cares, here it is, you can even download a .app for Mac :)

Friday, October 23, 2009

binary

#!/bin/bash
# Show binary representation of a number

if [ $# -ne 1 ]; then
    echo "usage: `basename $0` NUMBER" 1>&2
    exit 1
fi

echo "obase=2; $1" | bc -l

Friday, October 09, 2009

strftime

#!/usr/bin/env python
'''Convert seconds since epoch to human readable time format.
   Useful in cases where logs dump time information in epoch.

    $ echo 1255099899 | strftime
    10/09/2009 07:51
    $
'''

from time import strftime, localtime
import re

def main(argv=None):
    import sys
    from optparse import OptionParser

    argv = argv or sys.argv

    default_format = "%m/%d/%Y %H:%M"
    parser = OptionParser("%prog [options] [FILENAME]")
    parser.add_option("--format", dest="format",
        help="time format [%s]" % default_format, default=default_format)
    parser.add_option("--doc", help="show strftime documentation",
        dest="doc", action="store_true", default=0)
    parser.add_option("--quote", help="quote result (e.g. '10/2/2009')",
        dest="quote", default="")

    opts, args = parser.parse_args(argv[1:])
    if len(args) not in (0, 1):
        parser.error("wrong number of arguments") # Will exit

    if opts.doc:
        import webbrowser
        url = "http://docs.python.org/library/time.html#time.strftime"
        webbrowser.open(url)
        raise SystemExit

    infile = args[0] if args else "-"
    if infile == "-":
        info = sys.stdin
    else:
        try:
            info = open(infile)
        except IOError, e:
            raise SystemExit("error: can't open %s - %s" % (infile, e))

    def callback(match):
        t = localtime(float(match.group()))
        human = strftime(opts.format, t)
        if opts.quote:
            human = human.replace(opts.quote, r"\\%s" % opts.quote)
            human = opts.quote + human + opts.quote

        return human

    # We assume 1XXXXXXXXX (2001 +) dates, so we won't convert *everything*
    print re.sub("1\d{9}(\.\d+)?", callback, info.read())


if __name__ == "__main__":
    main()