Want To Be More Productive? - Improve Your I/O

It always surprise me that people trying to makes themselves more productive overlook their I/O - reading and writing.

I still see many people slowly and inefficiently reading on the input side and pecking their way on the keyboard in the output side.

I guess the reason that people don't invest time in improving their I/O is that it's boring. I'll be the first to agree that learning speed reading or touch typing is boring, however the benefits you'll reap are big - the investment will pay itself every time you interact with your computer.

After getting the basics of speed reading and touch typing out of the way [1], there are many more ways you can improve your I/O. Here are few examples:

• Learn to efficiently skim through documents
• Learn to pick what to read
• Master your IDE
• Automate everything
• ...

[1] And you should never stop practicing.

Running "one off" Conversions In PostgreSQL

The other day we needed to runs a one-off job for converting datetime.weekday() values to day names. Here's what we came up with.

Getting Good Errors from Python Map/Reduce Jobs

At work, we use some Python map/reduce jobs (using Hadoop streaming).

Debugging can be difficult, since Hadoop does not keep the Python stacktraces and even if it does - it's very hard to find it. We decided to use crashlog, and now we get wonderful emails with detailed description of what went wrong.

Notes:

• The current mapper input file is in map_input_file environment variable
• Don't forget to add crashlog.py with -file (see here)
• You must add "." to PYTHONPATH in order to import
• import sys; sys.path.append('.') should do the trick
• Email is not the best solution for distributed logging (you get a lot of email when things go South). I'm going to play with graylog2 in the future.