If it won't be simple, it simply won't be. [source code] by Miki Tebeka, CEO, 353Solutions

Monday, May 20, 2013

Want To Be More Productive? - Improve Your I/O

It always surprise me that people trying to makes themselves more productive overlook their I/O - reading and writing.

I still see many people slowly and inefficiently reading on the input side and pecking their way on the keyboard in the output side.

I guess the reason that people don't invest time in improving their I/O is that it's boring. I'll be the first to agree that learning speed reading or touch typing is boring, however the benefits you'll reap are big - the investment will pay itself every time you interact with your computer.

After getting the basics of speed reading and touch typing out of the way [1], there are many more ways you can improve your I/O. Here are few examples:

  • Learn to efficiently skim through documents
  • Learn to pick what to read
  • Master your IDE
  • Automate everything
  • ...
[1] And you should never stop practicing.

Friday, May 17, 2013

Running "one off" Conversions In PostgreSQL

The other day we needed to runs a one-off job for converting datetime.weekday() values to day names. Here's what we came up with.

Wednesday, May 01, 2013

Getting Good Errors from Python Map/Reduce Jobs

At work, we use some Python map/reduce jobs (using Hadoop streaming).

Debugging can be difficult, since Hadoop does not keep the Python stacktraces and even if it does - it's very hard to find it. We decided to use crashlog, and now we get wonderful emails with detailed description of what went wrong.

  • The current mapper input file is in map_input_file environment variable
  • Don't forget to add crashlog.py with -file (see here)
  • You must add "." to PYTHONPATH in order to import
    • import sys; sys.path.append('.') should do the trick
  • Email is not the best solution for distributed logging (you get a lot of email when things go South). I'm going to play with graylog2 in the future.

Blog Archive