One of my challenges with parsing large amounts of data, and that after all my excitement of getting my scripts working…the 32 bit version of Python has a memory limitation!

How did I stumble across this problem?

I ran my scripts…everything was going well, so I decided to leave it overnight. I woke up to a very short and stubborn response from Python:

After some quick googling, I figured out what could be the cause – I had installed the 32bit version of Python instead of the 64bit version…doh!

After a quick uninstall (32bit) and install (64bit) I was able to run my scripts just fine:

One interesting that I noticed was that I was parsing about 10,000 – 20,000 more lines per second when running in 64bit than when running 32bit.

Recommendation – if your OS supports it, make sure you install the 64bit version!