It jsut dawned on me today that I write a huge amount of code to do things that people never see and certainly take for granted. This week's chore is data exporting and importing. I developed a few weeks ago the ability to export an dimport Tornado server data. That means you can export from mysql and import into SQL Server. Mmmmm cross platform data movement. This hasn't been without its hurdles though. v1 had been in and working fine for some weeks. This week I had the job of moving some customer data from postgreSQL 8.0 to postgreSQL 7.4, yes, dear readers - backwards. The postgres tools really annoy the crap out of me, and in this case weren't happy about going backwards in time to an older version! No prob I thought, I'll just use the Tornado data exports and import them.
Things were going great until I tried to import a 3.8MB file and the server ran out of memory. Upon closer examination the 3.8MB .pdx file (compressed XML) expanded out to 158MB and 49,000 records (over 3 million lines of XML!!). The XML parser I was sugin in the code tried to load the entire XML document into memory before processing it. Ouch.
So the last two days have been converting the code over to use the SAX parser and ensure the data integrity remains. Not a lot of fun, but by the time you guys get the code, it'll just work like you expect ;-)
/start firefox rant
I've been using firefox on the Mac (oh yeah I LOVE my powerbook!) for well over a year, upgrading with each version. ff1.5 is very much a let down. The developers have changed some of the basic behaviour like non-movable alert/confirm boxes at the top of the screen and clicking in the address bar does not
highlight the address so you can just type over it. I was amazed at how much these two "little" changes peeved me. 1.5 also breaks the in browser rich text editor (which I am using right now to write this). CSS that was working fine before now floods the firefox error log with messages. Bring on 1.51...