Puakma: Under the hood

Ok, the title is a bit misleading since anyone who knows me knows I LIVE for change. I find the day to day predictability droll and do what ever I can to liven it up.

Except when it comes to compilers.

Yup. You read it right. I still develop Tornado with Netbeans 3.6. According to this page , Netbeans 3.6 went gold on 13 April 2004 (trivia: coincidentally my grandfather's birthday). The important question here is why. Some may agree that Netbeans 3.6 is not be the perfect IDE and it's over three and a half years old to boot. Here's why I'm still using an antique compiler: Because it takes me too long to learn a new compiler and I lose too much productive time trying to work out where that feature I use all the time has been hidden in the new release. I would rather suffer through the compiler I know than face the pain of trying to learn something new.

This didn't all just happen by some accident. I went through JBuilder 1, 2, 3, 3.5. Netbeans 3, 3.5. Each change in compiler was a HUGE speed hump. You need to set up a new project in a new format with a new set of project options. Source files may need to be copied or moved. And you're not even sure that once you go through all this pain that you'll even like the new version! 

Having said all that I am planning a move to Eclipse in the near future (but no firm date). I do all my consulting work using our Vortex IDE running on top of the Eclipse framework and I like it, but more importantly, it helps me write better code. Eclipse does a lot of the precompilation work to tell you which parts of your source are not used, which imports need not be and so on. 

Fear the change : I think I'm getting old! 

This last week I've had to chop and change between: Oracle, MySQL, PostgreSQL and HSQLDB. All good databases and all quite different. Some of the differences are a little annoying which makes it a challenge to write an application that works reliably across all platforms.

MySQL (on Linux) requires you to get the case right for table names, so in your code if you have a "SELECT * FROM TABLE" this is not the same (and will fail) on a "SELECT * FROM Table". This is because MySQL stores its data in a file or two for each table called "TABLE.MYD" and Linux being case sensitive means the OS can't find a file called Table.MYD. This approach is nice because backups and restores of a single table or database are easy.

PostgreSQL doesn't suffer the same case sensitivity issues because it stores all its data in a myriad of oddly named files in a bunch of directories. What postgres does choke on though is inserts. For example you have a table with an INTEGER field called age (but you wouldn't right? this is just an example). You use Tornado's TableManager object to insert the data: t.setField("Age", "34"); Hmmm here we go trying to store a String (which converts nicely to an int) into an INTEGER column in the table. Postgres will spit it right back. Mysql happily converts it on the fly and stores it in the table.

Another thing that annoys me is the LIMIT clause. This is not part of the SQL standard so vendors are free to implement this however they like.

Postgres: "SELECT * FROM TABLE LIMIT 10 OFFSET 32"

MySQL: "SELECT * FROM TABLE LIMIT 32,10"

HSQLDB: "SELECT LIMIT 32,10 * FROM TABLE"

Oracle:  "SELECT * FROM TABLE LIMIT 10" (no support for offset)

Microsoft SQL Server: Forget it, the method is too bizarre to discuss.

What amuses me more is if you pass a mysql formatted LIMIT clause to postgres, it throws and error and tells you exactly what you need to change for use on postgres. Hmm. If you're smart enough to be able to tell me what I did wrong, why not just swallow the LIMIT and do the work?? 

My primary goal when writing Tornado was to be able to abstract the database layer to the point where customers could swap out the database server with whatever they wanted to. While this goal has been mostly achieved, as a developer you need to be aware of these little idiosyncrasies in order to make it possible. 

Here's a hot tip for anyone attempting a skypecast: DON'T.

Last night's inaugural Tornado Developers meeting went well, but the skypecast didn't. After wrestling with the time (making sure the cast would start when I expected it to) I finally got the skypecast ready for people to join. Unfortunately it appears in the public skype directory meaning EVERYONE on the planet can join in. BAD, VERY BAD. All these goobers from all over the world started connecting in and yelling and hollering into their microphones completely destroying the meeting. A little like having a bunch of rowdy gatecrashers overtake your party. The meeting was quickly moved.

I had a wee chat with Chris from podmatcher last night and we're going to work together to roll in the ajax chat system I wrote last week with podmatcher. I am optimistic this won't take too long and we can move our meetings permanently to podmatcher. That should save a huge number of headaches and make the chat easily accessible to everyone - even from your work, through a firewall :-)

3,001 Ajax web chat applications have been written. It's almost a right of passage for developers learning the capabilities of the Ajax paradigm. I have been putting off writing one because there were already 3,000 in existence, but last week I bit the bullet and scratched my itch. Sometimes it's good to do stuff just because you can.

The problem I had to solve is: We are about to start a weekly Tornado meeting. I want everyone to be able to find the chat session easily and have it start at a designated time (and finish so noone can rejoin an old one later!). Further, everyone shouldn't need a fancy schmancy chat client and an account somewhere (eg, skype, msn, icq, ...). 

So how does it work?

I have two tables, one called CHATEVENT which holds the meeting information such as start date, time, description etc. The other table is called CHATDATA and this holds the text people type, only row per chat message tagged to a CHATEVENT. At this stage I am not concerned about managing invitees but that could be easily added later. YAGNI 

On the client side, each time you type some text in the input field, it is sent to hte server via an ajax request and poked into the CHATDATA field. There is also a client side thread that polls the server for new messages, reading the CHATDATA table to find any new messages since the last poll. Any new messages are returned and dynamically added to the div which holds the chat messages, creating the illusion of messages being pushed to the client.

The trick I found was to ensure everyone only uses the server time for timestamping messages and have the server return the time of the last message. If you don't you'll get things doubled up - or missed.

On performance, because the ajax chat mechanism works on a polling basis you need to tweak the poll interval so that each client will poll regularly enough so the user experience is nice and infrequently enough to reduce the server load.

Hopefully we can get this integrated into podmatcher soon and start using it as the chat medium, that way if you miss a meeting you can come back later and read the pod meeting's transcript.

If you'd like a copy of this Tornado app, email me :-)