Puakma: Under the hood

I'm Brendon Upson, jack-of-all-trades, master of one or two. I'm talking about life running a small ISV tackling business issues and leaping technology hurdles in a single bound.

webWise Network Consultants is based in Sydney, Australia and develops the groundbreaking Tornado Server technology.

Merry Xmas

Filed under: by Brendon Upson on 2005-12-23

This year once again rolled around far too quickly. I swear I had an extra two weeks to go until the 25th, but alas I'm proven wrong. As usual there's just too many things to do and not enough time to do them in.

The big ticket items for imminent release are: new puakma.net website and Vortex IDE. Watch this space for an early January launch. Here's a sneak preview:

New puakma.net site

The last couple of days have been spent refining the ability to import and export data (.pdx) in the same way you can export an application's design (.pmx file). This has had a couple of gotchas, like schema differences between source and destination. Currently this works on a table by table case - and it works great! Moving data between different databases is a snap, yesterday I moved some between postgresql and mysql with no problems. The intermediary file is a compressed xml file (as is a .pmx), and we've opened up the API to allow better access to process the data. Next step is I guess a full database export which will be great for backups. Did I mention all this is done in a web browser?

For those of you still foolish enough to be working at this time of year (and everyone else) I wish you the very best for the holiday season and a happy and healthy new year.

More power!

Filed under: by Brendon Upson on 2005-12-20

Build 317 of the IDE was released today. This has some cool functionality for favourites (a short list of apps you regularly work on) and just gets better with every release. The client project I am working on at the moment sees me using the IDE every day and giving it a fair kick. I can say with a high degree of certainty it works great! I find I am resorting to the webdesign app less and less wich is a good thing. We'll continue to support webdesign.pma though as it's still very useful for those times when you have to take a look at the design collection and you don't have your monster Eclipse installation with you.

We're continuing to power on with the redesign of puakma.net. It's always a struggle when you've got a high customer workload on AND you're trying to continue to move forward as fast as possible.

I read something the other day saying the popularity of Java is dropping in favour of scripting languages such as PHP. I squarely blame the myriad of confusing acronyms and Sun's repeated attemps to further confuse the development community. Tornado is a major effort to reduce that confusion and move to using vanilla Java.

2006: The year Tornado Server comes of age.

Crafty

Filed under: by Brendon Upson on 2005-12-15

Yesterday I said I had a crafty plan to work around the file upload and long processing issue. The answer was a combination of HTTP and then some AJAX on the destination page.

The issue I was having is that the project I am working on involves the upload and processing of a xls file. The file once uploaded is parsed and data moved to a relational database. Some files are huge and generate in excess of 10,000 records. As you can imagine there's a fair amount of number crunching involved to import the file, consequently it can take up to 5 minutes to import! The previous solution involved the browser posting the data then sitting there and waiting for a reply from the server. After a short while the browser decided the server is not listening so sends the request again :-(

To get around this I used the clever streamToClient(byte[]) method available in Tornado actions. This allows me to send bytes directly back to the browser (I have to manually craft the HTTP response though, which is OK because we're only talking a few lines) while the action continues to execute. In this case I send a 302 (redirect) to tell the browser to go to a PleaseWait page. The PleaseWait page then has information about the import and some AJAX that polls the server checking whether the import has completed. The user can either sit on the current page and wait to see when the import completes, or continue working while the import chugs away in the background.

I did discover one issue with streamToClient(). The server did not call flush() on the socket during the method call, so the outputstream buffered the reply. This meant that for small http replies, streamToClient() would only seem to take effect when the action completed (which called flush()/close() on the socket). An extra flush was added to the server code and it's working like magic!

Welcome back

Filed under: by Brendon Upson on 2005-12-14

10 days in Thailand was a welcome break. Nearly decided to stay ;-)

Now it's back to the grindstone with the usual 101 things on the boil, juggling and trying to squeeze everything in before xmas. Yesterday hooked up my first Ajax code in anger and was quite happy with the results (and totally amazed at the speed!). Today spent some time trying to decide how to handle a long running server action over a http connection. The major app we're workign on at the moment requires the upload and processing of some hefty files and sometimes the browser times out. There's not a great deal that can be done over a http connection. I have a cruel and cunning plan I'm going to implement tomorrow and see how we go it's be a combination of a crafty 302 redirect and maybe some ajax in the destination page to poll for the results of the upload.