Puakma: Under the hood

I'm Brendon Upson, jack-of-all-trades, master of one or two. I'm talking about life running a small ISV tackling business issues and leaping technology hurdles in a single bound.

webWise Network Consultants is based in Sydney, Australia and develops the groundbreaking Tornado Server technology.


Filed under: by Brendon Upson on 2005-12-15

Yesterday I said I had a crafty plan to work around the file upload and long processing issue. The answer was a combination of HTTP and then some AJAX on the destination page.

The issue I was having is that the project I am working on involves the upload and processing of a xls file. The file once uploaded is parsed and data moved to a relational database. Some files are huge and generate in excess of 10,000 records. As you can imagine there's a fair amount of number crunching involved to import the file, consequently it can take up to 5 minutes to import! The previous solution involved the browser posting the data then sitting there and waiting for a reply from the server. After a short while the browser decided the server is not listening so sends the request again :-(

To get around this I used the clever streamToClient(byte[]) method available in Tornado actions. This allows me to send bytes directly back to the browser (I have to manually craft the HTTP response though, which is OK because we're only talking a few lines) while the action continues to execute. In this case I send a 302 (redirect) to tell the browser to go to a PleaseWait page. The PleaseWait page then has information about the import and some AJAX that polls the server checking whether the import has completed. The user can either sit on the current page and wait to see when the import completes, or continue working while the import chugs away in the background.

I did discover one issue with streamToClient(). The server did not call flush() on the socket during the method call, so the outputstream buffered the reply. This meant that for small http replies, streamToClient() would only seem to take effect when the action completed (which called flush()/close() on the socket). An extra flush was added to the server code and it's working like magic!