Puakma: Under the hood

I'm Brendon Upson, jack-of-all-trades, master of one or two. I'm talking about life running a small ISV tackling business issues and leaping technology hurdles in a single bound.

webWise Network Consultants is based in Sydney, Australia and develops the groundbreaking Tornado Server technology.

Puakma

Puakma platform stats

Filed under: by Brendon Upson on 2009-10-15

I've been tinkering for a while now on how to make a web UI to front end the Web Booster ESSO product. Bascially, a way to manage and review the Booster instance from a web browser, mostly an easy way to control the configuration params. The second part of this job was to provide some useful statistics on how the server is performing.

Ok, sounded like an easy job. Until I got digging. I wanted to be able to record statistics in a rolling time window, like http hits per minute, logins per hour etc. Plus also the ability to store/record non numeric values oen off values, like last http hit (date/time). Hmmm. First off I thought I'd just make a simply list that recorded a timestamp (occurrence) and a value. If I wanted hits per hour, simply filter the list for all entries that fell in that time period. This was quickly discounted as a busy server with tens of thousands of hits per hour would quickly use up valuable memory.

The solution was to make periods, a start timestamp and an end timestamp with a counter. This way I could just pass the current time and a value and the stats code would work out which period slot to increment. The second part was to prune the list. We don't want the list to grow and grow, again using up valuable memory. When the stats object is created we specify how many periods of history we want to keep, like if we're recording hourly we don't want more than 24 periods.

This data will drive some fancy charts on the dashboard of the Web Booster admin app.

Tornado does Oracle 10g

Filed under: by Brendon Upson on 2009-07-07

Let me start by saying for an "industry standard" database I was deeply underwhelmed. I'm no Oracle DBA (so let the flaming begin). As I see it, Oracle has 3 major weaknesses, first to actually CREATE a new database is a huge effort and requires new management ports opened in the firewall to administer the instance. Second the JDBC implmentation is not as complete as you would expect. Third, its handling of NULLs vs empty strings is questionable at best.

As you may have guessed, last week involved porting Tornado to use Oracle for its system tables. Boy did I learn a lot.

The database creation thing is merely a comment on scalability. Sure if you have a ton of hardware and a team of DBAs, then this is not so much of an issue, but then I guess if you can afford the software, you'll probably have these things.

The JDBC implementation screwed me over badly. In particular ResultSet.getBytes(). Foolishly I expected this method call to return me the bytes stored in that column. Sadly, Oracle returns 86 bytes every time. It seems the 86 bytes are a blob locator. Why the programmers at Oracle would ever think any human might want the blob locator is well beyond me. So, if you're using Oracle JDBC and blobs, be sure to deal with streams only.

Finally, the NULL handling really killed me. With Oracle if you write an empty string "" to a column, the server will store it as a NULL. Apparently this goes way back before ANSI database standards were written and Oracle do not want to change the behaviour for their existing customer base. OK, I get that, users now expect a certain behaviour. How about a config setting in the database so we can choose to be standard or not?? ANSI92 was written 17 years ago..... What makes this more painful is you cannot write a query "...WHERE ColumnX=NULL" you must write "...WHERE ColumnX IS NULL". Apparently NULL is "unknown" and unknown!=unknown.

Feels good to get that out of my system.

Tornado's new composite design elements

Filed under: by Brendon Upson on 2009-05-07

Hot on the heels of "Idea #21887 for Tornado server" is the new composite design element functionality.

The problem:
You like to store each of your javascript methods in seperate files so that like functionality is all grouped together. For example, maybe you put all your validation routines in one, you use prototype, mootools, ajax routines in another and so on. This make it easy from a development and maintenance perspective to keep up to date and find things. However, from a performance perspective the web app becomes sloooow. This is because for each included resource (css, javascript), the browser must ask the server if it has changed. Mostly the server replies with a "304 Not Modified", but the browser still has to ask, just in case, and there's a small overhead with each request.

 

The solution - Composite design elements:
We now have a new design element for example called "all.js". This design element is tagged as a composite and simply contains a list of other resources or filesystem based files to imclude into one larger file. Each page in the web app only links to "all.js" and Tornado takes care of assembling that file on the fly. Plus, you can use Tornado's built in ability to shrink the javascript AND it's dynamic gzip ability to compress that minified file. That's 3 perfromance optimizations layered on top of each other :-) The following is what it looks like in the webdesign app, 9 separate javascript files that get amalgamated and sent as one to the browser:

 

composite_element.png

This will work for any text based resources!

Idea #21887 for Tornado server

Filed under: by Brendon Upson on 2009-03-19

In our project management app ProjectReator, there's a bunch of web 2.0 (gee I HATE that term!) whizzbangery. Like popup calendars, prototype, scriptaculous, shadowed boxes, flyout menus, ...  Each of these typically uses its own .js and .css file(s). It works great, but the rub is performance.

In the land of http, when you request a page, your browser gets the page back, looks in the <head> for css and javascript includes and makes suibsequent requests for them. Wouldn't it be great if the browser pulled down only one .css file and one .js file? Hmm yes but as a developer what a nightmare to update one HUGE file instead of a few small ones.

So here's the idea.

On the Tornado server side, create a new design element which contains a list of all the files to concatenate in its response. For example, the page has 'all.css' listed in the <head> when the browser asks the server for 'all.css' it known to grab all the css files (resources) listed that resource and return them as a single reply.

The best of both worlds, from a programming and maintenance perspective things are easy to manage and from a performance perspective the site appears snappy. Now I just have to build it...

Product reinvigoration - Single Signon

Filed under: by Brendon Upson on 2009-03-04

In the last 12 months we have been blindsided by a huge amount of consulting work, building apps on top of Tornado Server. While this has been a busy and interesting time, it has unfortunately removed resources from developing and improving our products. To that end, last week I decided that we really must dedicate time to make booster all that it can be. Looking at the feedback we've garnered from customers and partners over the last period, here's the short list:

- Better documentation of the configuration settings, particularly in puakma.config.
- A web UI to make managing, monitoring and setting up booster much easier
- A more flexible licensing model for very small customers (eg companies with <100 users). Booster is now affordable for one person companies!
- A reseller branded version so that users know to contact the reseller for on site help, upgrades etc

 

Do you have customers that need web single sign on?

It's really easy to become a reseller! Just sign an agreement and you're away. Resellers get a percentage of each product sold and ongoing maintenance, as well as access to fantastic technical support and beta versions. To increase your product portfolio, contact us today.

Browser timeouts

Filed under: by Brendon Upson on 2008-07-16

Some of the code we write here, particularly reports, can take a long time to process due to the report's inherent complexity. One recent project took over 3 minutes to generate a customer designed-on-the-fly report andspit out the pdf. Since the customer has complete control over the data and report contents, the time it takes to produce the report expands in a linear fashion. You can optimize the hell out of the code, but soon there will be a point where it's spitting out 10,000 pages instead of 1,000 and you're back to square one.

So what can you do?

The obvious choice is to batch process the report and send it to the user by some other means, eg email. This has some serious drawbacks. The user doesn't know if the report has run but the email failed, they can click the generate button 30 times and grind the server to a halt because all the processing is happening in the background. Plus it's not very intuitive: generate a report then check your email. It works, but could be better.

With HTTP 1.0, you don't need to send a content-length so you can "stream" the report to the browser as it is put together. Until, of course, the customer asks for page numbers "Page X of Y". You don't know what Y is until the report is complete and if you've already sent it to the browser...

Then I thought up a trick to fool the browser into not timing out. 

With HTTP 1.1, the server sends a "Content-Length: 999" to the browser, which means the server must process the report completely to get its size in bytes, then tell the browser how many bytes to expect. From the time the report is started to the time the browser gets its first reply from the server can be several minutes and the browser may disconnect because it thinks the web server is not responding.

In order to stop the connection timing out, the web server needs to send some data to the browser. Enter dummy http headers. As we process the request the report code sends dummy http headers to the browser, eg "X-PageNumber-1: ok" etc. This trick requires responding completely to the http request yourself rather than relying on the web server so you have to craft your own HTTP/1.1 200 OK etc and set the other mandatory headers. The basic process is:

1. Send the standard headers
2. send an extra dummy header for each report page
3. Send the content-length:
4. Send the data payload

 

dummyheaders.png

 

 

Birthday!

Filed under: by Brendon Upson on 2008-06-13

Weeeeeell. Today marks the "official" 7th birthday of Puakma. Truth be told I don't know the exact date it all started, but it is definitely somewhere around this time.

It's been a surreal last 6+ months with all sorts of weird and wonderful projects here at WNC, and I'll be the first to admit that public releases of Tornado have not been near the top of the list. Today we'll change that. The 4.1 release is now public.

Puakma Enterprise Server Platform v4.1.23 Build:818 - 4 June 2008
http://www.puakma.net/wnc/dltk.pma/GetLatestSoftware?OpenAction&PartNumber=PT0022 
Enjoy ;-) 

Eating dogfood with i18n

Filed under: by Brendon Upson on 2008-04-09

So Tornado is getting pretty mature, but all the work to date has pretty much revolved around a single company server where everyone is in the same timezone and speaks the same language. Until now :-(

As we peg away at ProjectReactor, things crop up. Important things. Let's say we have a project with 2 project managers, one in Sydney Australia and one in New York USA. The Sydney PM enters a milestone to be due mid-day friday, when the New York PM looks at the same record he also sees due mid-day friday. Disaster! Different time zones.

Then there's the date input formats. Sydney PM uses "dd/MM/yy" and New York PM uses "MM/dd/yy". Ouch. How do we make one application design easily cater for the different formats (and also "yy/mm/dd" and many other variations)?

When you have the sourcecode, anything is possible :-) 

Step one: Extend the session object. Add two new attributes to the session; time zone and locale. Upon login we get the timezone from the browser and the locale (language) from their account record. The server code then uses these new properties (if they have been set) when dealing with dates. If they are never set by the application then we default to the server's locale.

Step two: Custom date input formats. This was the tough one. The Tornado app uses <P@Date @P> fields with a format attribute, eg <P@Date name="x" format="dd/MM/yy" @P> The format attribute is stored in a page design element and is certainly not easy to change on a per user basis! Custom session objects to the rescue. Tornado allows any kind of object to be stored on a user's session. So now we just store a session object called "dd/MM/yy" on the session and give it value we want for that user eg "yy.MM.dd". Now when the server is parsing and rendering dates it does a check on the session to see if there is a session object of the same name as the format attribute's value, if there is use this value instead of the format attribute value. If there is no session object by that name, then just use the value this allows it to work either way.

i18n_pr.png

My head hurts. I'm off to bed.