Puakma: Under the hood

I'm Brendon Upson, jack-of-all-trades, master of one or two. I'm talking about life running a small ISV tackling business issues and leaping technology hurdles in a single bound.

webWise Network Consultants is based in Sydney, Australia and develops the groundbreaking Tornado Server technology.

Poor neglected blog

Filed under: by Brendon Upson on 2010-03-22

Times change.... Blogging was once flavour of the month and is now sadly relegated to a sideline activity. Much work is still occurring but finding the time to talk about it is proving difficult as work and family dominate the waking hours....

Puakma platform stats

Filed under: by Brendon Upson on 2009-10-15

I've been tinkering for a while now on how to make a web UI to front end the Web Booster ESSO product. Bascially, a way to manage and review the Booster instance from a web browser, mostly an easy way to control the configuration params. The second part of this job was to provide some useful statistics on how the server is performing.

Ok, sounded like an easy job. Until I got digging. I wanted to be able to record statistics in a rolling time window, like http hits per minute, logins per hour etc. Plus also the ability to store/record non numeric values oen off values, like last http hit (date/time). Hmmm. First off I thought I'd just make a simply list that recorded a timestamp (occurrence) and a value. If I wanted hits per hour, simply filter the list for all entries that fell in that time period. This was quickly discounted as a busy server with tens of thousands of hits per hour would quickly use up valuable memory.

The solution was to make periods, a start timestamp and an end timestamp with a counter. This way I could just pass the current time and a value and the stats code would work out which period slot to increment. The second part was to prune the list. We don't want the list to grow and grow, again using up valuable memory. When the stats object is created we specify how many periods of history we want to keep, like if we're recording hourly we don't want more than 24 periods.

This data will drive some fancy charts on the dashboard of the Web Booster admin app.

Tornado does Oracle 10g

Filed under: by Brendon Upson on 2009-07-07

Let me start by saying for an "industry standard" database I was deeply underwhelmed. I'm no Oracle DBA (so let the flaming begin). As I see it, Oracle has 3 major weaknesses, first to actually CREATE a new database is a huge effort and requires new management ports opened in the firewall to administer the instance. Second the JDBC implmentation is not as complete as you would expect. Third, its handling of NULLs vs empty strings is questionable at best.

As you may have guessed, last week involved porting Tornado to use Oracle for its system tables. Boy did I learn a lot.

The database creation thing is merely a comment on scalability. Sure if you have a ton of hardware and a team of DBAs, then this is not so much of an issue, but then I guess if you can afford the software, you'll probably have these things.

The JDBC implementation screwed me over badly. In particular ResultSet.getBytes(). Foolishly I expected this method call to return me the bytes stored in that column. Sadly, Oracle returns 86 bytes every time. It seems the 86 bytes are a blob locator. Why the programmers at Oracle would ever think any human might want the blob locator is well beyond me. So, if you're using Oracle JDBC and blobs, be sure to deal with streams only.

Finally, the NULL handling really killed me. With Oracle if you write an empty string "" to a column, the server will store it as a NULL. Apparently this goes way back before ANSI database standards were written and Oracle do not want to change the behaviour for their existing customer base. OK, I get that, users now expect a certain behaviour. How about a config setting in the database so we can choose to be standard or not?? ANSI92 was written 17 years ago..... What makes this more painful is you cannot write a query "...WHERE ColumnX=NULL" you must write "...WHERE ColumnX IS NULL". Apparently NULL is "unknown" and unknown!=unknown.

Feels good to get that out of my system.

Fedora Core 10 network issues

Filed under: by Brendon Upson on 2009-05-14

Hmm. The new "NetworkManager" stuff is just plain broke. Very friggin annoying. So I set a static IP and unchecked the network manager setting only to find each time I reboot the machine, the network interface eth0 did not start. Arrrrgh!!! Finally figured out what was going. By default the 'network' service does not start. So after enabling the service and restarting it, eth0 appears. Yay. Rebooted the machine as a test and all is (finally) good. 

Tornado's new composite design elements

Filed under: by Brendon Upson on 2009-05-07

Hot on the heels of "Idea #21887 for Tornado server" is the new composite design element functionality.

The problem:
You like to store each of your javascript methods in seperate files so that like functionality is all grouped together. For example, maybe you put all your validation routines in one, you use prototype, mootools, ajax routines in another and so on. This make it easy from a development and maintenance perspective to keep up to date and find things. However, from a performance perspective the web app becomes sloooow. This is because for each included resource (css, javascript), the browser must ask the server if it has changed. Mostly the server replies with a "304 Not Modified", but the browser still has to ask, just in case, and there's a small overhead with each request.

 

The solution - Composite design elements:
We now have a new design element for example called "all.js". This design element is tagged as a composite and simply contains a list of other resources or filesystem based files to imclude into one larger file. Each page in the web app only links to "all.js" and Tornado takes care of assembling that file on the fly. Plus, you can use Tornado's built in ability to shrink the javascript AND it's dynamic gzip ability to compress that minified file. That's 3 perfromance optimizations layered on top of each other :-) The following is what it looks like in the webdesign app, 9 separate javascript files that get amalgamated and sent as one to the browser:

 

composite_element.png

This will work for any text based resources!

Idea #21887 for Tornado server

Filed under: by Brendon Upson on 2009-03-19

In our project management app ProjectReator, there's a bunch of web 2.0 (gee I HATE that term!) whizzbangery. Like popup calendars, prototype, scriptaculous, shadowed boxes, flyout menus, ...  Each of these typically uses its own .js and .css file(s). It works great, but the rub is performance.

In the land of http, when you request a page, your browser gets the page back, looks in the <head> for css and javascript includes and makes suibsequent requests for them. Wouldn't it be great if the browser pulled down only one .css file and one .js file? Hmm yes but as a developer what a nightmare to update one HUGE file instead of a few small ones.

So here's the idea.

On the Tornado server side, create a new design element which contains a list of all the files to concatenate in its response. For example, the page has 'all.css' listed in the <head> when the browser asks the server for 'all.css' it known to grab all the css files (resources) listed that resource and return them as a single reply.

The best of both worlds, from a programming and maintenance perspective things are easy to manage and from a performance perspective the site appears snappy. Now I just have to build it...

Product reinvigoration - Single Signon

Filed under: by Brendon Upson on 2009-03-04

In the last 12 months we have been blindsided by a huge amount of consulting work, building apps on top of Tornado Server. While this has been a busy and interesting time, it has unfortunately removed resources from developing and improving our products. To that end, last week I decided that we really must dedicate time to make booster all that it can be. Looking at the feedback we've garnered from customers and partners over the last period, here's the short list:

- Better documentation of the configuration settings, particularly in puakma.config.
- A web UI to make managing, monitoring and setting up booster much easier
- A more flexible licensing model for very small customers (eg companies with <100 users). Booster is now affordable for one person companies!
- A reseller branded version so that users know to contact the reseller for on site help, upgrades etc

 

Do you have customers that need web single sign on?

It's really easy to become a reseller! Just sign an agreement and you're away. Resellers get a percentage of each product sold and ongoing maintenance, as well as access to fantastic technical support and beta versions. To increase your product portfolio, contact us today.

The email move

Filed under: by Brendon Upson on 2009-02-08

I've been running the company email system on Notes for a long time. Since I was a Notes admin it seemed like the easy choice, but unfortunately over time Notes became less and less effective at stopping the web nasties from pouring in, eventually getting to the point where I was clearing out (manually) over 50 mails a day, and that's after the thunderbird spam filter cleared out another few hundred on top of that. This was all brought to a head with the purchase of the iPhone. The iPhone connects directly to the IMAP mail server so if Thunderbird is not running on my laptop, no junk gets cleaned. The iPhone became instantly useless for mail.

Now I know my way around Linux and had briefly set up and run a postfix server but it's a world of pain.I wanted something that just worked with the minimum of fuss: POP, IMAP, SMTP, webmail, with virus scanning and anti-spam server side.

Then I stumbled across qmail.

I downloaded a premade virtual machine from  http://www.qmtiso.com/ and gave it a run. Seemed to fit the bill, but trying to load it into my ESXi server was a world of pain as it was built with IDE drivers and ESXi runs SCSI. Back to the site, I purchased a copy of their latest ISO (20 bucks!) and installed it straight into ESXi. Setting up the domains and accounts was a snap, then following the setup guide the rest was running perfectly. Took maybe an hour all up which is pretty amazing, since I've burned many hours in the past trying to set up opensource servers using unintelligible config files.

If you're looking for a new mail system, I recommend qmail. Oh and my level of spam has dropped to around 3 a day. :-)