Ok, sounded like an easy job. Until I got digging. I wanted to be able to record statistics in a rolling time window, like http hits per minute, logins per hour etc. Plus also the ability to store/record non numeric values oen off values, like last http hit (date/time). Hmmm. First off I thought I'd just make a simply list that recorded a timestamp (occurrence) and a value. If I wanted hits per hour, simply filter the list for all entries that fell in that time period. This was quickly discounted as a busy server with tens of thousands of hits per hour would quickly use up valuable memory.

The solution was to make periods, a start timestamp and an end timestamp with a counter. This way I could just pass the current time and a value and the stats code would work out which period slot to increment. The second part was to prune the list. We don't want the list to grow and grow, again using up valuable memory. When the stats object is created we specify how many periods of history we want to keep, like if we're recording hourly we don't want more than 24 periods.

This data will drive some fancy charts on the dashboard of the Web Booster admin app.

As you may have guessed, last week involved porting Tornado to use Oracle for its system tables. Boy did I learn a lot.

The database creation thing is merely a comment on scalability. Sure if you have a ton of hardware and a team of DBAs, then this is not so much of an issue, but then I guess if you can afford the software, you'll probably have these things.

The JDBC implementation screwed me over badly. In particular ResultSet.getBytes(). Foolishly I expected this method call to return me the bytes stored in that column. Sadly, Oracle returns 86 bytes every time. It seems the 86 bytes are a blob locator. Why the programmers at Oracle would ever think any human might want the blob locator is well beyond me. So, if you're using Oracle JDBC and blobs, be sure to deal with streams only.

Finally, the NULL handling really killed me. With Oracle if you write an empty string "" to a column, the server will store it as a NULL. Apparently this goes way back before ANSI database standards were written and Oracle do not want to change the behaviour for their existing customer base. OK, I get that, users now expect a certain behaviour. How about a config setting in the database so we can choose to be standard or not?? ANSI92 was written 17 years ago..... What makes this more painful is you cannot write a query "...WHERE ColumnX=NULL" you must write "...WHERE ColumnX IS NULL". Apparently NULL is "unknown" and unknown!=unknown.

Feels good to get that out of my system.

The problem:
You like to store each of your javascript methods in seperate files so that like functionality is all grouped together. For example, maybe you put all your validation routines in one, you use prototype, mootools, ajax routines in another and so on. This make it easy from a development and maintenance perspective to keep up to date and find things. However, from a performance perspective the web app becomes sloooow. This is because for each included resource (css, javascript), the browser must ask the server if it has changed. Mostly the server replies with a "304 Not Modified", but the browser still has to ask, just in case, and there's a small overhead with each request.

The solution - Composite design elements:
We now have a new design element for example called "all.js". This design element is tagged as a composite and simply contains a list of other resources or filesystem based files to imclude into one larger file. Each page in the web app only links to "all.js" and Tornado takes care of assembling that file on the fly. Plus, you can use Tornado's built in ability to shrink the javascript AND it's dynamic gzip ability to compress that minified file. That's 3 perfromance optimizations layered on top of each other :-) The following is what it looks like in the webdesign app, 9 separate javascript files that get amalgamated and sent as one to the browser:

This will work for any text based resources!

In the land of http, when you request a page, your browser gets the page back, looks in the <head> for css and javascript includes and makes suibsequent requests for them. Wouldn't it be great if the browser pulled down only one .css file and one .js file? Hmm yes but as a developer what a nightmare to update one HUGE file instead of a few small ones.

So here's the idea.

On the Tornado server side, create a new design element which contains a list of all the files to concatenate in its response. For example, the page has 'all.css' listed in the <head> when the browser asks the server for 'all.css' it known to grab all the css files (resources) listed that resource and return them as a single reply.

The best of both worlds, from a programming and maintenance perspective things are easy to manage and from a performance perspective the site appears snappy. Now I just have to build it...

- Better documentation of the configuration settings, particularly in puakma.config.
- A web UI to make managing, monitoring and setting up booster much easier
- A more flexible licensing model for very small customers (eg companies with <100 users). Booster is now affordable for one person companies!
- A reseller branded version so that users know to contact the reseller for on site help, upgrades etc

Do you have customers that need web single sign on?

It's really easy to become a reseller! Just sign an agreement and you're away. Resellers get a percentage of each product sold and ongoing maintenance, as well as access to fantastic technical support and beta versions. To increase your product portfolio, contact us today.

Now I know my way around Linux and had briefly set up and run a postfix server but it's a world of pain.I wanted something that just worked with the minimum of fuss: POP, IMAP, SMTP, webmail, with virus scanning and anti-spam server side.

Then I stumbled across qmail.

I downloaded a premade virtual machine from  http://www.qmtiso.com/ and gave it a run. Seemed to fit the bill, but trying to load it into my ESXi server was a world of pain as it was built with IDE drivers and ESXi runs SCSI. Back to the site, I purchased a copy of their latest ISO (20 bucks!) and installed it straight into ESXi. Setting up the domains and accounts was a snap, then following the setup guide the rest was running perfectly. Took maybe an hour all up which is pretty amazing, since I've burned many hours in the past trying to set up opensource servers using unintelligible config files.

If you're looking for a new mail system, I recommend qmail. Oh and my level of spam has dropped to around 3 a day. :-)

On the music side of things, all my new guitar amp gear has arrived (Mesa boogie Triaxis and 20/20 power amp) and it sounds FANTASTIC. Well worth the pain, cost and effort to get the pieces here from the US. To go with the new gear, I built a pair of speaker boxes to go with the new stereo guitar amp :-) To buy new here they're stupidly expensive and very hard to get. They're only single 12" boxes but sound HUGE with great bass response, based on a Thiele design for Electrovoice (TL806). The only downside is the full kit is now more difficult to transport and takes a bit longer to set up, but did I mention it sounds FANTASTIC??

Rock on!

As with all Apple products its industrial design is fantastic. The finish is slick, it has enough weight to let you know it's solid but light enough to easily cart about. Again like most Apple products it has a limited number of external buttons and it quite intuitive to use. In fact the manual in the box is almost non-existant!

What I like:
- The finish, form and styling
- The screen is amazing as is the OS
- The gesture based UI is very intuitive
- You can switch between 3G and GSM networks (yeeha!)
- Wireless performance is good, easily locates and connects to access points
- The app store and apps are a great way to extend the phone, weather, times, conversions, ...
- The virtual keyboard guesses well when you make a typo and magically fixes it
- Having a decent web browser so I can look stuff up while I'm on the go
- GPS finds your location quickly

What I don't like:
- Every time I plug it in to the laptop iPhoto starts up. Annoying
- Synching is via a cable rather than bluetooth
- Uses a non-standard charger. Would have been nice if it accepted Nokia style power.
- Google maps for navigation is rubbish. No voice navigation, useless in the car, plus the screen powers off after the timeout. No map cache so you have to be connected to the internet all the time, which would be fine if the 3g network was better :-(
- You can't easily use it without looking at the screen (Don't use your mobile while driving!) because there are no raised buttons to feel for
- There are some UI quirks where it occasionally (rarely) becomes temporarily unresponsive
- Bluetooth is incredibly limited, no way to send photos, contacts etc between phones - this was great on the nokia
- No MMS. Not a train smash but really, would it kill Apple to include it??
- There's a ton of cr@p in the app store
- No easy way to access the phones 3g internet from my laptop
- No external memory cards - again would it kill Apple to include one? 8 or 16gig is limiting if you have a pile of videos
- the 'buttons' on the virtual keyboard are quite small, if you have big fingers it would be more difficult to use

Despite the negatives I really liking it. I assume that software upgrades will solve most iof the niggles. Email is my world so having easy access is great - and a drain - 'hey, I'll just check my email again while we're waiting for the entrees to come out'.  The lines are now getting very blurred between mobile phone and computer. Over time as processing power and battery technology improves we'll see the phone become more of a computer and less of a phone.

Sadly the quality of the 3g network is its weakness. As soon as your 3g connection drops out (and you don't have a wireless network) the phone is partially lobotomized.

May all your endings be happy :-)

In the true tradition of 'buy once, buy right', I've gone the MesaBoogie route. I have a Triaxis on its way from the US - they are almost non-existant second hand here and extortionate new: \$4,850 ! Ouch. And that's just the preamp. I'm now on the lookout for a mesa 20/20 valve power amp to go with it. I also dug up some plans for a Thiele design 1 x 12" speaker of which mesa make a version. 

When all is said and done I should have a killer rig that I can take into my twilight years.

My band is in the process of recording the first song with me on guitar so I'm looking forward to hearign how that turns out. Hopefully the Mesa will arive shortly (it should!!) and I can use that for the recording. Fingers crossed.

Here's my scenario. I am on a 24 month mobile phone contract with 3 (18 months in). I go on a trip to New Zealand and make a number of calls back to Australia. I expect to pay more, no problem. While I'm there I get a call from the lovely Indian call centre telling me I have to pay \$500 within the next 2 days as my account has "high usage". Here I am on a PLAN, paying ~ \$130 per month and my bill has "blown out" to \$500. If I don't pay my phone will be suspended. 

Here's the kicker. In no way can I itemise the \$500+ bill to see what I am actually paying for.

So it seems 3 has an arbitrary "high usage" policy, you have no way of knowing before the end of the month what you are paying for and if you don't pay the "excess" charges BEFORE THE END OF THE PERIOD, your "account" will be suspended. 

Call me a slow learner. I was with 3 mobile a few years back and they screwed me over then. Foolishly I signed up with them again.

Here's how it should have gone:

3: Hello sir, we notice some high usage on your account. Are you aware of it?

Me: Why yes, I am in New Zealand at the moment so my international calls will have gone up.

3: Your account is currently \$500. Is that Ok?

Me: No problem. Thanks for letting me know.

Instead we have a frustrated conversation with a non-native english speaker, this ends a few days later with my phone suspended and me looking at what other phone companies offer. This poor customer service will cost 3 in excess of \$3,000 when I move to another provider. Was it worth it?

On top of that I've been giving the guitar(s) a good workout. I am suffering a serious bout of GAS (Gear Acquisition Syndrome) which will hopefully be cured with some Mesa/Boogie amplification. Surely SOMEONE will be wanting to sell a Triaxis for \$200!! I live in hope....

On the home front we're busy each weekend still trying to tame the half acre and turn it into an "easycare" block. Needless to say, the chainsaw is getting a good workout.

Christmas will be here before we know it.

When I started out I used emails and scraps of paper. Life was simple then.

Over time with more customers and more simultaneous projects, more team members my head began to explode. Customers would call or email asking (reasonably) for a status update and I'd have to review my paper scraps, talk to the team, ... On top of that, we had multiple cusomers each with a potentially different hourly rate (for the type of work undertaken) and others on fixed priced deliverables. How would I know who to bill what and when? What if a customer queried their bill and wanted to know exactly what they were being billed for (not an unreasonable request!)?

I did what any software developer would do: I wrote my own project management system. Initially I had no thoughts of anyone else using it and it was a purely internal thing. One of the key componentsof the system is allowing customers to become part of the process. They control what goes into each phase of the project, as well as the approval of deliverables and seeing what their bill will beas the project goes. This gives the customer a high level of visibility of the project and also means I don't need to constantly contact them to tell them where something is up to - they log in and see the latest state of the project (and if there's anything that requires their attention).

As an added benefit, customers started adding new deliverables to their projects which meant more billable work for us to complete. Perfect!

A number of our customers had asked if they can use the system to manage their own projects, so last year we began building www.ProjectReactor.net. As with small business, the development of this was done "in between" the other jobs, so it took a little longer than we expected.

For us, EVERYTHING goes through Project Reactor, customer work, internal jobs, marketing, documentation, you name it. This is a critical component of managing what we do.

The good news for you, dear reader, is that ProjectReactor is now ready for your use :-). 

So there you have it. To run your business effectively you need a means of quantifying and tracking what you are doing. Take a look at Project Reactor - it may just suit you perfectly :-)

Newbie mistake #1 is to look at the job decide on how many hours TECHNICAL effort is required and supply a quote based on that. When you've been around the block a few times you realise the technical effort is only one part of the overall solution. Techies regularly forget the 8 hours it took to do the technical design, then the estimate based on that design. Those are 8 hours you'll never get back so they need to be loaded into the project estimate.

Then there's the multitude of hour long phone conversations with your customer as the project progresses. While your talking to your customer you won't be doing the "technical" work. More lost time. Again, this time needs to be allowed for in the overall project sum.

There's also an overhead for "managing" the project. This is why full time project managers exist. The project manager's role is to ensure the client is happy, deliverables are being tracked and managed, and the project team is happy and productive, amongst other tasks. Again, this requires time to make sure the project is in order, and much more time to put it back in order if it's not. More time to be loaded into the overall project sum.

Newbie mistake #2 is to completely ignore risk. Sometimes we're asked to do things that we've never done before. We know pretty much how to do them, but there's always some bits and pieces that leave us wonering if the project is even possible. There's no way to know for sure until you actually do the work, but if the client only wants a quote for what it will cost it doesn't make sense to do the work first! So there's a risk that the project is going to have some catastrophic speed hump that will kill all in its path. So to mitigate the risk, a lump is added to the project sum for risk. More risk, more dollars. If the client complains about the price, explain the risks to them and they'll either change the brief to remove/reduce the risk, allow you to perform that part on an hourly rate or absorb the project sum. There is always risk in all IT projects simply due to the vast array of software and hardware that has to work together and each piece can be configured in 101 ways. Work out which parts of the project are a bit scary and add some dollars just in case you have to do extra work (or rework) to meet the project brief.

Newbie mistake #3 is not having any cashflow. So you've given your client the perfect quote you know you can deliver, it's four million dollars and will take two years to complete. Awesome. When do you get paid? Four million bucks at the end when you've finished? What if your client goes out of business? What if your client doesn't like what you've done and won't pay? What if your client has an internal restructure and funding for your project is pulled? This is where you may need to negotiate a little. The client would obviously prefer to pay the bill at the end, you'd rather they pay it all up front. In practice you should have a schedule of deliverables. These deliverables mark major events in the project and have a payment amount attached to them, for example:

Project acceptance 10%
Technical design document  10%
Signoff on the UI 40%
Ecommerce linkages built 20%
Final project delivery 20%

This serves a number of purposes. First, you get paid for your work as you go - congratulations you can eat this month! Secondly, the client becomes integral to the project process (very important) and they are checking and signing off on the project progress as you go so at there end there should be few surprises. Finally if the project falls terminally ill at any point, you can walk away with compensation for the work you have completed or conversely if your client wants out they can use the work they have paid for. Ideally you will want to try to push for more payment up front and smaller payments toward the end just to cover yourself. Include enough milestones to ensure you have a steady stream of cash coming in.

Be sure to make the milestones non subjective, that way there is no question that they have been met. Don't be afraid to insist on payment before continuing, with old clients who have a good payment history this is not an issue, but new clients may like to play some payment games. You meet the miestone, issue the invoice and merrily begin on the next phase while your invoice disappears into the ether. Keep an eye on where payment is at and don't be afraid to ask why you haven't been paid yet - just because you seem to be working for a large company doesn't mean that don't have cashflow issues of their own! If you don't get paid STOP. You'll be amazed at how quickly cash will materialise!

Next time: Tracking your work. "I've got 5 customers and 11 projects all running simultaneously. Where are we up to?"

The first question you need to ask yourself is: Why do I want to do it? If you answer is "for the money", beware. Consulting is a services based business which means you have two natural constraints, time and the amount people will pay for your service. There is a limit to what you can earn and it's actually not that high. If your answer is "for the money" look at selling some kind of product, or turn your service into a product if possible. This is all beyond the scope of what I'm going to talk about here.

The biggest mistake freelance consultants make is not charging enough. We've been conditioned to think in terms of cheap. "If XYZ Co charge \$2000 a day, then I'll charge \$1000 a day and I'll be flooded with work". Wrong. First, the amount you charge creates a perception of quality. Second, XYZ charge this amount for a reason. They have calculated training, annual leave, sick leave, down time, overheads (office rent, phone, payroll admin, ...), basically, in order to run a sustainable business this is the amount you NEED to charge.

How do you work out how much to charge?

It's actually quite easy. First is your salary. If you're leaving a \$60,000 a year job to start a consulting company I assume you want to earn at least as much. The formula is basically: Your salary (or rate) + Overheads + Profit margin = your rate to a customer. Overheads are those bills that roll in every month whether you are working or not (sometimes called fixed costs). These include office rent, computer equipment, phone costs etc. The profit margin is the premium you add for being in business. Businesses are about making a profit. This can be whatever you want, youcould use around 10% as a base line.

In the previous example, we'll assume monthly overheads of \$2,000, your gross monthly salary is \$5,000 and you want to make a 10% profit. The formula is:

(5,000 + 2,000) * 1.1 =  \$7,700 per month.

Now that we have the monthly figure it's easy to work out an hourly rate. There are about 160 working hours in a month, so 7,700 / 160 = \$48.12 per hour.

Sloooow down skippy, we're not done yet. 

It is pretty unlikely you will bill the full 160 hours every month so we need to load in a contingency factor. Let's assume you'll be paid for 70% of the time. Those 160 hours now fall to 120 hours but of course your overheads and gross salary remain the same, so 7,700 / 120 = \$64.16 per hour. 

At this point STOP. Compare that rate to what you can get as a contractor or permanent employee. They get paid for the full 160 hours every month! If the numbers don't add and you really want to run your own consulting business, look at which overheads you can cut, then re-run the figures.

Now you know what you need to make as a minimum just to stay in business. Now compare that figure to your competitors. Are you high or low. If you're too high, abort, the numbers don't stack up ;-) If you're too low (great!) increase your rate to that of your competitors to maintain your level of perceived quality.

Next time we'll talk about fixed price work. That's a whole different can of worms!

So what can you do?

The obvious choice is to batch process the report and send it to the user by some other means, eg email. This has some serious drawbacks. The user doesn't know if the report has run but the email failed, they can click the generate button 30 times and grind the server to a halt because all the processing is happening in the background. Plus it's not very intuitive: generate a report then check your email. It works, but could be better.

With HTTP 1.0, you don't need to send a content-length so you can "stream" the report to the browser as it is put together. Until, of course, the customer asks for page numbers "Page X of Y". You don't know what Y is until the report is complete and if you've already sent it to the browser...

Then I thought up a trick to fool the browser into not timing out. 

With HTTP 1.1, the server sends a "Content-Length: 999" to the browser, which means the server must process the report completely to get its size in bytes, then tell the browser how many bytes to expect. From the time the report is started to the time the browser gets its first reply from the server can be several minutes and the browser may disconnect because it thinks the web server is not responding.

In order to stop the connection timing out, the web server needs to send some data to the browser. Enter dummy http headers. As we process the request the report code sends dummy http headers to the browser, eg "X-PageNumber-1: ok" etc. This trick requires responding completely to the http request yourself rather than relying on the web server so you have to craft your own HTTP/1.1 200 OK etc and set the other mandatory headers. The basic process is:

1. Send the standard headers
2. send an extra dummy header for each report page
3. Send the content-length:
4. Send the data payload

Which then got me thinking. The trend is to add more and more cores to CPUs, which is great so long as the software that runs on it is multithreaded. Sure you'll get a good boost by running multiple applications, but what about web applications? Yes, two incoming web requests run in their own threads and there is a certain amount of parallelization (is that a word?) within that request, but at some point the web server will run your custom code which does the actual work. How well does that cater for a parallel approach. My guess is in most cases, not at all. Here we have a bottle neck. Add 100 more cores and your code will go no faster. 

Ok, so just make it multithreaded?

Well, not so fast. This is a difficult LOGICAL problem. Consider you are writing a lump of code to dress a person. Can you put shoes and socks on at the same time? Do trousers come before or after shoes? For most tasks there is a sequential order. It is a difficult task to determine which tasks can be run in parallel and which are sequential and usually there is a combination of both task types.

As programmers we are on the cusp of having to change the way we think about problems and probably use some new tools to get that work done.

1. I managed to snap my right arrow key on my macbook pro (my fault completely....). I though I could get a replacement KEY. No. I have to buy an entire KEYBOARD. Thats \$95 plus \$75 for a half hour labour. Robbery. Apple will never make it in the server market until they change their hardware support (which is notoriously bad) and pricing.

2. iPhone 3g - big deal?

Sure, it's "just a phone". The device itself is not entirely a market changer, but the way we will use it will be. More and more we will use this (and devices like it) more than we use our PC. Why? Because it's convenient, it's always with us and always connected. This is why the blackberry has been so popular. The difference with the iphone is that for the first time we have a decent size screen and an intuitive way of interacting with it. IMHO microsoft have totally lost the plot. The Windows franchise is dying quickly as worthy competitors in Mac OS X and Linux are hammering at the gates and their online strategy is seemingly non-existant. Devices like the iphone and later Andriod from Google will serve to completely bypass the traditional PC business, meaning Windows and desktop operating systems will become redundant. The phone IS the PC.