Some of the code we write here, particularly reports, can take a long time to process due to the report's inherent complexity. One recent project took over 3 minutes to generate a customer designed-on-the-fly report andspit out the pdf. Since the customer has complete control over the data and report contents, the time it takes to produce the report expands in a linear fashion. You can optimize the hell out of the code, but soon there will be a point where it's spitting out 10,000 pages instead of 1,000 and you're back to square one.
So what can you do?
The obvious choice is to batch process the report and send it to the user by some other means, eg email. This has some serious drawbacks. The user doesn't know if the report has run but the email failed, they can click the generate button 30 times and grind the server to a halt because all the processing is happening in the background. Plus it's not very intuitive: generate a report then check your email. It works, but could be better.
With HTTP 1.0, you don't need to send a content-length so you can "stream" the report to the browser as it is put together. Until, of course, the customer asks for page numbers "Page X of Y". You don't know what Y is until the report is complete and if you've already sent it to the browser...
Then I thought up a trick to fool the browser into not timing out.
With HTTP 1.1, the server sends a "Content-Length: 999" to the browser, which means the server must process the report completely to get its size in bytes, then tell the browser how many bytes to expect. From the time the report is started to the time the browser gets its first reply from the server can be several minutes and the browser may disconnect because it thinks the web server is not responding.
In order to stop the connection timing out, the web server needs to send some data to the browser. Enter dummy http headers. As we process the request the report code sends dummy http headers to the browser, eg "X-PageNumber-1: ok" etc. This trick requires responding completely to the http request yourself rather than relying on the web server so you have to craft your own HTTP/1.1 200 OK etc and set the other mandatory headers. The basic process is:
1. Send the standard headers
2. send an extra dummy header for each report page
3. Send the content-length:
4. Send the data payload