This question is for experienced architects - How do the big boys do it? :)
Overview
I am building this high traffic, analytics like solution based on .NET, it ultimately will hosted on Azure. Let's assume this web app will receive 500M+ "transactions" every day, and these are very quick hits to our servers, very little DB querying for each will be required, virtually all of the heavy lifting will be done on a server side on set intervals. I am pretty sure that I have to implement some sort of a queue that will store all of the incoming hits and implement "aggregators" on a back-end that will run every minute or so to process new items from the queue.
Suggested Solution
Correct me if I am wrong but I am thinking writing these transactions straight to the database (some sort of the log table) would be a mistake, so I will be utilizing the Azure Storage Account (Table) for my queue and spin off couple of Azure Worker Roles (based on need) to process the data and update the database. Thoughts?
It's important to remember that Azure Storage is mostly based on a per-transation model, so I would have to pay for all incoming transations (writes) AND for the transactions for my aggregators (reads). So 500M writes and 500M reads per day, which comes out to be around $100/day. Does that make sense? Also, with using Azure Storage can I read a block of rows (to account for a single transaction) or I would have to read the queue one record at the time?
Lastly, performing a DB insert/update for each row would be an overkill for my aggregators, so I am thinking each one should probably aggregate the workload in memory and then purge it to the database.