I am getting data from 50 different clients every minute 50*1440*31 = 2232000.I need the data to be processed and updated real-time.Each client data to be processed synchronously.All the components i am using is Azure like (Web, DB, Services)
So initially i am doing processing once i get the message and respond back to client.Due to delay in the processing on server.Client response is getting delayed.So i looked for 3 options to give immediate response.
a) Using Service Bus Queue.
b) Running using Parallel Processing (Task.StartNew()).
c) Store the Message and Process it later using worker role continuously.
So i started using Service Bus Queue.
I added one queue and try to load all the 50 clients data into one then process it from queue using Azure Worker role.Then queue length is getting increasing due to delay in processing of message for each client.Each msg processing is taking some where around 15sec.But i need the data to be processed real-time with in that minute the message receives.
So i added separate queue for each client to load the message specific to the client.By using azure worker role i am processing the messages from these 50 queues using 50 queue clients and try to process the messages like as shown below
public override void Run() { Trace.WriteLine("Starting processing of messages"); OnMessageOptions options = new OnMessageOptions(); options.AutoComplete = true; options.MaxConcurrentCalls = 1; options.ExceptionReceived += LogErrors; _client1.OnMessage((receivedMessage) => { ProcessMessage(receivedMessage); }, options); _client2.OnMessage((receivedMessage) => { ProcessMessage(receivedMessage); }, options); . . . _client50.OnMessage((receivedMessage) => { ProcessMessage(receivedMessage); }, options); CompletedEvent.WaitOne(); } public override bool OnStart() { // Set the maximum number of concurrent connections ServicePointManager.DefaultConnectionLimit = 12; // Create the queue if it does not exist already string connectionString = CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString"); var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString); if (!namespaceManager.QueueExists(QueueName1)) { namespaceManager.CreateQueue(QueueName1); } if (!namespaceManager.QueueExists(QueueName2)) { namespaceManager.CreateQueue(QueueName2); } . . if (!namespaceManager.QueueExists(QueueName50)) { namespaceManager.CreateQueue(QueueName50); } _client1 = QueueClient.CreateFromConnectionString(connectionString, QueueName1); _client2 = QueueClient.CreateFromConnectionString(connectionString, QueueName2); . . _client50 = QueueClient.CreateFromConnectionString(connectionString, QueueName50); return base.OnStart(); } public override void OnStop() { // Close the connection to Service Bus Queue _client1.Close(); _client2.Close(); . . _client50.Close(); CompletedEvent.Set(); base.OnStop(); }
Clarifications required:
a) Is the above code works parallel execution of data from multiple clients?If not please let me know what needs to change in code
b) Is this the best way to process the message from 50 queue clients with in that minute?
c) For real-time data processing is it better to use queues in Azure Service Bus or any other options we can use in Azure?
d) One more thing need clarification is how the Azure Usage costs for this intermediate layer added as Service Bus Queue for 50 Queues for the total number of Messages and the Worker role calls for every minute for a monthly transactions around 50*1440*31 = 2232000.If the monthly costs are not more than $100 per month for message storing and message processing using worker role as above we can look for retaining this option of Service Bus Queue.Other wise i need to look for other options in Azure to process it for cost optimizations.I did not get very clear on Service Bus Brokered Connection and costs on Azure Website.That why i am asking about it.
e) How reliable this Azure Service Bus Queue.Once we added all data to be Service Bus Queue.This component needs to be available all time during the day for whole year.This will be critical component for us.If this Service Bus Queue is down for some reason then whole system is down.Such a mission critical.Is it good to use Service Bus Queue for this kind of scenarios?
f)Is there any thing we need to do to make the Service Bus Queue up all the time for the whole year.Is there a way we can get notification email if the service bus queue goes down.To handle processing in different manner.
Please suggest some one from Azure Team or people who is using Azure in these kind of scenarios in their applications.
Thanks in advance