Hello everyone
My application requires users to upload 50mb+ files that I need to then process and update the database with details after processing.
Here is my current workflow when web app user tries to upload the file:
1. web app calls my web service to get sass uri to upload file to in azure storage
2. web app uploads the file
3. web app creates a work request in service bus
4. processor worker role is listening for new messages on service bus and takes the work request
5. processor worker processes file, deletes it from azure blob since its no longer needed and updates the database with processing details
My concerns are with steps 2 & 3.
What if web app uploads the file but never get arounds to putting a work request in service bus (due to some bug or network error or whatever), I would be left with orphan files in blob that would never get cleaned up or processed.
What seems like a better solution to me is to have a notification mechanism from azure blob to worker role so that whenever a new file upload happens to a particular container in the blob, worker role would be notified of it and it can process it.
Unfortunately, there is no notifications mechanism built into azure blob.
Do you guys see any other better alternatives for my workflow, do you have any suggestions on how to improve it?
Thanks!