We have developed a WebApi platform which is hosted on Azure, and using MongoDB as a database.
Before release, we did several load tests and it supports up to 11000 requests per second, and right now we have an average load of around 200 requests per second, and everything goes smoothly.
The problem is that, randomly and without any apparent pattern, peaks appear in the queue of HTTP requests and connections to MongoDB, which generates timeouts on the client.
After exchanging several emails with the Azure support department and checking the logs of both MongoDB and Azure, we have no idea what might be happening.
My questions are:
Has anyone experienced the same problem? I see similar cases but they are not exactly the same.
Some tool-library-framework or something to monitor each request, the time it takes each function, and hunt what is causing these spikes?
Greetings