I am trying to optimize performance on GAE but once I deploy I get very unstable results. It's really hard to see if each optimization actually works because datastore and memcache operations take a very variable time (it ranges from milliseconds to seconds for the same operations).
For these tests I am the only one making only one request on the application by refreshing the homepage. There is no other people/traffic happening (besides my own browser requesting images/css/js files from the page).
Edit: To make sure that the drops were not due to concurrent requests from the browser (images/css/js), I've redone the test by requesting ONLY the page with urllib2.urlopen(). Problem persists.
My questions are:
- 1) Is this something to expect due to the fact that machines/resources are shared?
- 2) What are the most common cases where this behavior can happen?
- 3) Where can I go from there?
Here is a very slow datastore get (memcache was just flushed): Full size
Here is a very slow memcache get (things are cached because of the previous request): Full size
Here is a slow but faster memcache get (same repro step as the previous one, different calls are slow): Full size