@phryk what I would suggest is to debug what gets served from cache and what does not.
Also look at 4xx / 5xx errors, (micro) caching these is important as well!
It might very well be that what brings your site down are 404 responses to /favicon.ico requestsor something just as unexpected.