Conversation

Notices

1. Embed this notice
   phryk 🏴 (phryk@mastodon.social)'s status on Tuesday, 05-Nov-2024 01:12:57 JST phryk 🏴

   Aaand the site died. Thanks Fedi. 😂

   • Embed this notice
     Michał "rysiek" Woźniak · 🇺🇦 (rysiek@mstdn.social)'s status on Tuesday, 05-Nov-2024 01:12:56 JST Michał "rysiek" Woźniak · 🇺🇦

     in reply to

     @phryk whoops.

     I should probably write up my set-up, with some pretty trivial microcaching that solves this issue.

   • Embed this notice
     Michał "rysiek" Woźniak · 🇺🇦 (rysiek@mstdn.social)'s status on Tuesday, 05-Nov-2024 01:41:58 JST Michał "rysiek" Woźniak · 🇺🇦

     in reply to

     @phryk seriously, just microcache. 🙂

   • Embed this notice
     phryk 🏴 (phryk@mastodon.social)'s status on Tuesday, 05-Nov-2024 01:41:59 JST phryk 🏴

     in reply to

     • Michał "rysiek" Woźniak · 🇺🇦

     @rysiek Yeah, I probably only have to fix like one line in my nginx caching setup.

     Or make my rate limiting more aggressive, but unless necessary, I'd rather have requests take longer than just giving people the custom HTTP 420 page. ^^

   • Embed this notice
     phryk 🏴 (phryk@mastodon.social)'s status on Tuesday, 05-Nov-2024 01:41:59 JST phryk 🏴

     in reply to

     • Michał "rysiek" Woźniak · 🇺🇦

     @rysiek OTOH, just fixing the issue with multiple UWSGI workers might have avoided this issue. Currently everything is just one UWSGI worker and the nginx caching.

     Anyhow, monitoring indicates the big spike is over and the site is reachable again. :)

   • Embed this notice
     Michał "rysiek" Woźniak · 🇺🇦 (rysiek@mstdn.social)'s status on Tuesday, 05-Nov-2024 01:50:34 JST Michał "rysiek" Woźniak · 🇺🇦

     in reply to

     @phryk microcaching is just shorter, a few seconds, and is useful if you have dynamic content. If you don't have dynamic content, longer caching periods are better.

     `uwsgi_caching` might be enough for UWSGI content, but it depends on the other configuration. There are a few tricky edge cases where cache gets busted for no good reason – often it's because of random query params ("?something=etc&something_else=etc") added by e.g. social media (like Facebook's "fbclid" param). Or cookies.

   • Embed this notice
     phryk 🏴 (phryk@mastodon.social)'s status on Tuesday, 05-Nov-2024 01:50:35 JST phryk 🏴

     in reply to

     • Michał "rysiek" Woźniak · 🇺🇦

     @rysiek Honestly, I have no idea what differentiates normal caching from microcaching.

     I use the uwsgi_cache directive to enable caching on the site – does that count? :thinkhappy:

   • Embed this notice
     Michał "rysiek" Woźniak · 🇺🇦 (rysiek@mstdn.social)'s status on Tuesday, 05-Nov-2024 01:52:32 JST Michał "rysiek" Woźniak · 🇺🇦

     in reply to

     @phryk what I would suggest is to debug what gets served from cache and what does not.

     Also look at 4xx / 5xx errors, (micro) caching these is important as well!

     It might very well be that what brings your site down are 404 responses to /favicon.ico requestsor something just as unexpected.

   • Embed this notice
     Michał "rysiek" Woźniak · 🇺🇦 (rysiek@mstdn.social)'s status on Tuesday, 05-Nov-2024 01:55:20 JST Michał "rysiek" Woźniak · 🇺🇦

     in reply to

     @phryk if you're talking about the Cache-Control header sent to the browser, then no, that's not it.

     If you're talking about the Cache-Control header sent from the UWSGI back-end to the nginx edge, then I would suggest configuring your nginx edge to *ignore* this header when making decisions whether or not to (micro) cache any content.

     You want all your content (micro) cached on your nginx edge.

   • Embed this notice
     phryk 🏴 (phryk@mastodon.social)'s status on Tuesday, 05-Nov-2024 01:55:21 JST phryk 🏴

     in reply to

     • Michał "rysiek" Woźniak · 🇺🇦

     @rysiek Wait a second, I think I know what the problem is – the site doesn't yet set any Cache-Control headers.

     So things just basically fall through the cache and never actually interact with it – the entire load was handled by just that one poor UWSGI worker. 😂

   • Embed this notice
     phryk 🏴 (phryk@mastodon.social)'s status on Tuesday, 05-Nov-2024 01:57:44 JST phryk 🏴

     in reply to

     • Michał "rysiek" Woźniak · 🇺🇦

     @rysiek In poobrains I had staggered tiers of caching duration since ancient times and I've never had to touch it since writing that code, so I completely forgot that this is a thing you actually have to write code for on the application side. 😅

   • Embed this notice
     Michał "rysiek" Woźniak · 🇺🇦 (rysiek@mstdn.social)'s status on Tuesday, 05-Nov-2024 01:57:44 JST Michał "rysiek" Woźniak · 🇺🇦

     in reply to

     @phryk most of the time, unless you Really, Positively, Definitely Know What You're Doing And Know That Some Content Should Never Be Cached, Honestly For Realz.

     If it's a blog-type site, I would recommend configuring your nginx edge such that it always caches (in case of static content) or micro-caches (in case of dynamic content) everything, including error messages.

     Otherwise you are complicating your life for no benefit.

   • Embed this notice
     phryk 🏴 (phryk@mastodon.social)'s status on Tuesday, 05-Nov-2024 02:00:31 JST phryk 🏴

     in reply to

     • Michał "rysiek" Woźniak · 🇺🇦

     @rysiek Mhh, is this actually better? I'm aware that this header is often ignored, but from my experience that's more because many applications don't use reasonable values there.

     And since nginx' caching honors it by default, it can be used to ensure for example that /admin is never cached so I don't get forms only reacting half the time.

   • Embed this notice
     Michał "rysiek" Woźniak · 🇺🇦 (rysiek@mstdn.social)'s status on Tuesday, 05-Nov-2024 02:00:31 JST Michał "rysiek" Woźniak · 🇺🇦

     in reply to

     @phryk it's much better to make *explicit* exclusions in nginx config for /admin etc stuff, and for anything with cookies set, and obviously for any non-GET / non-HEAD request.

     Again, unless you *really* know what you're doing and you have a large list of always-changing endpoints that need to be dynamic.

     Which I am guessing here is not the case.