Conversation

Notices

1. Embed this notice
   ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Thursday, 16-Oct-2025 13:06:49 JST ✧✦Catherine✦✧

   quick question. Is this good

   • Embed this notice
     Jann Horn (jann@infosec.exchange)'s status on Friday, 17-Oct-2025 04:37:47 JST Jann Horn

     in reply to

     @whitequark are you serving an endpoint for "perform some expensive operation on the entire history of some repository" (like git log or git blame)? or what is that?

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Friday, 17-Oct-2025 04:37:47 JST ✧✦Catherine✦✧

     in reply to

     • Jann Horn

     @jann serving static pages from an s3 compatible service

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Friday, 17-Oct-2025 06:44:17 JST ✧✦Catherine✦✧

     in reply to

     here is a heatmap for the ~same period, and all graphs zoomed in at the latency spike

     (i'm a complete novice at performance engineering, if it's not clear yet)

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Friday, 17-Oct-2025 08:24:44 JST ✧✦Catherine✦✧

     in reply to

     • yopp

     @alex these are all good suggestions, thank you! I do not have detailed timings (only per-request timings, and for every GET to my frontend, there is a GET 404, GET 200, GET 200 _or_ GET 200, GET 200 from the backend)

   • Embed this notice
     yopp (alex@feed.yopp.me)'s status on Friday, 17-Oct-2025 08:24:46 JST yopp

     in reply to

     @whitequark hard to tell without info on how request time is measured, but overall doesn’t look good

     Can you trace detailed timings of s3 http query: dns, connect, tls, transfer? On top of that try to poke bucket with s3cmd and measure latencies from single host. Try to fetch same object at least 3 times and watch if latency drops after first GET. If so, it’s most likely related to pull of object from storage to POP in your region. If so, then GET of same object in different region will have same pattern. If latency is not consistent over long run then most likely they don’t have enough capacity to do “free egress” reliably

   • Embed this notice
     Drahflow (drahflow@infosec.exchange)'s status on Friday, 17-Oct-2025 08:33:36 JST Drahflow

     in reply to

     @whitequark As people noted, the data is a bit thin, maybe throw some k6, siege or ab traffic at it?
     But (assuming it's impossible to get better data):
     * You have some weird latency going on already between frontend and backend (_sometimes_ BE is <10ms, yet corresponding FE is >50ms). If it's the same machine, I'd check CPU and I/O load, if not, network overload between them?
     * During a latency spike there are _no_ fast requests. Likely they all blocked on the same resource. You could verify this by checking the request logs and see if they all completed at nearly the same time (but started differently).
     * If this is still static-from-S3, maybe consider https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html#optimizing-performance-guidelines-retry and just fire of every request to S3 a bunch of times and take the earliest answer.

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Friday, 17-Oct-2025 08:33:36 JST ✧✦Catherine✦✧

     in reply to

     • Drahflow

     @drahflow it should definitely be possible to get better data, I just need a tool/service that can do better than "hammering one URL" since that just gets cached instantly. I'm unfamiliar with this space.
     point by point response now:
     1. BE lives in a different datacenter in the same region (it is an S3-like service I pay Tigris for); I do not know the exact geographic locations or peering involved and I'm unfamiliar with tooling to look this up

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Friday, 17-Oct-2025 08:35:53 JST ✧✦Catherine✦✧

     in reply to

     • Drahflow

     @drahflow
     2. I should have request completion times in the traces (OTLM or something like it) we send to Sentry, I'll try to write a query to look it up! We unconditionally submit traces for every FE >=500ms request
     3. Yep, still static-from-S3, but I'm using a service called Tigris. I don't really know what their scale is, maybe we could try doing that--it's pretty cursed/wasteful though to have to do that...

   • Embed this notice
     Jann Horn (jann@infosec.exchange)'s status on Friday, 17-Oct-2025 08:40:21 JST Jann Horn

     in reply to

     @whitequark i have a stupid idea: what exactly are you measuring? if the client is not acknowledging TCP data for a while, does that create backpressure that causes blocking that counts towards that time measurement?

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Friday, 17-Oct-2025 08:40:21 JST ✧✦Catherine✦✧

     in reply to

     • Jann Horn

     @jann end to end times using a go s3 client library https://codeberg.org/git-pages/git-pages/src/commit/cfeb2d0dbe53773be1d9ccc911c6d4ab761c4d30/src/backend_s3.go#L223-L245

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Friday, 17-Oct-2025 09:18:21 JST ✧✦Catherine✦✧

     in reply to

     • Jann Horn
     • Paul Khuong

     @pkhuong @jann wow. til!

   • Embed this notice
     Paul Khuong (pkhuong@discuss.systems)'s status on Friday, 17-Oct-2025 09:18:22 JST Paul Khuong

     in reply to

     • Jann Horn

     @whitequark @jann p99.9 on real S3 is surprisingly trash. You need request hedging for reasonable tail latencies… but I have no idea if that even work on S3-compatible service of the day.

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Friday, 17-Oct-2025 19:41:16 JST ✧✦Catherine✦✧

     in reply to

     • yopp

     @alex git-pages, the "backend" is us measuring S3 timings

   • Embed this notice
     yopp (alex@feed.yopp.me)'s status on Friday, 17-Oct-2025 19:41:17 JST yopp

     in reply to

     @whitequark Did some quick poking and yeah, my bet is on “dynamic placement optimisation” being shit

     I see consistent 2-10x difference between first and consequential queries. But not as terrible as on your screenshots, more like 40 vs 150ms

     I wonder how latencies look like grouped by server. Is this git-pages or caddy metrics?