Conversation

Notices

1. Embed this notice
   idea_enjoyer (idea_enjoyer@nicecrew.digital)'s status on Tuesday, 02-Apr-2024 05:25:17 JST idea_enjoyer

   • Matty
   @matty nicecrew has been acting sluggish, but other sites have not. Is there something going on that you know of?
   • Embed this notice
     reeeeeelman (realman543@annihilation.social)'s status on Tuesday, 02-Apr-2024 05:25:16 JST reeeeeelman

     in reply to

     • 
     • Matty
     @idea_enjoyer @matty
     Apparently @mint is having a similar issue. Maybe they are related? Check the logs.
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 02-Apr-2024 05:25:16 JST 

     in reply to

     • Matty
     • reeeeeelman
     @realman543 @idea_enjoyer @matty Matty claims "The instance isn't under a heavy load", which is the opposite of my postgres issues.
     ✙ dcc :pedomustdie: :phear_slackware: likes this.
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 02-Apr-2024 05:28:30 JST 

     in reply to

     • Matty
     • reeeeeelman
     @matty @realman543 @idea_enjoyer I've been struggling with long CPU usage spikes from postgres lately, that's about it.
   • Embed this notice
     Matty (matty@nicecrew.digital)'s status on Tuesday, 02-Apr-2024 05:28:31 JST Matty

     in reply to

     • 
     • reeeeeelman
     What?
   • Embed this notice
     Matty (matty@nicecrew.digital)'s status on Tuesday, 02-Apr-2024 05:30:43 JST Matty

     in reply to

     • 
     • Matty
     • reeeeeelman
     And when is the last time you repacked or vacuumed your database?
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 02-Apr-2024 05:30:43 JST 

     in reply to

     • Matty
     • reeeeeelman
     @matty @realman543 @idea_enjoyer Yes and yes. Repacked it as soon as the issues started.
   • Embed this notice
     Matty (matty@nicecrew.digital)'s status on Tuesday, 02-Apr-2024 05:30:44 JST Matty

     in reply to

     • 
     • reeeeeelman
     Have you tuned Postgre?
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 02-Apr-2024 05:34:19 JST 

     in reply to

     • Matty
     • reeeeeelman
     @matty @realman543 @idea_enjoyer No idea. Phoenix dashboard shows some long-ass query often running for up to 15 seconds, which I assume is a timeline.
   • Embed this notice
     Matty (matty@nicecrew.digital)'s status on Tuesday, 02-Apr-2024 05:34:20 JST Matty

     in reply to

     • 
     • reeeeeelman
     What Postgre job is causing the hang?
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 02-Apr-2024 05:37:20 JST 

     in reply to

     • Matty
     • reeeeeelman

     @matty @realman543 @idea_enjoyer SELECT a0."id", a0."data", a0."local", a0."actor", a0."recipients", a0."inserted_at", a0."updated_at", o1."id", o1."data", o1."inserted_at", o1."updated_at" FROM "activities" AS a0 INNER JOIN "objects" AS o1 ON (o1."data"->>'id') = associated_object_id(a0."data") INNER JOIN "users" AS u2 ON (a0."actor" = u2."ap_id") AND (u2."is_active" = TRUE) INNER JOIN "users" AS u3 ON (a0."actor" = u3."ap_id") AND (u3."invisible" = FALSE) WHERE ($1 && a0."recipients") AND (a0."actor" = $2) AND (a0."data"->>'type' = $3) AND (not(o1."data"->>'type' = 'Answer')) AND (not(o1."data"->>'type' = 'ChatMessage')) ORDER BY a0."id" desc nulls last LIMIT $4

     I never enabled activity expiration to begin with, so it's unrelated.

   • Embed this notice
     Matty (matty@nicecrew.digital)'s status on Tuesday, 02-Apr-2024 05:37:21 JST Matty

     in reply to

     • 
     • reeeeeelman
     Which query specifically? I'm asking because I had this same issue when I had activity expiration on like a year and a half ago.
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 02-Apr-2024 05:44:28 JST 

     in reply to

     • Matty
     • reeeeeelman
     @matty @realman543 @idea_enjoyer Randomly but frequently. Pretty sure it is timeline since more of the same queries spawn when I just spam the new tab with, say, home timeline.
   • Embed this notice
     Matty (matty@nicecrew.digital)'s status on Tuesday, 02-Apr-2024 05:44:29 JST Matty

     in reply to

     • 
     • reeeeeelman
     Is someone spamming chats on your instance? Does this happen periodically like on a cycle or just randomly?
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 02-Apr-2024 05:46:55 JST 

     in reply to

     • :archlinux:
     • Matty
     • reeeeeelman
     @lewdthewides @realman543 @idea_enjoyer @matty No unusual activity in logs.
   • Embed this notice
     :archlinux: (lewdthewides@hidamari.apartments)'s status on Tuesday, 02-Apr-2024 05:46:56 JST :archlinux:

     in reply to

     • 
     • Matty
     • reeeeeelman
     @mint @realman543 @idea_enjoyer @matty Are you being scraped?
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 02-Apr-2024 05:56:13 JST 

     in reply to

     • :archlinux:
     • Matty
     • reeeeeelman
     @lewdthewides @realman543 @idea_enjoyer @matty There were two IPs that might or might not be scrapers, but dropping them in firewall did literally nothing. Pretty much all other timeline requests are done by legitimate users.
   • Embed this notice
     :archlinux: (lewdthewides@hidamari.apartments)'s status on Tuesday, 02-Apr-2024 05:56:14 JST :archlinux:

     in reply to

     • 
     • Matty
     • reeeeeelman
     @mint @realman543 @idea_enjoyer @matty disable public viewing of timelines and profiles and see what happens :yui_shrug:
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 02-Apr-2024 05:58:41 JST 

     in reply to

     • :archlinux:
     • 
     • Matty
     • reeeeeelman
     @lewdthewides @idea_enjoyer @matty @realman543 On closer inspection, their since_id requests have the same delay between them, it's just the timing between requests from both of them happen to be synchronized. Red herring.
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 02-Apr-2024 06:09:59 JST 

     in reply to

     • :archlinux:
     • Matty
     • reeeeeelman
     @lewdthewides @realman543 @idea_enjoyer @matty Guess the disk is shitting itself then. Some lower numbers might be due to me setting CPU governor to powersave and not turning postgres off in meanwhile, though.
     fio.txt
   • Embed this notice
     :archlinux: (lewdthewides@hidamari.apartments)'s status on Tuesday, 02-Apr-2024 06:10:00 JST :archlinux:

     in reply to

     • 
     • Matty
     • reeeeeelman
     @mint @realman543 @idea_enjoyer @matty I tried. Only other thing I can think of is fio your database drive and check if random reads are where it's suppose to be
   • Embed this notice
     munir (munir@fedi.munir.tokyo)'s status on Tuesday, 02-Apr-2024 06:12:43 JST munir

     in reply to

     • 
     • Matty
     • reeeeeelman
     @mint @realman543 @idea_enjoyer @matty its hard to say because the the parameters are missing from the query, im skewing through the code right now but it's hard to find the query because they're using language extensions that simply get translated into a query, grrrrr i hate pleroma
      likes this.
   • Embed this notice
     :archlinux: (lewdthewides@hidamari.apartments)'s status on Tuesday, 02-Apr-2024 06:13:25 JST :archlinux:

     in reply to

     • 
     • Matty
     • reeeeeelman
     @mint @realman543 @idea_enjoyer @matty iops aren't terrible, but what good is that when the data isn't being fed in fast enough
      likes this.
   • Embed this notice
      (mint@ryona.agency)'s status on Tuesday, 02-Apr-2024 06:15:21 JST 

     in reply to

     • :archlinux:
     • Matty
     • reeeeeelman
     @lewdthewides @realman543 @idea_enjoyer @matty I will try scoring some better SSD one day, but guess I'd have to bear with it for now.