Conversation

Notices

1. Embed this notice
   anime graf mays ?️? (graf@poa.st)'s status on Friday, 06-Sep-2024 22:24:45 JST anime graf mays ?️?

   • Fish of Rage
   @sun how many federator_incoming oban jobs do you have for misskey?
   • Embed this notice
     Fish of Rage (sun@shitposter.world)'s status on Friday, 06-Sep-2024 22:24:44 JST Fish of Rage

     in reply to
     @graf I reject misskey, will that make a difference?
   • Embed this notice
     Fish of Rage (sun@shitposter.world)'s status on Friday, 06-Sep-2024 22:25:13 JST Fish of Rage

     in reply to

     • Fish of Rage
     @graf wait I mean misskey.io. what are we looking for?
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 06-Sep-2024 22:25:43 JST feld

     in reply to

     • ZOP
     • Fish of Rage
     @sun @graf
     
     @zop was saying in IRC he got flooded with thousands and thousands of jobs from Misskey
   • Embed this notice
     Fish of Rage (sun@shitposter.world)'s status on Friday, 06-Sep-2024 22:26:32 JST Fish of Rage

     in reply to

     • feld
     @graf @feld still confused are we talking about any misskey or just misskey.io
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Friday, 06-Sep-2024 22:26:33 JST anime graf mays ?️?

     in reply to

     • ZOP
     • Fish of Rage
     • feld
     @feld @zop @sun yeah definitely. poast is blocked by misskey.io afaik and we don't have an issue, but my private instance has the same bug with incoming oban jobs from misskey as vd15 yesterday afternoon . 93 million oban jobs for a handful of misskey users
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 06-Sep-2024 22:26:36 JST feld

     in reply to

     • ZOP
     • Fish of Rage
     @graf @zop @sun
     
     > 93 million oban jobs
     
     haha no way!! whaaaaaaat????
   • Embed this notice
     Fish of Rage (sun@shitposter.world)'s status on Friday, 06-Sep-2024 22:27:27 JST Fish of Rage

     in reply to

     • feld
     @graf @feld I have under 2,000 oban job rows
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Friday, 06-Sep-2024 22:27:28 JST anime graf mays ?️?

     in reply to

     • ZOP
     • Fish of Rage
     • feld
     @feld @zop @sun it filled syslog (130+ GB) and pleroma locked up out of disk space
     
     pastebin.com/rQjtzF9b
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 06-Sep-2024 22:28:08 JST feld

     in reply to

     • ZOP
     • Fish of Rage
     @graf @zop @sun how is that even possible? How do they successfully deliver that may messages to Pleroma's inbox in such a short period of time?
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Friday, 06-Sep-2024 22:29:57 JST anime graf mays ?️?

     in reply to

     • ZOP
     • Fish of Rage
     • feld
     @feld @zop @sun I wasn't paying attention to that instance for maybe a day or two, but pleroma seems to be trying to reinsert several times per second per user and per post
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 06-Sep-2024 22:29:57 JST feld

     in reply to

     • ZOP
     • Fish of Rage
     @graf @zop @sun we need a new release with some of my fixes in it to discard some of these jobs after certain failures 😭
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 06-Sep-2024 22:33:54 JST feld

     in reply to

     • ZOP
     • Fish of Rage
     @graf @zop @sun since it's only RemoteFetcher it's perfectly harmless to delete everything in the table for that Worker, but make sure you stop Pleroma when you do it.
     
     Time to add more safeguards against this.
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Friday, 06-Sep-2024 22:33:55 JST anime graf mays ?️?

     in reply to

     • ZOP
     • Fish of Rage
     • feld
     @feld @zop @sun here's a cross section, it's literally the same handful of posts by a handful of users lmfao
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 06-Sep-2024 22:48:03 JST feld

     in reply to

     • skylar :confederateflag:??? :z:
     • ZOP
     • Fish of Rage
     @graf @zop @skylar @sun
     
     here's a fix that would have at least stopped the duplicates from being accumulated into the queue
     
     https://git.pleroma.social/pleroma/pleroma/-/merge_requests/4253
   • Embed this notice
     anime graf mays ?️? (graf@poa.st)'s status on Friday, 06-Sep-2024 22:48:04 JST anime graf mays ?️?

     in reply to

     • skylar :confederateflag:??? :z:
     • ZOP
     • Fish of Rage
     • feld
     @skylar @zop @feld @sun i imagine this is going to break a lot of single user instances if they come across these posts tbh
   • Embed this notice
     skylar :confederateflag:??? :z: (skylar@misskey.yandere.love)'s status on Friday, 06-Sep-2024 22:48:05 JST skylar :confederateflag:??? :z:

     in reply to

     • ZOP
     • Fish of Rage
     • feld
     @graf @zop @feld @sun my instance was doing the same thing the other day
     misskey machine broke?
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Saturday, 07-Sep-2024 00:08:15 JST feld

     in reply to

     • ZOP
     • Fish of Rage
     @zop @sun dumping RemoteFetcher is always safe, it's just a side effect of processing a thread or fetching users encountered in new posts etc. If you kill it and load the target thread in the future it should generate the required jobs to reconstruct it backfill like it was trying to originally
   • Embed this notice
     ZOP (zop@7td.org)'s status on Saturday, 07-Sep-2024 00:08:16 JST ZOP

     in reply to

     • Fish of Rage
     • feld
     @feld @A3NmXoRtC3rBTN23nM.graf@poa.st @sun Yeah I had...heck 14M, 15M RemoteFetcherWorker's queued. It...looked more like each job created a new job and the original job stayed, but, idk, I didn't have any time to investigate. Couldn't really human read the logs because they were so high volume (milliseconds between entries) so had to turn logging totally off. Didn't know if dumping those jobs was "safe" or not.
   • Embed this notice
     PalePimp (palepimp@poa.st)'s status on Saturday, 07-Sep-2024 04:27:33 JST PalePimp

     in reply to

     • ZOP
     • Fish of Rage
     • feld
     @graf @feld @zop @sun Learned the hard way years ago that /home and /var/log go into their own separate volumes on production boxes.
     
     /var/log /dev/mapper/vg_system-lv_log
     Fish of Rage likes this.