Conversation

Notices

1. Embed this notice
   Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Sunday, 22-Sep-2024 09:24:02 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

   You ever just add debug logs to the #Python standard library because it doesn’t have logs where they’re needed? :blobfoxgooglycry:

   (I’m working with a heavily multi-processed program so pdb won’t do shit.)

   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Sunday, 22-Sep-2024 09:26:55 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to
     And adding those logs fixed where it was previously getting stuck. :blobfoxdeadinside: #Python #Heisenbug
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Sunday, 22-Sep-2024 09:32:02 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to
     Maybe it's time to rewrite this shit in #Kotlin so that I don't need to deal with IPC. :blobfoxdead:
     Sick Sun likes this.
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Sunday, 22-Sep-2024 10:05:03 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     I think I can fix it by using a Unix datagram socket to forward the logs to the logger thread, rather than trying to get multiprocessing.Queue to be inherited by child processes and their child processes. When I got rid of the latter logging approach, it stopped getting stuck. :blobfoxgooglytrash:

     All of this because of the #GIL. :blobfoxangrylaugh: #Python

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     mr64bit (mr64bit@p.mr64.net)'s status on Sunday, 22-Sep-2024 10:13:58 JST mr64bit

     in reply to
     @tk in essence those shouldn't really be too different. sounds like a problem with forking a process with locked mutexes rather than the GIL
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: likes this.
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Sunday, 22-Sep-2024 10:18:30 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     • mr64bit

     @mr64bit I’m using the spawn context, though.

   • Embed this notice
     mr64bit (mr64bit@p.mr64.net)'s status on Sunday, 22-Sep-2024 10:31:36 JST mr64bit

     in reply to
     @tk huh, weird. i wonder if it's something to do with multiple levels of spawned children? don't know if i've ever had to do that before, but seems like something that should be supported without issue.
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: likes this.
   • Embed this notice
     mr64bit (mr64bit@p.mr64.net)'s status on Sunday, 22-Sep-2024 10:32:03 JST mr64bit

     in reply to

     • mr64bit
     @tk the multiprocessing library is a bunch of super cursed code though, (hard to believe it's in the standard library tbh) so maybe weirdness is to be expected.
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: likes this.
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Sunday, 22-Sep-2024 10:32:34 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     • mr64bit
     @mr64bit Yeah, it's definitely cursed. :blobfox0_0:
     
     Hopefully the eventually GIL-ectomy will remove some of the need. :blobfoxgoogly:
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Monday, 23-Sep-2024 10:05:59 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     • mr64bit

     It ended up working perfectly. :blobfoxthumbsup: #Python

   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Monday, 23-Sep-2024 12:21:21 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     Never, ever, use the fork context in multiprocessing. :blobfoxsweating: POSIX OSes (aside from macOS) have that as the default. :blobfoxangrylaugh: Always use the spawn context! :blobfoxdead:

     Of course, unless you have an extreme edge case. :blobfoxgoogly: (Yes, the names are funny. :blobfoxgooglytrash: )

     #Python #multiprocessing

   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Monday, 23-Sep-2024 12:26:41 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     I feel so dead inside after learning that the hard way. :blobfoxdeadinside:

     Signs:

     1. There are far more processes than makes sense when using ProcessPoolExecutors. :blobfoxscared:
     2. Files are happily corrupted. :blobfoxupsidedowndizzy: That even includes the stuff generated by faulthandler! :blobfoxdead:
     3. Possibly really weird deadlocks. :blobfoxthonking:
     4. Your life is irreversibly altered. :blobfoxthumbsup: :blobfoxsweating:
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Monday, 23-Sep-2024 12:34:48 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     Here are some faulthandler outputs. I couldn’t make heads or tails of the first one (fork), and went so far as to write my own home-grown version of it, which suffered the same fate anyway! :blobfoxupsidedowndizzy: The second one looks like it makes sense, so now I can actually track down what I think is a deadlock. :blobfoxthonking: :blobfoxthumbsup:

   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Monday, 23-Sep-2024 12:35:18 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     • Sick Sun
     @sun @mikoto ^^^
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Monday, 23-Sep-2024 12:39:04 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     • glitter
     :blobfoxcomfycomputer: I figured out all of this shit while stoned at the end of a fun time out with @glitter , which included lunch at Red Lobster with a reasonable enough glass of black tea. :blobfoxcomfytea:
     
     Thinking I might get away from my soda habit on most days via black #tea instead. I've tried all kinds of tea over the years, but my tastes have changed a lot since then. :blobfoxgoogly:
     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Haelwenn /элвэн/ :triskell: (lanodan@queer.hacktivis.me)'s status on Monday, 23-Sep-2024 13:03:46 JST Haelwenn /элвэн/ :triskell:

     in reply to
     @tk GIL always makes me wonder how Python ended up so popular for writing applications and daemons.
     You'd think it would be stuck to things like scripts and simple utilities.
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: likes this.
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Tuesday, 24-Sep-2024 03:06:56 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     You ever just write your own scheduling algorithm? :blobfoxgoogly: #Python #programming

   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Tuesday, 24-Sep-2024 03:08:05 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to
     Am I a real programmer now? :blobfoxcute:
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Tuesday, 24-Sep-2024 06:17:18 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     I think I might be running into some queue blocking in concurrent.futures.submit(), but there’s no way to change the behavior. :blobfoxgooglycry: #Python

   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Tuesday, 24-Sep-2024 09:03:30 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     What an adventure.

     :blobfoxgoogly: (:blobfoxdeadinside:) #Python

   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Tuesday, 24-Sep-2024 09:04:02 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     • Sick Sun
     @mikoto @sun ^^^
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Tuesday, 24-Sep-2024 23:13:23 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to
     1. Learned the hard way that the (default on most POSIX) fork context is bad news. :blobfoxsweating:
     2. Wrote a Unix domain datagram based log infrastructure. :blobfoxthumbsup:
     3. Wrote an algorithm that operates kind of like concurrent.futures.as_completed(), except it has a priority queue and doesn’t eagerly load the list of futures. :blobfoxgoogly:
     4. Discovered that it’s possible to overload concurrent.futures.ProcessPoolExecutor with futures, preventing any actual background processing after a point. :blobfox0_0:
     5. Got TONS of practice optimizing stuff for large datasets.
     6. Learned that taking breaks is important for reasons than most people are aware of. :blobfoxcomfysip:
     7. My life was permanently altered by this project. I basically nerd-sniped myself. :blobfoxdeadinside:

     #Python

     Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Tuesday, 24-Sep-2024 23:54:05 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to
     This whole mess got me thinking seriously about learning #Kotlin, since it would eliminate needing to use IPC to make use of multiple CPU cores. (Yes, I actually like the #JVM, but don’t want to boilerplate myself to death with #Java. :blobfoxgooglymlem: )
     
     I’ll probably still look into learning Kotlin, but not for this project. :blobcatshrug:
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Wednesday, 25-Sep-2024 02:30:36 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to
     I love Unix domain sockets. :blobfoxcomfycomputer:
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Wednesday, 25-Sep-2024 06:07:13 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     • Sick Sun

     :blobfoxsweating:

     [2024-09-24 14:03:55,126] [INFO] [SpawnProcess-2:4(32637)] [MainThread] [_digest.py:66] [log_digest] [/]: shitposter.club.tar.zstd (475.64 GB) = 0079ff5ea1250caba488c4056717b74a299fafd67c8e220508aff222cdd32721113c2020f44f9fe7b20f395430cc73c5f0aa5170f5fd5a94144ce01e769b1929 (0:34:02.450908)

     @sun

   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Wednesday, 09-Oct-2024 04:58:20 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to

     :blobfoxowo: #Kotlin

     public fun getDigest(group: String, file: String): String? { return executePreparedStatement(SQL_QUERIES.q("get-digest")) { statement -> statement.setString(1, group) statement.setString(2, file) statement.executeQuery().use { resultSet -> if (!resultSet.next()) throw NoSuchElementException("No digest for group=${group}, file=${file}") resultSet.getString("digest") } } }
   • Embed this notice
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: (tk@bbs.kawa-kun.com)'s status on Wednesday, 09-Oct-2024 05:01:09 JST Doughnut Lollipop 【記録係】:blobfoxgooglymlem:

     in reply to
     This is so much more enjoyable than #Java, and I feel like I can trust the #JVM more than I can trust #Python. :blobfoxgoogly: