Conversation

Notices

1. Embed this notice
   ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 00:25:21 JST ✧✦Catherine✦✧

   mildly upsetting compiler content

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 00:34:25 JST ✧✦Catherine✦✧

     in reply to

     additional mildly upsetting compiler content

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 00:35:01 JST ✧✦Catherine✦✧

     in reply to

     • medium rare bird

     @migratory Yes, precisely

   • Embed this notice
     medium rare bird (migratory@jorts.horse)'s status on Monday, 27-Oct-2025 00:35:03 JST medium rare bird

     in reply to

     @whitequark to me this is beautiful "plug X into Y" like the Free Universal Construction Kit

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 00:36:42 JST ✧✦Catherine✦✧

     in reply to

     basically the idea is that i'm going to turn every process spawn of clang (within this build tree) into an HTTP request to a predefined endpoint, which then gets submitted to an auto-scaling HTTP service. which makes LLVM builds faster

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 00:38:33 JST ✧✦Catherine✦✧

     in reply to

     you should be able to build webkit like this too

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 01:57:28 JST ✧✦Catherine✦✧

     in reply to

     ok check this out

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 01:58:10 JST ✧✦Catherine✦✧

     in reply to

     now i just need to scale it to 1000 machines and i can build llvm Very Quickly. lets go

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 02:28:52 JST ✧✦Catherine✦✧

     in reply to

     turns out, and this isn't a massive surprise, that my naive i-scribbled-it-in-an-evening solution is _hopelessly_ bottlenecked by the network. like, "average of 20 Mbps * N per -jN" level of bottlenecked

     we aren't about to launch 50 concurrent builds when 5 saturate my uplink haha

   • Embed this notice
     Ted Mielczarek (tedmielczarek@mastodon.social)'s status on Monday, 27-Oct-2025 02:29:01 JST Ted Mielczarek

     in reply to

     @whitequark I think you are reinventing Bazel's remote execution API: https://github.com/bazelbuild/remote-apis

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 02:31:42 JST ✧✦Catherine✦✧

     in reply to

     • Ted Mielczarek

     @tedmielczarek this is exactly what i'm doing, yes.

   • Embed this notice
     Ted Mielczarek (tedmielczarek@mastodon.social)'s status on Monday, 27-Oct-2025 02:33:18 JST Ted Mielczarek

     in reply to

     @whitequark are there common things you're uploading for each build? A lot of distributed compilation systems wind up benefiting from an artifact cache where you upload things keyed by a content hash, and then just give the remote build process a list of hashes for it to fetch from the cache.

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 02:36:13 JST ✧✦Catherine✦✧

     in reply to

     • Ted Mielczarek

     @tedmielczarek "every header llvm can possibly include", yes.

     this is an obvious design choice that i've considered, but decided to implement the naive version first to see how bad it is (very)

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 02:36:48 JST ✧✦Catherine✦✧

     in reply to

     i'm not feeling like finishing it today so i'll probably do it some other day

     bazel has a remote execution API which implements basically the same architecture.

     however, i will not be using it because: bazel does not spark joy

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 02:37:28 JST ✧✦Catherine✦✧

     in reply to

     • Billy O'Neal

     @malwareminigun classic "the proper way to build this software is to get a job at google but some alternative workarounds are available if that's not feasible" notice in this repo

   • Embed this notice
     Billy O'Neal (malwareminigun@infosec.exchange)'s status on Monday, 27-Oct-2025 02:37:29 JST Billy O'Neal

     in reply to

     @whitequark *gestures to https://chromium.googlesource.com/infra/goma/client/+/6bae3f8023bfb7b861b2c0b1e12aa0be55f0f829/README.md *

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 02:42:51 JST ✧✦Catherine✦✧

     in reply to

     the other reason not to reuse what bazel does is because i want the things i'm doing to not be like "only the exclusive club of software that someone applied the google build system to gets to enjoy the benefit"

     architecturally, it is not necessary to envelope everything in an all-consuming blaze. you can just use cmake (if you are able to parse a compiler cmdline, or approximate it). and most people do not use, or want to use, or benefit from, the google build system

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 02:43:05 JST ✧✦Catherine✦✧

     in reply to

     plus if it uses wasm for isolation then suddenly running a public instance becomes a viable and reasonable thing to do, which nobody in their right mind will do with bazel/goma/whatever

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 02:44:48 JST ✧✦Catherine✦✧

     in reply to

     (i could, technically, conceivably reuse the _protocol_ while bolting my own software on both the sending and the receiving side. the protocol is basically good. however, it's large and complex, and like i mentioned, no part of bazel sparks joy. so why would i do this to myself)

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 03:24:47 JST ✧✦Catherine✦✧

     in reply to

     • Rogan Dawes

     @RoganDawes ccache doesn't help when i actually need to rebuild stuff. i already use it and all it does is make me neurotic about touching anything that triggers a full rebuild, like an otherwise irrelevant cflags change...

   • Embed this notice
     Rogan Dawes (rogandawes@infosec.exchange)'s status on Monday, 27-Oct-2025 03:24:48 JST Rogan Dawes

     in reply to

     @whitequark is ccache not an option for some reason?

   • Embed this notice
     mhoye (mhoye@mastodon.social)'s status on Monday, 27-Oct-2025 03:30:02 JST mhoye

     in reply to

     @whitequark Have you considered leveraging 304-not-modified to speed up incremental compilation?

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 03:39:38 JST ✧✦Catherine✦✧

     in reply to

     • mhoye

     @mhoye ha, that's clever and is also something that will naturally fall out of the architecture i have in mind. yup, would work ^^

   • Embed this notice
     mhoye (mhoye@mastodon.social)'s status on Monday, 27-Oct-2025 03:42:10 JST mhoye

     in reply to

     @whitequark A friend of mine once described me as "occasionally useful, in the same way an occasional table is a table."

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 03:44:07 JST ✧✦Catherine✦✧

     in reply to

     • mhoye

     @mhoye so basically the architecture i have [partly impl'd, partly in mind] right now speculatively looks at everything path-like in the command line, then uploads all of it, runs the command on the server, then downloads all of it back (sped up by caching/dedup)

     naturally if you have `-o foo.o` its hash will get uploaded, and then the server can return 304

   • Embed this notice
     Ted Mielczarek (tedmielczarek@mastodon.social)'s status on Monday, 27-Oct-2025 03:52:35 JST Ted Mielczarek

     in reply to

     @whitequark working with Google "open source" projects rarely sparks joy.

   • Embed this notice
     mhoye (mhoye@mastodon.social)'s status on Monday, 27-Oct-2025 04:06:59 JST mhoye

     in reply to

     @whitequark If you preload the remote with an rsync-type process, maybe you skip the "send everything over" part, or at least do it a bit at a time as you work?

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 04:18:43 JST ✧✦Catherine✦✧

     in reply to

     • mhoye

     @mhoye I think you have to send All Of The Headers at least once, but then you can shove them into S3 or something and basically never have to send them again ever (because the checksums match)

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 04:25:55 JST ✧✦Catherine✦✧

     in reply to

     • mhoye

     @mhoye (you have to do it At Least Once because i do not know upfront which files will be accessed by the compiler, and i'm consciously choosing to use an architecture which does not easily allow me to do so)

   • Embed this notice
     mhoye (mhoye@mastodon.social)'s status on Monday, 27-Oct-2025 04:45:32 JST mhoye

     in reply to

     @whitequark Yeah, I'm just thinking that whatever can be made incremental should be. The first bite at the problem will have to be a big one.

   • Embed this notice
     David Chisnall (*Now with 50% more sarcasm!*) (david_chisnall@infosec.exchange)'s status on Monday, 27-Oct-2025 21:07:49 JST David Chisnall (*Now with 50% more sarcasm!*)

     in reply to

     @whitequark Someone did this a few years back. They wrote an AWS Lambda that was clang. You needed to do the preprocessing locally, which limited the scaling.

     The thing I pondered after seeing it was to make the local clone a file share that I could mount over CIFS and also mount in the function and make the build directory a similar share where only the final build products needed to be exported, so I could run the compiler and linker steps entirely as cloud functions. I didn’t every get around to implementing it, so I didn’t have a solution to things like TableGen.

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 27-Oct-2025 21:07:49 JST ✧✦Catherine✦✧

     in reply to

     • David Chisnall (*Now with 50% more sarcasm!*)

     @david_chisnall yeah, I want to solve all of that with Wasm and I think I know exactly how.

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Wednesday, 29-Oct-2025 07:29:35 JST ✧✦Catherine✦✧

     in reply to

     • Demi Marie Obenour

     @alwayscurious no software is completely bug-free but this particular risk becomes negligible compared to the more general issue of "do i trust someone else to compile things for me?"

   • Embed this notice
     Demi Marie Obenour (alwayscurious@infosec.exchange)'s status on Wednesday, 29-Oct-2025 07:29:36 JST Demi Marie Obenour

     in reply to

     @whitequark Using a compiler compiled to WebAssembly? WebAssembly runtimes do have vulnerabilities some of the time. Microsoft wrote a research paper about running WebAssembly modules in VMs.

   • Embed this notice
     ✧✦Catherine✦✧ (whitequark@mastodon.social)'s status on Monday, 03-Nov-2025 00:32:45 JST ✧✦Catherine✦✧

     in reply to

     • malte :yayblob:

     @malte I'm sure I would have enjoyed using it if I was a Google employee. I am, however, not

   • Embed this notice
     malte :yayblob: (malte@chaos.social)'s status on Monday, 03-Nov-2025 00:32:46 JST malte :yayblob:

     in reply to

     @whitequark I'm clearly biased here, but the protocol is already used by other clients, like rec, BuildStream, Buck2. There's also a diverse ecosystem of backends, like BuildBuddy, Buildbarn, NativeLink, Buildfarm.
     I understand why people don't like Bazel, but the protocol is solid.