Janneke
If you run "guix pull" today, you get a package graph of more than 22,000 nodes rooted in a 357-byte program---something that had never been achieved, to our knowledge, since the birth of Unix: a Full-Source Bootstrap.
#GnuMes
#bootstrappable
#BootstrappableBuilds
#ReproducibleBuilds
@fsf
@fsfe
Janneke
@fsf @fsfe
And here's the blog post:
https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/
dave
@janneke holy shit I didn't know that this much progress had been made in bootstrapping. I thought we were still maybe years away from full source bootstrap. incredible work!
Janneke
@dthompson This is currently only i686-linux and x86_64-linux.
Work has been ongoing for ARMv7 (and AArch64) for quite some time now, but is stalled, probably until we have RISC-V.
dave
@janneke does this work for all architectures that guix supports or a subset?
Sergey Bugaev
@janneke not to underpaint the importance and coolness of this achievement, here's an uninformed question that you probably get a lot: how does this work wrt to depending on a Linux kernel (which is tons of C), some basic userland (or can it run as PID 1-and-only?), and x86 hardware (which... who knows what it does) to run this 357 byte binary?
If you can't trust a compiler to build your program correctly, why can you trust a kernel and some hardware to run your binary correctly?
Janneke
@bugaevc
Good question! Of course: you can't.
There is currently no good answer to that other than that we chose to start on getting rid of the obviously unnecessary and "easy" binary seeds first. Or: different people have different interests and competences, if we start then eventually we'll probably get there someday. There are some ideas, though.
The least elegant but easiest "solution" would be to revert to Diverse Double Compliing (DDC, https://dwheeler.com/trusting-trust/). The low level tools (stage0, m2-planet, and mes) can easily do cross builds. You could build on different architectures, and kernels if you like and compare package checksums.
We did something like this for Mes (all x86_64-linux, though) at the fifth reproducible builds conference (RB-V, https://guix.gnu.org/en/blog/2019/reproducible-builds-summit-5th-edition/)
Running as PID 1: During the same RB-V conference, Ludovic Courtès prototyped building a Guix package in the initial ramdisk. After the build the package is discarded, but before that its checksum is printed and can be checked with a build under GNU/Linux.
People have been working to build tiny kernels, such as: https://github.com/ironmeld/boot2now.
Also, Stage0 was designed to also run on the Knight VM, one could imagine running that on simpler hardware, or running the VM on different machines/architectures, dunno.
Ludovic Courtès
@bugaevc Speaking of the role of the kernel, an interesting question is how to implement isolated builds on the #Hurd—see “Isolated build environments” at https://guix.gnu.org/en/blog/2020/childhurds-and-substitutes/ for an overview.
I’m curious what you think of this!
Jonathan Frederickson
@civodul @bugaevc @janneke This now got me thinking... I found a Hurd post talking about how it adds POSIX compatibility to Mach: https://www.gnu.org/software/hurd/community/weblogs/ArneBab/technical-advantages-of-the-hurd.html
And it says it still provides access to the capability-based permissions underneath, which sounds nice. But it also got me thinking: there's likely to be a lot more software targeting WASI soon, which is natively capability-based. Could it be possible for Hurd to have WASI compatibility too?
Janneke
@joeyh
Thanks! Yeah, got to have some more hacking to look forward too ;)
see shy jo
@janneke great accomplishment
Needing a kernel is an important footnote though.
clacke
Ludovic Courtès
@janneke It took a while to get it merged, but now we can celebrate! 🎉
Also, great to see that #NLnet is funding the next steps, with @ekaitz_zarraga and others hard at work!
davidak
@janneke what an impressive and historic achievement!
I appreciate the efforts of all those involved. You have my greatest respect!
#NixOS is also implementing it: https://github.com/NixOS/nixpkgs/pull/227914
Janneke
@byterhymer @fsf @fsfe
For me personally (see the blog post) that would be: cleaning-up the FSB---we cut quite some corners---, getting rid of the ancient gcc-2.95.3 dependency (directly build gcc-4.6.4), getting Gash/Gash-Utils to run on top of Mes, and and RISC-V support, hopefully followed by ARM/AArch64.
But yeah, I really hope that others will address the hardware and kernel bits of the bootstrap!
グレェ「grey」
What next? Pairing something such as this with LiveHD (e.g. https://github.com/masc-ucsc/livehd) to bootstrap the "hardware" from source too?
NGI Zero open source funding
Wow congratulations @janneke that is truly impressive!
jonny (good kind)
@janneke
@hipsterelectron this seems like something up ur alley
Maltimore
What I don't understand: such a bootstrap must've been done at least once before in history, otherwise we wouldn't have any compiled programs, right? Why wasn't it possible to go the route of the previous bootstrap again? I'd appreciate a link to some further reading on this. 🙏
Janneke
@maltimore @fsf @fsfe
Some reasons for this include: The process wasn't documented, the code was lost, used many different hardwares, it took about 50 years, untangling history is _hard_.
For that last remark, just look at the Java or Rust bootstraps (they needed _many_, _many_ steps) or the sheer impossibility to bootstrap the NPM/Node distaster.
Andrius Štikonas
@janneke @maltimore @fsf @fsfe Also historical bootstrap might have used proprietary programs, so even historical bootstrap documentation might not help.
Janneke
@clacke
I'd like to think that I'm well funded which enablse me to do things that are important and fun.
I'd also like to think that all my time is my own; technically my current Hurd work isn't funded, but that's how Mes started off too, so yeah.
Andrius Štikonas
@janneke @aziz @fsf @fsfe @reproducible_builds @ekaitz_zarraga Indeed! Right now we can bootstrap all the way from #hex0 to #Mes, then use #MesCC to build very first build of #tinycc (we can call it mes-tcc). mes-tcc can then build the next build of tinycc (boot0-tcc). Unfortunately, at the moment boot0-tcc segfaults. Today, I fixed one crash which was due to Global Offset Table being all zeros but it turns out we are now hitting another segfault, so more work is needed.
Janneke
@aziz @fsf @fsfe @reproducible_builds
Thank you! We (well, mostly @ekaitz_zarraga and @stikonas) are getting very close to bootstrapping tinycc on riscv64. Exciting times ahead!
Amirouche
@janneke @fsf @fsfe @reproducible_builds very inspiring very happy best wishes
Janneke
@stikonas @maltimore @fsf @fsfe
When I said "source code was lost" I didn't even think of this, but you're right: GNU's not Unix!
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.