Oof, libsodium is waaaay too opinionated for this project. I don't think I can use it at all.
I love their approach though! The problem is that I'm doing something really similar and they're kind of incompatible with each other.
Libsodium's API seems to be a bunch of instances of: Oh, you're doing cryptography application X? (eg authenticated encryption), then you should use primitive Y. A bunch of experts got together to pick Y and they're great choices!
The problem with that is the best choice of Y changes over time. My library (Burger Identities) embraces that change and uses a kind of pluggable "backend" for each cryptography application. So today you might be using Y, but tomorrow the best answer might be Z. And Burger Identities gives you a way to migrate to the better primitives while still keeping compatibility with the people you're exchanging messages with.
So today I'm implementing a "backend" with RSA. Soon I'll get around to implementing one that uses ECC. And then later on, some kind of post-quantum implementation too. But that means I need to use something lower-level than libsodium.