Conversation

Notices

1. Embed this notice
   Thomas Steen Rasmussen (tykling@mastodon.social)'s status on Thursday, 16-Nov-2023 02:11:41 JST Thomas Steen Rasmussen

   I was investigating an MITM today where the attacker was using a real LetsEncrypt certificate. This was possible because an NS record domain expired and was re-registered by the attacker

   A few weeks ago there was also the jabber.ru MITM issue where a valid LE cert had also been issued.

   Both of these attacks could have been avoided by using CAA account pinning.

   You should add this on all domains today!

   For an example see the CAA record for bornhack.dk https://caatest.co.uk/bornhack.dk

   Spread the word!

   • Embed this notice
     feld (feld@bikeshed.party)'s status on Thursday, 16-Nov-2023 02:11:38 JST feld

     in reply to
     @tykling What's ridiculous about this is that Caddy, by default, will randomly get a certificate from LetsEncrypt or ZeroSSL and doesn't guarantee that the certificate will be renewed from the same issuer
     
     So if you want CAA to not break your site in the future you need to know you must deploy extra configuration to pin to one of these two providers... *sigh*
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Thursday, 16-Nov-2023 02:13:57 JST feld

     in reply to
     @tykling it just means an attacker only has to look and see your CAA supports both but the current certificate is from LetsEncrypt, so they can repeat this attack by asking ZeroSSL to issue the cert
     
     at least for your described scenario
   • Embed this notice
     Thomas Steen Rasmussen (tykling@mastodon.social)'s status on Thursday, 16-Nov-2023 02:13:58 JST Thomas Steen Rasmussen

     in reply to

     • feld

     @feld you can pin both, no problem with that

   • Embed this notice
     feld (feld@bikeshed.party)'s status on Thursday, 16-Nov-2023 02:16:02 JST feld

     in reply to

     • feld
     @tykling actually it wouldn't really matter anyway. Both ZeroSSL and LetsEncrypt allow you to issue certs, throw away your account keys, and issue those same certs again under a new account
     
     The certificates are not in any way protected by your account keys
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Thursday, 16-Nov-2023 02:18:58 JST feld

     in reply to
     @tykling How? I don't see anywhere in the CAA specs that demonstrates this?
     
     What does as CAA record look like that pins it to your ACME private keys?
     
     https://datatracker.ietf.org/doc/html/rfc6844
   • Embed this notice
     Thomas Steen Rasmussen (tykling@mastodon.social)'s status on Thursday, 16-Nov-2023 02:18:59 JST Thomas Steen Rasmussen

     in reply to

     • feld

     @feld that is what the account pinning prevents. The attacker doesn't control the private keys for the ACME account pinned in the CAA record. ZeroSSL would refuse to issue because the attacker is using a different account.

   • Embed this notice
     feld (feld@bikeshed.party)'s status on Thursday, 16-Nov-2023 02:22:09 JST feld

     in reply to

     • feld
     @tykling ahh my eyes went right past it
     
     > A certificate issuer MAY specify additional parameters that allow
     customers to specify additional parameters governing certificate
     issuance. This might be the Certificate Policy under which the
     certificate is to be issued, the authentication process to be used
     might be specified, or an account number specified by the CA to
     enable these parameters to be retrieved.
     
     > $ORIGIN example.com CAA 0 issue "ca.example.net; account=230123"
     
     
     Do both LetsEncrypt and ZeroSSL honor this account parameter? I've never seen this "account" parameter used in the wild, but I don't look at CAs regularly
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Thursday, 16-Nov-2023 02:24:56 JST feld

     in reply to

     • feld
     @tykling Continuing this line of thought:
     
     your original scenario had the attacker control an NS. If they can control an NS they control whether or not it serves CAA records.
     
     Does LetsEncrypt and ZeroSSL ensure responses from *ALL* NSes are identical before proceeding?
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Thursday, 16-Nov-2023 02:32:38 JST feld

     in reply to

     • feld
     @tykling The only way around this is to require the CAA record not be served from the NS but is a Glue record instead
     
     I think this is a major security hole everyone is ignoring. It's much harder for an attacker to tamper wither your Glue records
   • Embed this notice
     Thomas Steen Rasmussen (tykling@mastodon.social)'s status on Thursday, 16-Nov-2023 02:37:56 JST Thomas Steen Rasmussen

     in reply to

     • feld

     @feld I believe LE currently does DNS checks from multiple AWS regions + from their own servers. So they likely would have gotten inconsistent answers and bailed out at that point. I don't know about ZeroSSL.

     The attackers likely had to try issuing multiple times to get lucky and have all the lookups hit the "bad" server.

     No guarantees here, but I would much, much rather have had CAA account pinning in place than not during this attack.

   • Embed this notice
     feld (feld@bikeshed.party)'s status on Thursday, 16-Nov-2023 02:37:56 JST feld

     in reply to
     @tykling
     
     > The attackers likely had to try issuing multiple times to get lucky and have all the lookups hit the "bad" server.
     
     They can't be checking every NS then; they check a random one and I'm not sure why they'd need to do it from multiple regions (other than working around networking issues).
     
     So that's the whole attack scenario. Pwn a single NS and then try a couple times until it picks the NS you control.
     
     Very disappointing.
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Thursday, 16-Nov-2023 02:45:16 JST feld

     in reply to

     • feld
     @tykling every acme tool should strongly suggest you do DNS validation and then it can automatically add CAA records for you as it already has all the info required to do it
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Thursday, 16-Nov-2023 03:39:39 JST feld

     in reply to
     @tykling No, it's not a CNAME, you want a NS for situations like AWS/Route53
     
     new zone: _acme-challenge.foo.com.
     
     In the parent zone make an NS for:
     
     _acme-challenge.foo.com.
     
     This delegates requests to your new sub-zone. And give the server a key that can only modify records in that subzone.
     
     (if you are using BIND and maybe others you don't need to do this, IIRC you can be more granular without needing to make an entirely new zone)
   • Embed this notice
     Thomas Steen Rasmussen (tykling@mastodon.social)'s status on Thursday, 16-Nov-2023 03:39:40 JST Thomas Steen Rasmussen

     in reply to

     • feld

     @feld I agree that DNS-01 should be pushed, and pinned. ACME tools should check for CAA records and recommend they be added.

     But IMO you should never let any tool on an internet-facing server edit your zone directly, or an attacker compromising the server can also edit your zone.

     Instead you should make CNAMEs for the _acme-challenge records to a dedicated subzone which is used exclusively for ACME challenges. This has all the advantages of DNS-01, but doesn't hand over control of your zone :)

   • Embed this notice
     feld (feld@bikeshed.party)'s status on Thursday, 16-Nov-2023 03:56:32 JST feld

     in reply to

     • feld

     @tykling The main problem I have with DNS-01 though is that it's slow as fuck so if you have a distributed cluster of webservers and need to dynamically issue certificates you can just cheat by allowing any server automatically respond to any HTTP-01 challenge without them needing to be aware a challenge was issued. This setup assumes you have a method for servers to automatically find certificates on some kind of shared storage.

     Here's my Caddy solution for it:

     (acme_standalone) { # ACME http-01 challenge sends a request to /.well-known/acme-challenge/TOKEN # and expects the server to have created a file at that location with the contents # being a string of TOKEN.THUMBPRINT to validate the challenge. # # We are operating in a multi-node cluster and really want to do the http-01 style # ACME validation, so we can use this regex trick to forge a valid response to any possible # challenge from any node that receives the request. This is much faster than dns-01 challenges. # # The thumbprint is derived from the account key that the ACME client has generated at first run. # Our account name is our email defined at the top of this config. # With Caddy this key file would normally be located at: # $THE_CADDY_DATA_DIR/acme/acme-v02.api.letsencrypt.org-directory/users/USERNAME/USERNAME.key # # The same key is used for all providers, so this is compatible with LetsEncrypt, ZeroSSL, etc. # # We are using an extension for Caddy that stores the Caddy data in an S3 bucket, so the key # can be found there and you can use a script in the ops git repo to get the thumbprint. # # Due to the fact that we're storing the production Caddy data in S3 it is unlikely you will # need to generate a new thumbprint, but that is how you would retrace these steps. @achallenge { path_regexp ch ^/\.well-known/acme-challenge/([-_a-zA-Z0-9]+)$ } respond @achallenge "{re.ch.1}.REDACTED_THUMBPRINT_HERE" }
   • Embed this notice
     Thomas Steen Rasmussen (tykling@mastodon.social)'s status on Thursday, 16-Nov-2023 05:34:21 JST Thomas Steen Rasmussen

     in reply to

     • feld

     @feld maybe you want an NS record, I want a CNAME. Making a delegation and seperate zone for every challenge sounds very ineffective to me, but whatever works for you.

     A CNAME does exactly what is needed, and as a bonus it can carry the name being challenged in the CNAME target (below the challenge zone), so:

     _acme-challenge.www.example.com CNAME www.example.com.acme.example.org

     Anyway, we clearly have different views on this, which is fine. I am not on bikeshed.party, so I will stop here :)

     feld likes this.