Just to be clear: this is a publicity stunt.
If there is a working design for a nuclear weapon on the internet, then LLMs will plagiarize it.
If there isn’t, they’re not going to come up with one.
This is just cheap journalist bait. Every fantastical headline-grabbing “OMG our imitation machine is too dangerous for humanity!!!! (attn: investors)” press release like this is a play for headlines.