it is, but it's an interesting thought. if these models and architectures are good at detecting patterns, they may be able to (i) find actual sources that correspond to portions of a given binary, or even (ii) find or "write" source code to behave like a given program, just as advertised by those pushing this sort of tech. then, source code would no longer be relevant or needed to make changes to a program, you'd just ask for another program that is like the one given as example, except for the changes you'd like. OTOH, those who control us through software would arrange so that you (or the tech) couldn't get access to the system, such as by offering it as SaaSS, with CAPTCHAs that prevented automated access. we're already there, to a large extent, even though the tech to realize that kind of job is nowhere close to exist AFAICT.
@lxo isn't it wishful thinking? AI needs compute that individuals and the environnent can't afford, and that corporates wouldn't let be applied against proprietary bits.