Notices by Falcon Darkstar (falcon@mastodon.falconk.rocks)

The value is in what the founders of our field did: figure out how to do something cool that you are not supposed to be able to do.

This is the antithesis of finding additional cluster members.

The "pentesting" AI fills the role of the person who keeps laughing about finding the same pattern over and over. It is valid, but the solution is not to scale that dude up.

That work is not useful work. It is rightly automated, but more in the sense of a design rules check. It is tragic that we waste people's time doing that. It is more tragic that this activity has not ceased to uncover vulnerabilities in production systems, because this shows we have learned nothing from the vulnerable patterns.

Clear, explicit specifications and output encoding help. Building simpler systems about which we can reason helps. An outside consultant critiquing the model helps.

AI, as we know it, is fundamentally about clustering, and sometimes about finding additional cluster members in randomness. An LLM does basically this. It cannot model the reason something exists or what might cause loss to the thing's operator. All that it can do is find patterns, or signals, and identify them as members of a cluster, some of which will be labelled bad.

This is like pentesting often is: some reproducible reproduction of known techniques laid out in advance.

If you strip away all the jargon about 0days and "IDOR" and whatever, you are left with this.

Vulnerabilities arise either because we didn't know about a potential threat yet, or because we keep doing the same old shit.

Structurally, AI is capable of finding the latter.

The latter should not exist, but we invested everything in trying to find each one individually instead of stamping them out at the root, because developing the practice of software engineering is apparently impossible.

So it is also with cryptography vulnerabilities. In the practice of engineering, if you build a boat anchor out of lithium, you will be fired and probably sued. Things come with stated limits and intended purposes.

You know, like GCM and 4GB of data.

In no other field does an engineer get to just put random parts together, run a process through it for 5 seconds, dump a bunch of water in and see that it fails, and then sell the assembly at scale. But that is how we build and test software.

So it is with that perennial nuisance, the injection vulnerability. All the fuzzers in the world cannot hope to find all of them.

There are two paths.

One, keep looking. Markets spring up for discoveries. Make measurements of density, value, and expected loss. Acknowledge they remain forever and keep getting hacked.

Two, apply the principles we have known since at least the development of Signalling System No. 7 in fucking 1984, and separate the control and data channels properly.

We should not, in software engineering, try to figure out how to automate the ruinous process of burying each other in boilerplate code and existing patterns. If there is a kernel of commonality among APIs, we should build a clean abstraction.

In other words, sure your IDE or some chatbot can generate a deep copy for a compiler to consume. But why bother? We all know what a deep copy is. Formally. The language should include a construct for the compiler to generate one.

I will be speaking at the final Shmoocon about eliminating the impact of SQL injection, and not about how AI cannot do pentests. But since the topic keeps coming up anyway I may as well squawk about it.

So much of the story about AI is this: what if instead of making necessary changes, we found a way to automate digging the hole deeper?

So it is with this. Scanners of all types, including genAI ones, can find the same old patterns. It is apparently fancy to do it with particular algorithms.

@skinnylatte if anyone has a bone to pick with the labour market in tech I suggest they look at the companies firing 20% of their team for absolutely no reason, or the people laying off entire departments and trying unsuccessfully to replace that labour with "AI", or more broadly, the investment community redirecting resources away from engineers building things and toward bidding up the price of hardware. It's been frustrating to watch less and less useful work get done because of investors.

@skinnylatte having come here via TN status, I have to say, the problems being complained about are mostly real (not job shortages though), but they are all caused by the system and not the immigrants. I could not bargain my salary properly for years because of it, which totally drove wages down. A work visa not tied to one employer would immediately solve that. Then there was the PERM process which did not require bona fide recruitment, but did require we accept resumes by fax.

@glyph @dalias @whitequark @mcc it is actually the NIST recommendation to *allow* password pasting, because the alternative, typing it, is completely worse. Passkeys are nice because it changes that mess into an actual defined protocol.

@ryanc you have to do the whole context switch and all the marshalling just to do one round of a cipher, which for example, to compute one block's key stream in AES-CTR on a modern x86 processor is one instruction.

@ryanc in CBC mode, can't be done. In XTS, CTR, CCM (decryption key only) or GCM, it's O(1) to do it yourself and the syscall overhead would be wild.

@ryanc this one is perhaps different from the usual (data breach for which there are no real consequences) in that it exposed how risky their product is, but also possibly mitigated by the fact that this doesn't say anything negative about whether it can do what it says it does.

@eff yup. Same story brewing with other security vendors. Tenable could soon take down the financial world with one similar bad update to their agent.

@inthehands @dekk I have built a portfolio that specifically excludes AI hype stocks for this exact reason, even to the point of avoiding index funds.

Though to be fair, I put my IRA in over a hundred individual stocks instead of index funds some time ago initially because the index funds would put me in defense contractors, Tesla, surveillance crap, tobacco, and oil, and zero commission trading makes this feasible.

Read this as advocacy for taking first-order account of the interests of stakeholders other than investors.

That probably looks like tax revenue funding open source, for example, without it being about defense or some objective beyond just making industry more efficient. Like any other project about maintaining the commons.

This isn't even so radical as expropriation. It's just recognition that the digital age made new types of public infrastructure.

Capital is dumping its resources into AI because it believes it will advance capital's interests of further accumulation of labour's products. It does not matter in the short term that capital is sorely mistaken. Much like a bad king, capital's misallocation reduces output available to labour too (by preferentially allocating resources to those who participate in the squander). We will suffer this for so long as we permit capital to make macro allocative decisions.

The current all hands on deck for AI approach of capital is like if a 20th century property developer were able to allocate substantially all road building, electrification, and water piping resources to a new subdivision. Which they are sure people want to live in, but actually smells bad and is mostly built atop a landfill.

@GossiTheDog I wonder why they cited, of all things, a King County Superior Court civil case here?