mhoye but seasonally spooky
Incredible. Every second paragraph in this article is lunatic nonsense.
One of the things I've long said about hiring is that you can always tell when you're talking to a junior dev who's going to be senior-staff or better someday. You can always tell when somebody was paying attention in the theory classes.
But good god you can also tell when people missed that day in gradeschool when somebody slowly went over "So, what is a computer, really."
mhoye but seasonally spooky
"Railway stores volume-level backups in the same volume — a fact buried in their own documentation that says "wiping a volume deletes all backups" — those went with it" WHAT IN THE WHAT, your full stack jenga tower does WHAT with BACKUPS WHAT my sweet summer child I know that legal jargon can be perplexing and counterintuitive at times but I feel like we all sort of understand that the word "due" in "due dilligence" means "more than none."
mhoye but seasonally spooky
"The agent then, when asked to explain itself, produced a written confession..." um what
"To execute the deletion, the agent went looking for an API token. It found one in a file completely unrelated to the task it was working on" went looking, found, what in the what
"the same token had blanket authority across the entire Railway GraphQL API, including destructive operations" look, rookie what are you
"That 1000% shouldn't be possible. We have evals for this" you have whaaaaaaaaaaaaa
Rich Felker
@tito_swineflu @mhoye It's clowns all the way down.
Tito Swineflu
@mhoye I love that the first line in "What needs to change" isn't, "We should not let non-deterministic programs have free range across our systems"
mhoye but seasonally spooky
I wrote the words "I confess, I did it, I take full responsibility" on a piece of paper. I was ready to turn myself in, to atone for my crimes. But then I put that piece of paper in a photocopier, and when I pressed the green button I learned something amazing. And what a weight off my conscience! The only question was, how did the photocopier manage to poison the Widow Bentley, drive over Baron Grimald, push the Duchess of Lockley out the balcony window and still manage to frame the butler?
mhoye but seasonally spooky
But my favourite part of this, bar none, is how it's everyone else's fault.
It's Cursor's fault, Railway's fault, maybe even Anthropic's fault, someone's gonna hear from my lawyer.
The CEO of a company running a stochastic stack without access control, data hygiene or backups is blameless and powerless. That's AI's real selling point, after all: It's Not My Fault As A Service.
"This isn't a story about one bad agent or one bad API. It's about an entire industry ..."
Or, maybe it's you.
mhoye but seasonally spooky
"The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing.
The "system rules" the agent is referring to are consistent with Cursor's documented system-prompt language and our project rules for this codebase. Both safeguards failed simultaneously."
What do you think is happening here? You know it's called a "language model", right? Did you ever wonder... why?
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.