Notices by David Chisnall (*Now with 50% more sarcasm!*) (david_chisnall@infosec.exchange), page 3

Someone should point out to Trump that if each Canadian province became a US state then there would be enough blue electoral college votes to ensure that his team never got in again, but if he sold the blue costal states to Canada then he would have enough votes for a constitutional amendment to make him dictator for life to pass.

@ryanc I was going for -INF, but -0 is probably better.

Housing efficiency subsidies rarely work well for the people who needed them most: tenants in cheap housing. For many years, I rented somewhere with single-glazed windows and no loft insulation. The cost of fixing it would have been at least £20k (probably a lot more, given the state of the roof), which was many years of the £350/month rent I was paying. It cost a lot to heat in the winter, I was basically running the boiler all of the time. There were some insulation subsidies available for the property owner, but they weren’t the ones paying the heating bill so didn’t see the saving and, because the subsidies didn’t cover 100% of the costs, they would have spent money to save me money. The incentives were not aligned.

So I have a simple proposal: mandate minimum efficiency requirements for rented accommodation with a 3+ year compliance requirement, require letting agents and private rental listings to include the average heating cost for the year in the advert, and provide government-backed loans to cover 100% of the cost of compliance. If you don’t have the liquid funds available to pay for upgrades (or don’t want to spend them), you can get a Bank of England base rate loan that covers the cost of the upgrade. The loan accrues interest at the base rate, but you don’t have to pay any of it back until you sell the property, at which point the capital plus interest must all be repaid (you can also pay it back in advance). The same technique could be applied to heat pumps, solar panels, and so on.

I would expect that this would stimulate the economy (jobs for builders), reduce emissions (less waste from inefficient heating), and have a high take up because the incentives are now aligned. Having a well-heated house reduces the risk of damp and similar things that can dramatically lower property values and the improvements are likely to increase property values with no up-front costs, making them attractive to landlords.

Anyone know why this wouldn’t work?

@ryanc @sophieschmieg @Lookatableflip I guess it’s not pure software, but anything running on a real computer has a hardware component. The randomness bit is pure software, using whatever it can from the environment as entropy sources, but none of the entropy sources alone (without a hardware random number generator) has enough entropy to be useful, and interrupt timings can sometimes be under attacker control (some fun attacks from the ‘90s involved sending packets at specific timing to influence the entropy collection).

@ryanc @Lookatableflip @sophieschmieg That depends a lot on the system. It will use all of the entropy sources available to the kernel. On modern systems, that typically includes at least one hardware entropy source. These are often a set of free-running ring oscillators, which then feed into some cryptographic hash function for whitening.

Without these, it will use much weaker things. The contents of the password file, the hash of the kernel binary, the cycle count at the time interrupts fire or devices are attached, and so on.

There have been some high-profile vulnerabilities from embedded devices that did things like generating private keys on first boot, with deterministic device attach time, and ended up with a handful of different private keys across the entire device fleet.

@feld

This is a terrible take and you should really know better. It's not different than chastising people who use higher level programming languages or Dreamweaver to make a website instead of studying HTML.

I feel like you didn’t read past the quoted section before replying with a needlessly confrontational reply.

It is very different. If you give someone a low-code end-user programming environment, they have a tool the helps them to unambiguously express their intent. It gives them a tool to do so concisely, often more concisely (at the expense of generality), which empowers the user. This is a valuable thing to do.

We should all be able to agree that giving people a way to use natural language to build little apps, tools, and automations that solve problems nobody is going to build a custom solution for is a good thing.

No, I disagree with that. Giving them a natural-language interface and you remove agency from them. The system, not the user, is responsible for filling in the blanks. And the system does so in a way that does not permit the user to learn. Rather than using the tool badly and then improving as a result of their failure, the system fills in the blanks in arbitrary ways.

A natural-language interface and an easy-to-learn interface are not the same thing. There is enormous value in creating easy-to-learn interfaces that empower users but giving them interfaces that use natural language is not the best (or even a very good) way of doing this.

@cesarb @tthbaltazar @mjg59 Don’t confuse on-package TPMs and fTPMs. A lot of fTPMs (which run on the main core in a privileged mode) are often vulnerable to side channels. Several of the recent transient execution attacks could leak fTPM secrets. I think most of these were patched by doing some aggressive state flushing on TPM events, but people keep finding new side channels. On-package TPMs, where the TPM is a separate component either in the same package or on the same die are typically not vulnerable to these attacks. On the MS Surface laptops, there’s a Pluton subsystem on die, which runs the TPM stack. Pluton is one of the few Microsoft security products I have a lot of faith in (I worked with that team, they’re great): it stood up to over a decade of attacks from people with physical access and a strong financial incentive to break it.

@baltauger In linguistics, the Whorf-Sapir hypothesis, also known as the Linguistic Relativity hypothesis, argues that language constrains thought. This was the idea behind Orwell's Newspeak. The strong variant argues that you cannot think an idea that your language cannot express (the goal of Newspeak), the weak variant argues that language guides thought. The strong variant is largely discredited because it turns out that humans are really good at just making up new language for new concepts. The weak variant is supported to varying degrees.

I keep trying to persuade linguists to study it in the context of programming languages, where humans are limited in the things that they can extend because a compiler / interpreter also needs to understand the language. I think there are some very interesting research results to be found there.

@jonmsterling The right mental model for interacting with an LLM is to treat it like a person being tortured: It will say whatever is most likely to make you stop, the only trustworthy answers are ones that you can instantly validate.

A lot of the current hype around LLMs revolves around one core idea, which I blame on Star Trek:

Wouldn't it be cool if we could use natural language to control things?

The problem is that this is, at the fundamental level, a terrible idea.

There's a reason that mathematics doesn't use English. There's a reason that every professional field comes with its own flavour of jargon. There's a reason that contracts are written in legalese, not plain natural language. Natural language is really bad at being unambiguous.

When I was a small child, I thought that a mature civilisation would evolve two languages. A language of poetry, that was rich in metaphor and delighted in ambiguity, and a language of science that required more detail and actively avoided ambiguity. The latter would have no homophones, no homonyms, unambiguous grammar, and so on.

Programming languages, including the ad-hoc programming languages that we refer to as 'user interfaces' are all attempts to build languages like the latter. They allow the user to unambiguously express intent so that it can be carried out. Natural languages are not designed and end up being examples of the former.

When I interact with a tool, I want it to do what I tell it. If I am willing to restrict my use of natural language to a clear and unambiguous subset, I have defined a language that is easy for deterministic parsers to understand with a fraction of the energy requirement of a language model. If I am not, then I am expressing myself ambiguously and no amount of processing can possibly remove the ambiguity that is intrinsic in the source, except a complete, fully synchronised, model of my own mind that knows what I meant (and not what some other person saying the same thing at the same time might have meant).

The hard part of programming is not writing things in some language's syntax, it's expressing the problem in a way that lacks ambiguity. LLMs don't help here, they pick an arbitrary, nondeterministic, option for the ambiguous cases. In C, compilers do this for undefined behaviour and it is widely regarded as a disaster. LLMs are built entirely out of undefined behaviour.

There are use cases where getting it wrong is fine. Choosing a radio station or album to listen to while driving, for example. It is far better to sometimes listen to the wrong thing than to take your attention away from the road and interact with a richer UI for ten seconds. In situations where your hands are unavailable (for example, controlling non-critical equipment while performing surgery, or cooking), a natural-language interface is better than no interface. It's rarely, if ever, the best.

@dalias @brokengoose @LPerry2 @josh0 It's amazingly difficult to find good book data. Most things use Amazon's database (with all of the tracking that comes with) because everything else is so much worse.

Publishers seem to treat their slice of the ISBN database as something to sell, rather than something that, as part of the commons, would increase the value of the books that they sell. This means any kind of mapping between ISBNs and books is hard (and it's a many-to-many relationship since an ISBN identifies a print volume, which may be a single edition of a book or an omnibus edition that includes multiple logical books). Building any kind of meaningful ontology on top of this is really hard. Wikidata trues but is missing a lot of things.

LibraryThing provided services to libraries before being bought by Amazon but their data is really bad. Lots of books seem to have been entered by using computer vision on the cover so the title fields include every word on the cover, such as 'the new novel in the X series' and so on. So much value is lost to society by there being no maintained database for this. I suspect the amount that half a dozen libraries pay as a result of it not existing could completely fund its development and maintenance.

@futurebird @servelan @fivetonsflax @nazokiyoubinbou @justafrog @clayote I’ve been actively avoiding Amazon for about ten years. I first realised that they were not the cheap option when I bought some garden furniture 15 years ago. I discovered that the same seller sold the four-seat version for the price I paid them for the two-seat version on Amazon. After that, I started using Amazon just for discovery: find the thing I want to buy there and then find the place I actually buy it from elsewhere. Often, searching for the product name and seller from Amazon will take you to another shop front that charges less because it isn’t giving Amazon a cut.

For book, even Hive, which supports local book sellers, is cheaper.

Niche things that used to be Amazon-only are now often sold through eBay as well (eBay seems to have become more of a generic shop front and less of a second-hand auction site now).

@lanodan @carbontwelve Spam filtering has been a good application for machine learning for ages. I think the first Bayesian spam filters were added around the end of the last century. It has several properties that make it a good fit for ML:

• The cost of letting spam through is low, the value in filtering most of it correctly is high.
• There isn’t a rule-based approach that works well. You can’t write a list of properties that make something spam. You can write a list of properties that indicate something has a higher chance of being spam.
• The problem changes rapidly. Spammers change their tactics depending on what gets through filters and so a system that adapts on the defence works well. You have a lot of data of ham vs spam to do the adaptation.

Note that this is not the same for intrusion detection and a lot of ML-based approaches for intrusion detection have failed. It is bad if you miss a compromise and you don’t have enough examples of malicious and non-malicious data for your categoriser to adapt rapidly.

The last point is part of why it worked well in my use case and was great for Project Silica when I was at MS. They were burning voxels into glass with lasers and then recovering the data. With a small calibration step (burn a load of known-value voxels into a corner of the glass) they could build an ML classifier that worked on any set of laser parameters. It might not have worked quite as well as a well-tuned rule-based system, but they could do experiments as fast as the laser could fire with the ML approach, whereas a rule-based system needed someone to classify the voxel shapes and redo the implementation, which took at least a week. That was a huge benefit. Their data included error-correction codes, so as long as their model was mostly right, ECC would fix the rest.

@iximeow I suspect that a big part of the 'computer people' vs 'not computer people' split is similar to the experience eating capsicum. When you eat capsicum, it causes pain and your body then creates dopamine to counter the pain. For a lot of people, the dopamine effect is greater than the pain and so the overall experience is pleasant (there was a fascinating experiment a few years ago that fed people chillies and blocked the dopamine response: universally, everyone hated the taste of chillies, even people who loved them normally).

Everyone gets frustrated by computers doing the wrong thing for bizarre reasons (which may be a simple misnamed thing), but some people really enjoy the experience that you get after you've found and fixed the problem. Whether that joy outweighs the suffering varies a lot between people.

@Daojoan

I miss the time when progressive tax rates meant that wealth accumulation plateaued well before individuals could afford to buy small countries. I don’t miss the fact that a small number of descendants of kings already had that much. I do miss the memory of Cromwell and Robespierre being front and centre of their minds if they tried to abuse that power.

I miss Labour and Civil Rights movements constantly gaining ground and the expectation that, even if things weren’t great for everyone now, they would be slightly better each year. I don’t miss the various forms of oppression that they fought against.

I miss corporations being accountable to governments. I don’t miss corporations like the East India Companies being de-facto governments.

I miss technology being exciting and each new advance making the world better. I don’t miss nuclear first-strike doctrine being acceptable military policy.

I miss the Geneva Convention being a thing that major powers took seriously and prosecuted war criminals who violated. I don’t miss all of the atrocities that led to people deciding it was necessary.

I am nostalgic for a small set of slices of the last 200 years of history, which never happened at the same time.

@carbontwelve I used machine learning in my PhD. The use case there was data prefetching. This was an ideal task for ML, because the benefits of a correct answer were high and the cost of an incorrect answer were low. In the worst case, your prefetching evicts something from cache that you need later, but a 60% accuracy in predictions is a big overall improvement.

Programming is the opposite. The benefits of being able to generate correct code faster 80% of the time are small but the costs of generating incorrect code even 1% of the time are high. The entire shift-left movement is about finding and preventing bugs earlier.

I finally turned off GitHub Copilot yesterday. I’ve been using it for about a year on the ‘free for open-source maintainers’ tier. I was skeptical but didn’t want to dismiss it without a fair trial.

It has cost me more time than it has saved. It lets me type faster, which has been useful when writing tests where I’m testing a variety of permutations of an API to check error handling for all of the conditions.

I can recall three places where it has introduced bugs that took me more time to to debug than the total time saving:

The first was something that initially impressed me. I pasted the prose description of how to communicate with an Ethernet MAC into a comment and then wrote some method prototypes. It autocompleted the bodies. All very plausible looking. Only it managed to flip a bit in the MDIO read and write register commands. MDIO is basically a multiplexing system. You have two device registers exposed, one sets the command (read or write a specific internal register) and the other is the value. It got the read and write the wrong way around, so when I thought I was writing a value, I was actually reading. When I thought I was reading, I was actually seeing the value in the last register I thought I had written. It took two of us over a day to debug this. The fix was simple, but the bug was in the middle of correct-looking code. If I’d manually transcribed the command from the data sheet, I would not have got this wrong because I’d have triple checked it.

Another case it had inverted the condition in an if statement inside an error-handling path. The error handling was a rare case and was asymmetric. Hitting the if case when you wanted the else case was okay but the converse was not. Lots of debugging. I learned from this to read the generated code more carefully, but that increased cognitive load and eliminated most of the benefit. Typing code is not the bottleneck and if I have to think about what I want and then read carefully to check it really is what I want, I am slower.

Most recently, I was writing a simple binary search and insertion-deletion operations for a sorted array. I assumed that this was something that had hundreds of examples in the training data and so would be fine. It had all sorts of corner-case bugs. I eventually gave up fixing them and rewrote the code from scratch.

Last week I did some work on a remote machine where I hadn’t set up Copilot and I felt much more productive. Autocomplete was either correct or not present, so I was spending more time thinking about what to write. I don’t entirely trust this kind of subjective judgement, but it was a data point. Around the same time I wrote some code without clangd set up and that really hurt. It turns out I really rely on AST-aware completion to explore APIs. I had to look up more things in the documentation. Copilot was never good for this because it would just bullshit APIs, so something showing up in autocomplete didn’t mean it was real. This would be improved by using a feedback system to require autocomplete outputs to type check, but then they would take much longer to create (probably at least a 10x increase in LLM compute time) and wouldn’t complete fragments, so I don’t see a good path to being able to do this without tight coupling to the LSP server and possibly not even then.

Yesterday I was writing bits of the CHERIoT Programmers’ Guide and it kept autocompleting text in a different writing style, some of which was obviously plagiarised (when I’m describing precisely how to implement a specific, and not very common, lock type with a futex and the autocomplete is a paragraph of text with a lot of detail, I’m confident you don’t have more than one or two examples of that in the training set). It was distracting and annoying. I wrote much faster after turning it off.

So, after giving it a fair try, I have concluded that it is both a net decrease in productivity and probably an increase in legal liability.

Discussions I am not interested in having:

• You are holding it wrong. Using Copilot with this magic config setting / prompt tweak makes it better. At its absolute best, it was a small productivity increase, if it needs more effort to use, that will be offset.
• This other LLM is much better. I don’t care. The costs of the bullshitting far outweighed the benefits when it worked, to be better it would have to not bullshit, and that’s not something LLMs can do.
• It’s great for boilerplate! No. APIs that require every user to write the same code are broken. Fix them, don’t fill the world with more code using them that will need fixing when the APIs change.
• Don’t use LLMs for autocomplete, use them for dialogues about the code. Tried that. It’s worse than a rubber duck, which at least knows to stay silent when it doesn’t know what it’s talking about.

The one place Copilot was vaguely useful was hinting at missing abstractions (if it can autocomplete big chunks then my APIs required too much boilerplate and needed better abstractions). The place I thought it might be useful was spotting inconsistent API names and parameter orders but it was actually very bad at this (presumably because of the way it tokenises identifiers?). With a load of examples with consistent names, it would suggest things that didn't match the convention. After using three APIs that all passed the same parameters in the same order, it would suggest flipping the order for the fourth.

#GitHubCopilot #CHERIoT

@davidgerard

I filled in the feedback form. I suggested that they employ a civil servant to be given a rolled-up newspaper and tasked with hitting any MP who says ‘AI’ on the nose and shouting ‘NO!’ at them.

It seemed much more sensible an idea than anything that they were proposing.

@nullagent Deciding that a carrion eater is the symbol of their ideals, apparently?

Finally, I managed to tell a joke that’s too geeky for Mastodon!

https://infosec.exchange/@david_chisnall/113668722646200120