✧✦Catherine✦✧
while evaluating "AI" for something i found a new atrocious failure mode. it goes like this:
me: [image] are there differential pairs on this image?
chatgpt: yes. [confident-sounding fluff about diffpairs] should i highlight them?
me: yes
chatgpt: [a similar and quite high resolution, but _completely brand new_ image, with some random-ass highlight, still not featuring any differential pairs]
✧✦Catherine✦✧
this is actively evil
poleguy looking for lost tools
@whitequark how is this new? I haven't used ai for combined chat and images much, but my attempts at generative images typically produce fairly random outputs loosely correlated to a prompt. So these images don't seem surprising. And the chat text sounds like typical llm output.
Isn't this expected? So far my ai expectation is that I should just reprompt until I am happy with the results, or give up if I notice I have strayed too far away from the training set. Either way I am the judge.
✧✦Catherine✦✧
@poleguy i did not expect "highlight something on the image" produce a completely new fucking image
poleguy looking for lost tools
@whitequark one trouble with the interface is that it is not clear if "highlight" is a magical first class feature that's added as actual coding on top of the llm and gan stuff or if it is just part of the llm and gan prompt engineering. It seems like it just modified the prompt so you should expect a whole new image?
✧✦Catherine✦✧
@poleguy the first image is something that i uploaded. it's not generated
✧✦Catherine✦✧
@f4grx the generated highlight to be overlaid on top of the image i uploaded
F4GRX Sébastien
@whitequark I dont know what you were expecting, tbh :)
crzwdjk ✅
@whitequark Seems like the thing that these models are most confidently and dangerously wrong about is their own capabilities and limitations. Which makes sense, data about that is almost entirely absent from the training set. Would you have thought to ask it to highlight if it didn't suggest it?
poleguy looking for lost tools
@whitequark that is interesting, and potentially could lead you to not notice an important change because we are biased to see what we expect. But unless they leaned on a non ai algorithm to highlight, this all sounds "about right" for ai.
poleguy looking for lost tools
@whitequark have you seen the "remove the electric wires from the photo of a house" example?
✧✦Catherine✦✧
@poleguy nope
✧✦Catherine✦✧
@crzwdjk yes
✧✦Catherine✦✧
@f4grx there are absolutely models that use "system commands" of sorts (this is how they're able to run `rm / -rf` on your system!), and i figured that if the LLM knows how to run the diffusion model, it might also know how to do an overlay instead of regenerating the entire image--indeed i didn't expect it to regenerate the entire image because the image is _huge_ and this can't possibly be cheap for openai
F4GRX Sébastien
@whitequark from the mental image I have of LLMs, this sounds impossible to do, this request is much too deterministic and does not fit in any kind of task that can be obtained through random processes.
Glyph
@whitequark for someone who knows zero about electronics (i.e. I don't know what a "differential pair" is) would you mind explaining why this is particularly bad
✧✦Catherine✦✧
@glyph
- the "actively evil" remark is because it presents an image transformed into latent space and back (with completely wrong, horribly nonsensical but _just_ plausible enough if you don't squint details) as "result of highlighting"
- diffpairs are these wigglies (left); they are a dead tell for PCI Express (right); i asked chatgpt to identify the standard (Legacy PCI or PCI Express) from the image because someone used the image in a news article about nvidia...
✧✦Catherine✦✧
@glyph and I thought it looks way too outdated and wrong in proportions, pin count, etc for it to be anything made in the last 15 years, and wondered if the models will be able to identify this incredibly, blatantly, unmistakably to a moderately trained eye difference or not
✧✦Catherine✦✧
@glyph there were other tells that this is a PCI card (the notch is on the wrong side, the pin count is wrong, there's no 'bulge' on the bracket side, the power/ground connections are in the wrong places, there's another parallel bus on it, the overall PCB layout and fabrication looked anachronistic) but the presence/absence of diffpairs is a 100% sensitivity, 100% specificity test for distinguishing PCI vs PCI Express
✧✦Catherine✦✧
@glyph the other differences were either probabilistic/speculative (eg PCB layout style or guesses about their power distribution) or required me to go and look it up (pin counts, notch locations), which... a model "should" still be able to do better than i do, so i focused on the one where there is no ambiguity whatsoever and we're on even ground
Brad
@whitequark It even replaced the blurred Realtek chip in the background, and some of the other components.
✧✦Catherine✦✧
@glyph oh, i should explain what makes something a differential pair vs how it looks. a differential pair is a coupled pair of microstrip lines (a type of waveguide, similar in concept to coaxial cables you're familiar with) that are routed at a fixed, precisely controlled distance from each other, and within a low, precisely controlled length difference (these wavy patterns on them keep this difference small)
✧✦Catherine✦✧
@glyph diffpairs have given us the last ~25 years of gains in peripheral interface throughput, have dedicated, highly specific tools for making them in PCB layout software, are one of the most distinctive and emphasized visual structures on a PCB, and are described in an unimaginable amount of training material
you would expect any of this to matter
✧✦Catherine✦✧
@bk1e and did a sharpen pass over the nvidia logo and lettering while keeping the background blurred
Glyph
@whitequark thanks so much for the thorough explanation. Feeling very smart / pleased with myself for correctly anticipating as I was reading why a trace might have little asymmetrical bumps like that :)
poleguy looking for lost tools
@whitequark I can't find it in my history. Summary: the ai feature removed electric wires but also changed the house from a duplex to a single family and made many other changes. The before and after results looked like a "find the differences between the two pictures" game from a kids magazine.
If you have to play that game after each prompt you are better removing the wires by hand in Krita. Even if you don't know how you would be better off skilling up in Krita than prompt engineering.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.