Conversation

Notices

1. Embed this notice
   Matt Campbell (matt@toot.cafe)'s status on Tuesday, 01-Apr-2025 06:52:41 JST Matt Campbell

   There's a new product that has been gaining some buzz in the blind community, a Windows app called Guide that uses AI to perform tasks on your computer. It's pitched as a way to get around web accessibility problems in particular. I won't link to the thing itself, because I don't want to give it that validation, but I'll link to a previous discussion thread about it: https://fed.interfree.ca/notes/a5wf4yss764nf6h7

   I've spent some time taking this app apart. The level of shoddy work here is deeply disgusting. 1/?

   • Embed this notice
     Matt Campbell (matt@toot.cafe)'s status on Tuesday, 01-Apr-2025 06:52:37 JST Matt Campbell

     in reply to

     Perhaps I need to more explicitly call out what is actually the scariest part here: if you use this product, you're letting an application take control of your computer, using the output of a large language model as input. I know better than to describe an LLM as "just" a next-word predictor, because we've all seen how surprisingly powerful that can be. But still, it's all too common for LLMs to output things that don't make sense, especially when venturing outside their training.

   • Embed this notice
     Matt Campbell (matt@toot.cafe)'s status on Tuesday, 01-Apr-2025 06:52:38 JST Matt Campbell

     in reply to

     The Python backend is packaged using pyinstaller. There's 30 MB (compressed) of Python bytecode in the executable, and then there's also an "_internal" directory with tons of dependencies, adding up to about 200 MB (uncompressed), again with no apparent attempt at eliminating dead code in the package. I readily admit that I'm perhaps overly obsessed with trying to make non-bloated software, but come on. 6/?

   • Embed this notice
     Matt Campbell (matt@toot.cafe)'s status on Tuesday, 01-Apr-2025 06:52:38 JST Matt Campbell

     in reply to

     It wouldn't be right for me to knock the product for the bloat alone. But taken together with the direct use of third-party services in the app on the user's machine, and the actual functionality problems detailed in the thread I linked to, the whole thing smells of something hastily cobbled together to catch a ride on the AI hype train. If this is the accelerated future of software development that businesses want, then as I said, it's deeply disgusting, and kind of scary. 7/7

     Steve's Place repeated this.
   • Embed this notice
     Matt Campbell (matt@toot.cafe)'s status on Tuesday, 01-Apr-2025 06:52:39 JST Matt Campbell

     in reply to

     But now let's talk about the Python backend. The first obvious question, of course, is what AI model it's using, and whether the inference is done locally or remotely. It's using Claude 3.7 Sonnet with its computer use feature. But here's the really crappy part: the connection to Claude, and to other services like Azure Speech and ElevenLabs (yes, both), is happening on the user's machine, using API keys embedded inside the application. 4/?

   • Embed this notice
     Matt Campbell (matt@toot.cafe)'s status on Tuesday, 01-Apr-2025 06:52:39 JST Matt Campbell

     in reply to

     To spell it out, the problem with directly connecting to third-party services using API keys inside an application running on a user's machine is that you're just begging to have someone steal those keys and run up your bills. Without having your own server in the mix, there's no hope of reining in that usage of third-party services and tying it to some kind of authorization system. They do have an API server (on Azure) for the license/subscription, but as I said, that's easily circumvented. 5/?

   • Embed this notice
     Matt Campbell (matt@toot.cafe)'s status on Tuesday, 01-Apr-2025 06:52:40 JST Matt Campbell

     in reply to

     First, it's an Electron+Python monstrosity. Specifically, the Python backend runs as a web server on the local machine, and the Electron frontend connects to that local web server. Along with the size of Electron itself, the frontend app is about 27 MB, mostly a node_modules tree with no hint of tree-shaking / dead code elimination. The front-end JavaScript code is not minified at all, so once you extract the .asar file, it's easy to look at it. 2/?

   • Embed this notice
     Matt Campbell (matt@toot.cafe)'s status on Tuesday, 01-Apr-2025 06:52:40 JST Matt Campbell

     in reply to

     The frontend being fully unobfuscated would of course be a good thing if this were supposed to be open source, but it's not. And that frontend seems to be the only part of the program that validates that you have the license/subscription. That's just begging to be cracked. 3/?