Conversation

Notices

1. Embed this notice
   Dachary (dachary@dacharycarey.social)'s status on Thursday, 14-Nov-2024 23:44:57 JST Dachary

   Oh no. I may have accidentally found a task that an LLM is suitable for.

   We have come up with what we think is a reasonable classification system for the code examples in our docs.

   We have ~27,000 things that *might* be code examples.

   My brain is down the rabbit hole of figuring out how to get an LLM to apply our rubric to classify our code examples.

   So far, early testing = acceptable accuracy. So no need to manually audit 27,000 examples, hopefully!

   • Embed this notice
     Eaton (eaton@phire.place)'s status on Thursday, 14-Nov-2024 23:44:56 JST Eaton

     in reply to

     @dachary So! I’ve been doing a buuunch of classification and categorization tests with LLMs lately and that’s really one of the best matches IME!

     Would love to compare notes!

   • Embed this notice
     Eaton (eaton@phire.place)'s status on Friday, 15-Nov-2024 04:52:10 JST Eaton

     in reply to

     @dachary Qwen 2.5 is the best of the local models in my testing for the same purpose — and the same gap w/gtp4o mini. The biggest thing that jumped out for me was to stay alert for taxonomy decisions that rely on intentions outside of the documents themselves (ie this is for beginners). Nice to see you’re seeing similar results!

   • Embed this notice
     Dachary (dachary@dacharycarey.social)'s status on Friday, 15-Nov-2024 04:52:11 JST Dachary

     in reply to

     • Eaton

     @eaton I’m not doing anything too special at the moment. I had reasonably accurate results with a GPT model in some manual testing, so I wrote a little Go code to iterate through a directory (a git repo) on my file system and ask an Ollama model to classify the code files based on my taxonomy. I tried the newish Ollama code-reasoning model, qwen2.5-coder, and it is - less accurate at the classification task than GPT. Trying to decide if I should tweak my prompt or try out some different models.

   • Embed this notice
     Eaton (eaton@phire.place)'s status on Friday, 15-Nov-2024 04:56:36 JST Eaton

     in reply to

     @dachary but yeah, A+ for Qwen out of all the local models https://vega.github.io/editor/#/url/vega-lite/N4IgJAzgxgFgpgWwIYgFwhgF0wBwqgegIDc4BzJAOjIEtMYBXAI0poHsDp5kTykBaADZ04JAKyUAVhDYA7EABoQAEzjQATjRyZ289AEEoUBuqRQAngAJS6iAwiWdCOJYDu8WZahJM5NpoAvGlkyS2U2Y2dZTAcmK0w2HBooRRUfFFRQE0E0DGw8QgJTV2o6RiZ7OHUoOV9oyhqEAjgfOSK2CDhZADM4QWUiuG6IAngkZRHkYII2Blw5ggQ2VUF+M2NTCwaIYhAAXyVXGmV6NAA2AAYLpXgaMiw0ABYrpRwkUwQINABtUFkkZy5BJkMiCOAAMQBNEE5lSnTBUEwaFAmHMODguRwbGCSKU3RofQmPxA3ShMJAAF0DiAmMFlLkwWQuvS9hSlF0aspgmRkSAAB68-GE3IIYJzNSpVHo3IARwYSGidB8NFIcO8YN5ASqbDQpMEnSUUox6AgMvUSL21NhmRJBP6uXWJjMsMNaONIDlCp0mGVqqU0CQGptWvUOtQeoNKjYU1kAFkkALUGILpbDWw2IIdDgfqAhfb0EsVpK3bl-Fz-jlDXQgyBY8s+vsFLm7fT0I7Ni6QN6a4YNs7G83hQWfLAJVXMDX45hR18DoP8yAqqHbJLq+6AKLqZezpu2ocgUWycVfceTsW+BwAFSQAGsugO9wuahBcV217kAMIdTCWYKWAAiGaCO8s5siAiRmHQ1qgDUshcjoci8m8HxAmwIJgpCorkkoxCBgwxoXJQACc1K4YI+FoIRABM1IQDASDSjaeatiSZKdkauSyNGwSBo2IA1II-iCi2QJwLAsg0HKGKuoxIBcYevGpiAwHmFUOYHu8N68hx6BYjicI0FqaAABwpqySiwfiPI2i+5hBqAwFMA2NqBnceggJo9yvkwSDwsE7qisoygakoyiJvwYihda-DGZalpAA/view