Untitled attachment

Notices where this attachment appears

1. Embed this notice
   iced depresso (icedquinn@blob.cat)'s status on Saturday, 23-Nov-2024 06:22:15 JST iced depresso
   huh. dumb idea ahead
   
   first break all words in to IPA (or, well, probably arpabet since the dataset for that is free) tokens and spacers. we want this so we can figure out how to get embeddings for words based on their sounds, like a human does it.
   
   then do the cortical.io trick of clustering sentences based on common words across sentences
   
   this should give us some kind of 2d approximation of spiking networks concieving of the words, which is enough for what that guy does using the numenta methods, BUT
   
   we're not going to use those engrams directly. we're gonna then throw some evolution solver or whatever works best, to alter some learning parameters of a liquid state machine or a spike net or something, so it will essentially learn to organize itself in a way that it generates the patterns we know
   
   :neocat_thonk: hm. this doesn't feel like something that is gonna work though. its closer but..
2. Embed this notice
   iced depresso (icedquinn@blob.cat)'s status on Monday, 14-Oct-2024 20:30:41 JST iced depresso

   in reply to
   @s8n word encoding is a huge idk for these. google nets use BERT or CLIP. converting english to spike trains is kind of unsolved, though cortical.io did publish an incomplete method to do that (which i filled in the blanks on.)
3. Embed this notice
   iced depresso (icedquinn@blob.cat)'s status on Wednesday, 18-Sep-2024 16:22:26 JST iced depresso

   in reply to
   @s8n to my knowledge their model was demoed as a very effective anomaly detector (its how i found out about them, through a foss project doinking their model to train those)
   
   there were a couple soft experiments here and there but i don't think anyone other than cortical.io (and maybe some bankers being very quiet) have done much with the work. none of the people on the forums put them in a closed loop to see what they would do.
   
   the damn things were capable of real-time learning though which nothing else is.
4. Embed this notice
   iced depresso (icedquinn@blob.cat)'s status on Tuesday, 21-Nov-2023 11:12:38 JST iced depresso

   in reply to
   @s8n idk i don't study googles shit these days. numentas is based on actual animal models.
   
   basically there was a hypothesis that the brain only uses a single data format and that seems to be holding up under scrutiny.
   
   so basically there are pulse trains of which cortical columns choose to activate, and they're connected in ways that let them contextually decide when to interfere with themselves. they call this a "sparse distributed representation" because you can model the pulse trains as a sparse bitmap.
   
   now the neurons understand *everything that can be understood or concieved* in this same pulse train format. visual memes, sounds, concepts, tactile profiles, whatever, the brain has one and only one method to store that and its basically a pattern of spatial pulse trains.
   
   this has the property that all things can be similar to all other things, even if you can't really fathom how a pancake is kind of similar to "the concept of christmas trees,"
   
   in the case of word2vec and cortical.io's code, its basically coming at the problem sideways in that all things are defined in relation to all other things, so when you train wordvec or BERT its using some janked system that says "in this context of these syllables, predict this middle part," and then they rip out the middle layer as a fingerprint. basically, training the neural network in the case of words is trying to produce these engrams without really understanding what they are doing.