Conversation

Notices

1. Embed this notice
   Peter Kaminski (peterkaminski@mastodon.social)'s status on Tuesday, 17-Dec-2024 07:34:18 JST Peter Kaminski

   This seems like a big deal.

   Meta AI proposes "Large Concept Models" to complement "Large Language Models".

   (i.e., reasoning in an embedding space of concepts rather than words)

   https://github.com/facebookresearch/large_concept_model

   https://arxiv.org/abs/2412.08821

   popularized article: https://www.marktechpost.com/2024/12/15/meta-ai-proposes-large-concept-models-lcms-a-semantic-leap-beyond-token-based-language-modeling/

   • Embed this notice
     John Abbe (aka Slow) (slowenough@mastodon.social)'s status on Tuesday, 17-Dec-2024 07:34:17 JST John Abbe (aka Slow)

     in reply to

     @peterkaminski (There are hierarchies, among our ways of thinking, sure. But ultimately they are not clean or coherent hierarchies, that is they branch in all kinds of weird ways that defy exclusively top-down control, organization, or even understanding.)

   • Embed this notice
     anderbill (band@hachyderm.io)'s status on Tuesday, 17-Dec-2024 07:34:17 JST anderbill

     in reply to

     • John Abbe (aka Slow)

     @slowenough @peterkaminski and that resistance to coherence and understanding is very good.

   • Embed this notice
     John Abbe (aka Slow) (slowenough@mastodon.social)'s status on Tuesday, 17-Dec-2024 07:34:18 JST John Abbe (aka Slow)

     in reply to

     @peterkaminski "LCMs employ a hierarchical structure, mirroring human reasoning processes"

     Not a good sign if they think our reasoning is hierarchical.

     "reduces sequence length compared to token-level processing, addressing the quadratic complexity of standard Transformers and enabling more efficient handling of long contexts"

     Now seeing "concepts" as a kind of compression (of strings of tokens), which I've seen articulated before as a way of understanding much of what's happening with LLMs.