A new LLM called phi-, with 1.3 billion tokens, scores over 50% on the HumanEval problem set.

twitter.com/SebastienBubeck/status/1671326369626853376

GPT-4 scores 67% - but uses 1.7 trillion tokens.

twitter.com/swyx/status/1671272883379908608

How did they achieve this miracle? They trained phi-1 using textbooks rather than on the internet.

What does it means? It means you can produce an AI that is smart enough to perform simple tasks and small enough to run on your laptop or, probably, your phone.

What else does it mean? It means to score 85% on that test using the same approach as GPT-4 you'd need something like 2 quadrillion tokens, which would cost billions of dollars to train even if you could find that much data. And then years to "align" to get it to stop giving obviously wrong answers because you stuffed it full of nonsense.

Garbage in, garbage out.

Also, phi-1 took four days to train.

arxiv.org/pdf/2306.11644.pdf

Speaking of garbage, don't use textbooks published after 2010.