The rewrite turned a parser that collected every token into a list up front into one that reads tokens on demand in a single pass.
His coach Jim remarked:
> "This is such a clean functional style API for your tokenizer, it's evolved and matured nicely."