@tero i think this is something of an artifact with how most people are doing it. i passed by a paper that was suggesting neural networks *are* decision trees, and some of the really old stuff on sparse networks (GMDH comes to mind) made attempts to winnow connections down to some useful subset at various layers.

it's kind of the modern "google-brained" thing to just slap dense nets of every parameter to every parameter and just throw a datacenter at the problem to make up for how ineffective this is.