Embed Notice

I don't think I have written it anywhere but deep neural networks have a unit bias. It's one of those things one just knows and doesn't even realize other people don't know that.

It's the reason why tree models generally win deep neural networks with tabular data. For decision tree based models one of the basic operations is a comparison, and those operations generally have meaning with tabular data. It has correct inductive biases respecting units of things. It doesn't sum ages to countries.

With neural networks the first operation is a weighted sum over *everything*. In tabular data that means ages, religions, countries, weights, dollars. What's 7 years plus 21 dollars? Does it make any sense? No?

That's why neural networks have hard time with tabular data. Their inductive bias is assuming that everything is in the same units.