As I said earlier in this thread, I have not read the papers, but right now I'm strongly suspecting that most models are incredibly stupid and they just throw 10mn$ at GPUs to train the crap out of them. So something like hashing every word, every 2 words, every 3 words, and so on, then adding up the bits in an array of 256 float32s might give you a pattern that you a text similarity filter that beats a lot of way more complex neural nets...