I have not read the LLAVA paper but I am assuming it doesn't actually give any spatial information to the text model. It gets a lot of elements in the picture right, but can't determine their positioning. (Llama-3-8B is the model used).