@aeva You don't, it's meant to have some non-trivial latency (and half of it likely comes from the client anyway).
If you *really* need low latency I'd rather use something like Jamulus instead, but that's harder to tune in and scales poorly with number of clients (its sub-50ms latencies make it possible to jam with people online though, which is fun).