Well you need the wifi on for handoff to work, and since your phone is also ok wifi and it's saved in iCloud it auto connects...

They took away the ability to set interface priority (drag in the list to reorder) so turning it off is kind of the only way to guarantee it does what you want now 😭