@feld If they have physical access to the computer, they can attack your hardware to extract the key or do some other software/EFI type attacks.

You don't need boot drive access to carry out software or hardware attacks - you can remove the case and jack yourself into the buses (you might need to delid the SoC case, but that's not strictly impossible to do).