Ha! I got the Home Assistant Voice PE working just fine(*), but still with limited capacity. I tried Ollama as an AI conversation agent, and the only model that can run on my Raspberry Pi 4 is not very "smart" (tinyllama). Looks like I need more hardware to make it work, or subscribe to a cloud AI service. Will think about it for a while.
* Except for the fact that I had to manually build the image to make it understand my custom CA.