A large language model is just that: a model. It's a statistical function. A black box. Data goes in and data comes out.
Put the burden of proof on the AI proponents by not implicitly accepting their premises. Maybe a pile of numbers can "think", but make them explain how that works. Don't let them rhetorically handwave away that inconvenient fact.