I understand the scenario you describe, but I don't see how it relates with the question.
say I have the model but not the blobs used to train it (and neither the BigNumber library source nor object code)
ISTM that I can start from that model and (1) study it to learn what it does, and train it further so as to adapt it to do what I wish; (2) make copies of the model and distribute it; (3) improve the model and distribute the improvements; and (0) run the model for any purpose.
so, despite not having the training data, ISTM that I have the four essential freedoms nevertheless.
do you not agree with this assessment?
note I'm not saying that having the training data (and free software used to create it) is not desirable, just that AFAICT it's not strictly necessary to have the essential freedoms