You probably had an earlier structure atop the motte with a stockade around the edge of the earthworks.
I'm not going on much of any reading, just my own interpretation of the lay of the land and a panel about the bridge at the visitor center. There was next to nothing about the castle to be found.