Sutton Valence Castle was probably built in the second half of the 12th century by the Counts of Aumale, most likely by Baldwin of Bethune, but perhaps by William le Gros, Baldwin's father-in-law.

The castle was built on a commanding position overlooking the strategic route between the towns of Maidstone, Rye and Old Winchelsea and the location was also known as Sudtone and Town Sutton.

It then passed to the families of Marshal and de Montfort, before being given by King Henry III to his half-brother William de Valence in 1265, from whom the castle takes its current name. It was abandoned in the early 14th century and fell into ruin.