Color( name)s *are* defined by association -- with things that bear that color -- to the point that in French there's a color-name grammar rule which applies differently depending on whether the color name has gained "autonomy" with respect to the thing name or not.
But maybe by "association" you meant synaesthesia because another sense than sight is involved? If so then how else do you propose to explain colors to a *blind* person than through synaesthesia? [1/2]