The stars do change. They adjust due to the wobble of the Earth. The location of critical navigation stars and constellations have changed over the centuries. The structures at Göbekli Tepe were built decades or centuries apart, but rotated with each new structure to line up with the correct star. Archaeologist date some religious structures based on the start they would have been facing in that century.

The speeds seem fast, but the distances are vast, and much of our galaxy is moving relative to the speed of the Earth. So the changes are going to seem very small unless you expand to a larger timescale.