Nearly lost when I attacked a fallen empire and forgot to lay claim to all worlds and just the home world. They ramped up a defense force in the interim years between and I had to let some of the empire suffer to match them and recover from the mistake.