@ct_bergstrom Algebraic Graph Theory by Godsil and Royle is good. If you are looking to really understand the abstract group and algebraic structure that leads to analyses like the much more concrete adjacency matrices and that kind of stuff, they go over the main details pretty well.