@zicklag "One of the problems with this is that your server only knows about things that have been sent to one of its inboxes."
This is untrue. Any AP server can request data from any other server to stay in synch. Literally every AP object comes with an URL to fetch its full information.