@zicklag That's an implementation problem; the counts are available from the remote server.
I highly recommend reading my upcoming book "ActivityPub: Programming for Social Web" from O'Reilly Media, coming out later this month. You can read a free pre-release version here:
https://www.oreilly.com/library/view/activitypub/9781098162733/