Does OpenLiteSpeed honour the header "Vary: Accept"? From examples I've seen, it honours other Vary values. You should be able to get separate caches for each Accept value – not just text/html and application/activity+json but any others sent.
Try setting that header in the plugin, clear the OpenLightSpeed cache, and test.
header('Vary: Accept');
@pfefferle do you plan to add that to the plugin? It should automatically fix nginx, and any other caches that support Vary headers.