Very close.
Your website (or application, or social web client) will still need a public endpoint to present the actor and the inbox, but to send messages you wouldn't need to go through an AP server, you'd just send the AS2 messages directly from your side.