@doot setting agent to Google crawler may also win for sites where pages are normally behind a "please subscribe" popup