Conversation

Notices

1. Embed this notice
   Siderea, Sibylla Bostoniensis (siderea@universeodon.com)'s status on Thursday, 24-Aug-2023 23:35:54 JST Siderea, Sibylla Bostoniensis

   in reply to

   • Emma Builds 🚀

   @emma These are all reasonable things but this is not a reasonable situation: the endpoint I am using has nothing to do with the documentation they sent me. I'm not using the authenticated API. I do not wish to use the authenticated API. I shouldn't need to use the authenticated API, because I am trying to get publicly available information, and really the only reason I'm using the API is its much nicer than scraping their screens.

   That said, I just saw somewhere here somebody posted a cute cartoon about Scraping Chad vs API-using Virgin which I am feeling *really hard* right now.

   • Embed this notice
     Siderea, Sibylla Bostoniensis (siderea@universeodon.com)'s status on Thursday, 24-Aug-2023 23:35:56 JST Siderea, Sibylla Bostoniensis

     in reply to

     Or rather, I am *absolutely* that vindictive, the question is whether the sensible part of me which had been thinking I ought to be engaging in better self-care and maybe should go to bed earlier tonight is going to prevail over my indignance that #patreon broke my script and deserves to have their server lightly pentested.

     There are two wolves inside me. One of them has a masters in mental health counseling and the other is a hacker.

   • Embed this notice
     Siderea, Sibylla Bostoniensis (siderea@universeodon.com)'s status on Thursday, 24-Aug-2023 23:35:56 JST Siderea, Sibylla Bostoniensis

     in reply to

     Okay, somebody check my math and my logic here.

     There were 15 headers. This means the total number of all possible combinations of those headers is 2^15, right?

     2^15 is 32,768.

     That doesn't *look* like that big a number, from a computer programming standpoint. Patreon has nice big, robust, industrial servers, presumably.

     If I want to query Patreon's nice API 32,768 times... Over what minimum span of time should I make those 32,768 GET requests, to minimize the chances their server will decide to stop answering them?

     Why, yes, I've written a perl script.

     At one per second, that works out to... 9.1 hours? Did I math that right?

   • Embed this notice
     Emma Builds 🚀 (emma@orbital.horse)'s status on Thursday, 24-Aug-2023 23:35:56 JST Emma Builds 🚀

     in reply to

     @siderea I'm used to dealing with APIs and from a glance at the docs you linked, looks like the headers to look for are the ones that return either the login cookie or the API key (if you have one.)

     The Firefox dev tools (F12) network tab is a good way to look for those.

     If you register with them it appears you can send an API key in the headers with a request to an endpoint.

     I'm a fan of the RESTClient web extension for Firefox because it lets me populate headers.

     Also, hell yeah, Perl script!

   • Embed this notice
     Siderea, Sibylla Bostoniensis (siderea@universeodon.com)'s status on Thursday, 24-Aug-2023 23:35:57 JST Siderea, Sibylla Bostoniensis

     in reply to

     Further debugging:

     The server running my cron job is in Europe, so today I tried it from a server in the US (New Jersey), and got exactly the same 403. I'm still able to access it from my browser, so it doesn't seem the distinction is geographical.

     So the next thing I did was spoof the user agent in wget. Initially, I sent a blank user agent; that didn't work. Then I sent a random user agent example I plucked out of Mozilla doc; that didn't work. Then I actually got the user agent out of the browser that had successfully accessed the API endpoint, and spoofed that in wget; that didn't work either.

     WTF.

     My sweetie suggested that I actually examine the HTTP headers my browser was sending to see if there was anything there Patreon's API endpoint might be reacting positively to that I could spoof with wget. I have not yet pursued this, as I have other things I need to do tonight.

     I am open to other hypotheses and suggestions.

     #Patreon has not yet responded to my help request.

   • Embed this notice
     Siderea, Sibylla Bostoniensis (siderea@universeodon.com)'s status on Thursday, 24-Aug-2023 23:35:57 JST Siderea, Sibylla Bostoniensis

     in reply to

     I have heard back from #patreon ! Support writes:

     > Thank you for reaching out to Patreon support — I'm happy to help here.
     >
     > We have a range of resources for using our public API. Self-serve documentation guides for using the API are available at https://docs.patreon.com/ and https://www.patreon.com/portal
     >
     > For additional questions you can find peer-to-peer resources in our Tech and Dev channel on Discord.
     >
     > Please feel free to reach out if there's anything else I can do to support you.
     >
     > Kindly, [NAME REDACTED]

     "You write a distributed map reduce function in Erlang!"

     "Did you just tell me to go fuck myself?"

     "I believe I did, Bob."

   • Embed this notice
     Siderea, Sibylla Bostoniensis (siderea@universeodon.com)'s status on Thursday, 24-Aug-2023 23:35:57 JST Siderea, Sibylla Bostoniensis

     in reply to

     Success! I finally noticed that when I load the API endpoint in my browser, Firefox helpfully has a "headers" tab, and I just poured them all into my wget and, lo! I got back a 200!

     It's returning the results gzipped. Huh. Well, I can update my script to handle that.

     I am trying to decide whether I care enough slash am vindictive enough to trouble to lovingly test header combinations to see which ones actually matter.

   • Embed this notice
     Siderea, Sibylla Bostoniensis (siderea@universeodon.com)'s status on Thursday, 24-Aug-2023 23:35:58 JST Siderea, Sibylla Bostoniensis

     Huh. #Patreon is serving me a "403: Forbidden" error in response to my trying to wget the public (unauthenticated!) API page for my account from my server (part of a cron job script, but also tested manually at the command line, which is how I found out about the 403), even while serving it to me just fine in the browser. Yes, even while not logged in.

     This change started sometime in the last ~28 hours. Script ran fine at 03:08:00+0100 yesterday.

     I filed a support request, but Patreon has largely disavowed their API. Poot.