Conversation

Notices

1. Embed this notice
   BeAware (beaware@social.beaware.live)'s status on Tuesday, 11-Jun-2024 04:30:58 JST BeAware

   in reply to

   • Sander van Kasteel

   @SuitedUpDev as long as you don't crawl Fedi domains.

   People here don't like that kinda thing.🤷♂️

   • Embed this notice
     Sander van Kasteel (suitedupdev@mastodon.online)'s status on Tuesday, 11-Jun-2024 04:30:59 JST Sander van Kasteel

     A quick question to my fellow #PHP developers here. Does anybody have a suggestion for Spider / crawler library in PHP?

     BeAware repeated this.
   • Embed this notice
     Sander van Kasteel (suitedupdev@mastodon.online)'s status on Tuesday, 11-Jun-2024 04:30:59 JST Sander van Kasteel

     in reply to

     I wanna crawl some domains under a certain TLD and keep track how much "outgoing" links are being referenced on the domains.

   • Embed this notice
     Sander van Kasteel (suitedupdev@mastodon.online)'s status on Tuesday, 11-Jun-2024 04:49:50 JST Sander van Kasteel

     in reply to

     @BeAware I'm actually planning to crawl North Korean websites.

   • Embed this notice
     Sander van Kasteel (suitedupdev@mastodon.online)'s status on Tuesday, 11-Jun-2024 04:55:00 JST Sander van Kasteel

     @BeAware I have a short list of domains that are reachable from the "regular" internet and I want to do some research on the data I can gather from their websites.

   • Embed this notice
     Sander van Kasteel (suitedupdev@mastodon.online)'s status on Tuesday, 11-Jun-2024 05:17:56 JST Sander van Kasteel

     @BeAware Thanks a lot 🙏 I already have some experience with web scraping (from work) but not on this deep of a level.