Conversation

Notices

1. Embed this notice
   Kiwix (kiwix@mastodon.social)'s status on Tuesday, 17-Jun-2025 21:30:08 JST Kiwix

   in reply to

   Current timeframe: The English Wikipedia bug is part of the 1.15.1 milestone and is therefore priority: it should restart before the end of this week.

   By the end of June, we will also have fixed most of the other impactful bugs listed in 1.16 and will restart the remaining recipes.

   • Embed this notice
     Kiwix (kiwix@mastodon.social)'s status on Tuesday, 17-Jun-2025 21:30:08 JST Kiwix

     in reply to

     Seeing how long it has been (and how many have been asking/waiting for an update) we are also seriously considering accepting a number of missing entries : 100? 1000?

     Out of 7 million entries it is peanuts, but if for some reason some of the missing entries (which we can not predict) are on this list

     https://en.wikipedia.org/wiki/Wikipedia:Top_25_Report

     it is a problem. We'll probably go ahead anyway, but we'll cross that bridge when we're there.

   • Embed this notice
     Kiwix (kiwix@mastodon.social)'s status on Tuesday, 17-Jun-2025 21:30:09 JST Kiwix

     About 99% of Wikipedia zim files are now back on a monthly update schedule. The remaining 1% (24 wikis) are impacted by a variety of edge cases, listed here.

     https://github.com/openzim/zim-requests/issues/1432

     It is not always the big wikis that fail, but chances of encountering such edge cases are mechanically higher when there are many articles to crawl.

   • Embed this notice
     Kiwix (kiwix@mastodon.social)'s status on Tuesday, 17-Jun-2025 21:30:09 JST Kiwix

     in reply to

     Roughly 2/3 of the failed recipes are related to errors preventing the retrieval of a given article; we've chosen to replace these errors on a case-by-case basis, only after analysis; we continue to discover new errors (InvariantException, etc.).

     Some of these are actually on the Wikimedia end of things, but we are talking and they have people working on them as well.

     https://phabricator.wikimedia.org/tag/affects-kiwix-and-openzim/

   • Embed this notice
     Kiwix (kiwix@mastodon.social)'s status on Tuesday, 17-Jun-2025 21:30:09 JST Kiwix

     in reply to

     1/3 of the failures remain scraper bugs to be fixed - some are fairly trivial and are being hot-fixed in 1.15.1, some are more subtle and will be fixed in milestone 1.16.

     The problem is that we don't when/where failures will occur.

     As a reminder it takes anywhere between 6 to 20 days of crawl and compute to generate a zim file with 7M entries. It this happens at the beginning of the crawl, it sucks. If it happens at the end of the crawl, this sucks big time.

   • Embed this notice
     Kiwix (kiwix@mastodon.social)'s status on Tuesday, 17-Jun-2025 21:36:39 JST Kiwix

     • Infoseepage

     @Infoseepage No malice, no, just a big, massive amount of data to parse during crawl.

     A link to known bugs is at the beginning of the thread.

   • Embed this notice
     Kiwix (kiwix@mastodon.social)'s status on Tuesday, 17-Jun-2025 22:03:32 JST Kiwix

     • Infoseepage

     @Infoseepage that's our next step.