GNU social JP
  • FAQ
  • Login
GNU social JPは日本のGNU socialサーバーです。
Usage/ToS/admin/test/Pleroma FE
  • Public

    • Public
    • Network
    • Groups
    • Featured
    • Popular
    • People

Notices by QOTO Announcements & Polls (qoto@qoto.org)

  1. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Wednesday, 09-Oct-2024 14:07:54 JST QOTO Announcements & Polls QOTO Announcements & Polls
    • 🎓 Doc Freemo :jpf: 🇳🇱

    Minor Update: So yesterday we had to tweak some resources server side, despite fixing the 503 with more resources things were still a bit sluggish. We boosted a few more resources (expanded the cluster itself) and now the system seems quite responsive as well.

    Please let @freemo know if you expiernce sluggish behavior or errors.

    Only thing left to fix that is broken is some UI issues with links. Some of that will actually expose features that are technically working like editing in place and translation capabilities. So as long as nothing else is broken that is more pressing we should have that fixed up in short order and then we can start working on the next version again and get out of cleanup mode.

    #QOTO #Mastodon #MastoAdmin

    In conversation about 8 months ago from qoto.org permalink
  2. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Wednesday, 09-Oct-2024 02:09:32 JST QOTO Announcements & Polls QOTO Announcements & Polls
    • 🎓 Doc Freemo :jpf: 🇳🇱

    So seems the upgrade went ok, a few small issues we should have fixed up pretty quick. Here are the broad points.

    * This was our first big move over to an entirely new architecture meant for quick upgrades with no down time. Ironically this meant a 2 day down time but the advantage is future upgrades should have very minimal downtime now.

    * We Did 5 upgrades in one go, lots of new features as a result, mostly all the new stuff from mastodon combined with the features QOTO already had.

    * The new setup was having some 503 errors at first, and it was a out of memory issue. We have since significantly boosted our resources. The system is now much faster and responsive.

    Some issues that remain, and when we will have it fixed:

    * There are a few small UI issues, the link to your own profile points to the wrong link, and the "local timeline" link is missing to bring it up. We should have those back in place very shortly (probably a day or two).

    * There were 503 issues periodically but it should now be fixed, please let me know if anyone sees a 503 again.

    * Editing posts in place (instead of delete & edit) now works but it is missing from the menu, so we need to add that.

    * Translation features should also be in place, but lacks the link.

    If anyone notices any other issues message @freemo . In the meantime give us a day or two and we should have these small issues all resolved, thanks.

    #QOTO #MastoAdmin #Mastodon

    In conversation about 8 months ago from qoto.org permalink
  3. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Wednesday, 25-Sep-2024 11:01:42 JST QOTO Announcements & Polls QOTO Announcements & Polls

    Just an FYI, on Thursday (2 days) we will be re-attemping our 5-in-1 update. It will be a very major update so expected as much as 3-days downtime as we are finally moving to the new architecture. We **will** be up at the other end with hopefully everything working and a few new features.

    In conversation about 8 months ago from qoto.org permalink
  4. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Saturday, 09-Mar-2024 02:10:56 JST QOTO Announcements & Polls QOTO Announcements & Polls

    Seems our load balancer is a little under powered and its stalling momentarily from time to time. Most of you probably didnt even notice it but some of you may have.

    I am going to upgrade that today to the next level. May result in up to 5 minutes down time (the new architecture will have multiple load balancers).

    #QOTO

    In conversation about a year ago from qoto.org permalink
  5. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Saturday, 02-Mar-2024 14:18:51 JST QOTO Announcements & Polls QOTO Announcements & Polls

    One problem fixed, another crops up...

    So while that last fix did make things better, there seems to be a bit of lingering slowness for a few things, but things are mostly working.

    I worked all day(and its midnight now) to bring up some very extensive monitoring tools for the team for us to help diagnose the issue.

    I will work through the weekend to help improve things further, though I may need to get some sleep before I can fully resolve this. At least things are working other than a slight bit of lag. I will keep everyone on #QOTO updated.

    In conversation about a year ago from qoto.org permalink
  6. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Tuesday, 27-Feb-2024 17:51:17 JST QOTO Announcements & Polls QOTO Announcements & Polls

    #QOTO is back up after a short downtime. As far as I can tell the fix went smoothly. Hopefully that will address the last of the problems from migration.

    In conversation about a year ago from qoto.org permalink
  7. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Tuesday, 27-Feb-2024 14:33:58 JST QOTO Announcements & Polls QOTO Announcements & Polls

    In about 10 minutes #QOTO will be going down shortly in an attempt to fix a 16G table that may be at the root of one small lingering problem post migration. Luckily we have good backups and the table can always be recreated from scratch.

    So should be back up shortly hopefully with the last needed fix in place and we can start the upgrades soon.

    In conversation about a year ago from qoto.org permalink

    Attachments


  8. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Sunday, 25-Feb-2024 07:37:54 JST QOTO Announcements & Polls QOTO Announcements & Polls

    Health checks and internal networks are now in place. It went smoothly!

    This should ensure that if our system crashes **for any reason** it will automatically restore itself. Should give us better uptime in the future.

    #QOTO
    QT: https://qoto.org/@QOTO/111988351143316298

    In conversation about a year ago from qoto.org permalink

    Attachments

    1. Domain not in remote thumbnail source whitelist: media.social.qoto.org
      QOTO Announcements & Polls (@QOTO@qoto.org)
      from QOTO Announcements & Polls
      We are going to try to add some healthchecks to our containers to help protect them in case they become unresponsive. No one on #QOTO should notice any real downtime unless i break something. If I do downtime should be mere seconds as I bring it back up. Just letting people know in case anything goes wrong, shouldnt be noticed if things go right..
  9. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Saturday, 24-Feb-2024 09:47:09 JST QOTO Announcements & Polls QOTO Announcements & Polls

    after another short down time everything is fixed! #QOTO had to recreate an index that got lost in the migration which was slowing down the DB and everything else. No amount of resources was going to help.

    But it is fixed now all ques are empty or very close to it. we will now downgrade the DB to a more sane level now that it is fixed (we upgrades shortly to maintain the system). But it will sill be a pretty hefty system for us, and we can always scale back up when needed.

    TL;DR everything is wording fine now.

    PS we are now going to work on a staging environment to test upgrades so we can safely start moving the main server through the upgrade cycle. stay tuned.

    In conversation about a year ago from qoto.org permalink

    Attachments



  10. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Saturday, 24-Feb-2024 08:35:48 JST QOTO Announcements & Polls QOTO Announcements & Polls

    So we found the real problem haunting us. Turns out we didnt even need the bigger database. There was just an index that got dropped during the migration. We are working now to put it back in place. At which point things should be back up to their normal speed.

    In conversation about a year ago from qoto.org permalink
  11. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Friday, 23-Feb-2024 15:56:30 JST QOTO Announcements & Polls QOTO Announcements & Polls

    One last update before I go to bed and disapear for 12 hours.

    The backlog has went from 1.2 mil at its peak earlier today to 0,4 mil now after we reconfigured things. It is steadily going down and should have everything up to working order before I get up.

    One or two people were able to get images loaded after a VERY long wait. So while images still arent working it seems very likely related to the backlog. In a few hours when the backlog clears I expect image uploads should work again. If not I will check what the problem is in the morning.

    Other than that most things appear to be working and everything should be functional soon.

    In conversation about a year ago from qoto.org permalink
  12. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Friday, 23-Feb-2024 15:51:34 JST QOTO Announcements & Polls QOTO Announcements & Polls

    The back log is about 2/3rds complete. This afternoon it peaked at 1.2 million and now it is ~0.4 million. I jut moved to pg_bouncer to sped that up a bit. Looks like I more than doubled the process time. Almost there.

    In conversation about a year ago from qoto.org permalink
  13. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Friday, 23-Feb-2024 09:08:38 JST QOTO Announcements & Polls QOTO Announcements & Polls

    So the Sidekiq backlog on #QOTO is still progressing. As of a few hours ago at the peak of the problem our backlog was 1.2 million jobs. As of right now its down to 0.7 million jobs. It is steadily decreasing there is just a lot to get through from the downtime. We are expecting everything to be back in working order when its done which should be around end of day. In the meantime things are still usable but you may experience very long lag on some actions.

    Images still cant be uploaded, we hope this is the same problem.

    In conversation about a year ago from qoto.org permalink
  14. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Friday, 23-Feb-2024 02:52:49 JST QOTO Announcements & Polls QOTO Announcements & Polls

    As spam comes in we will block the servers on the #Fediverse. Please be patient this is happening across the whole fedi and we are working on better ways to address it.

    #QOTO

    In conversation about a year ago from qoto.org permalink
  15. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Friday, 23-Feb-2024 00:19:48 JST QOTO Announcements & Polls QOTO Announcements & Polls

    #QOTO update.

    So I went to bed last night and work up to find the sidekiq workers were backlogging and we have 800K backlogged jobs. It was due to a misconfiguration that I have now fixed and it appears the backlog is quickly resolving itself.

    If you noticed any weirdness this should be resolved int he next few hours as the backlog clears.

    In conversation about a year ago from qoto.org permalink

    Attachments

    1. Domain not in remote thumbnail source whitelist: cdn2.dan.com
      update.so - Domain Name For Sale | Dan.com
      from @undeveloped
      I found a great domain name for sale on Dan.com. Check it out!
  16. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Thursday, 22-Feb-2024 15:47:23 JST QOTO Announcements & Polls QOTO Announcements & Polls

    #QOTO is back up.

    Please keep in mind #QOTO will need quite a few hours to handle the backlog from downtime. Tomorrow we are going to split out the workers so they can begin using the scaling. So tomorrow this should be fixed. For the next 24 hours expect things to be a bit slow and uploading pictures probably wont work until we fix that.

    Give it 24 hours and hopefully things will be back to normal at that time.

    In conversation about a year ago from qoto.org permalink
  17. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Friday, 02-Feb-2024 01:34:26 JST QOTO Announcements & Polls QOTO Announcements & Polls

    Please note today's server migration was moved to Friday. Expect some downtime on Friday, EST morning. We are migrating servers and begining the upgrade process for several updates we have in the pipeline. Expect brand new features in the near future.

    In conversation Friday, 02-Feb-2024 01:34:26 JST from qoto.org permalink
  18. Embed this notice
    QOTO Announcements & Polls (qoto@qoto.org)'s status on Wednesday, 31-Jan-2024 03:57:17 JST QOTO Announcements & Polls QOTO Announcements & Polls
    in reply to
    • 🎓 Doc Freemo :jpf: 🇳🇱
    • Martin Ruskov

    @mapto Hi, so this is generally not a monitored account, so im sorry you didnt get a response for 5-days. I will tag the main admin and he will see this and be able to respond.

    @freemo

    In conversation Wednesday, 31-Jan-2024 03:57:17 JST from qoto.org permalink

User actions

    QOTO Announcements & Polls

    QOTO Announcements & Polls

    We only posts announcements here and occasionally use polls to help us make decisions that effect our user base. We don't usually respond quickly to direct messages to this account. If you need help with anything related to the QOTO servers, including moderation, then please contact one of our administrators. They are listed out about page:https://qoto.org/about/more

    Tags
    • (None)

    Following 0

      Followers 0

        Groups 0

          Statistics

          User ID
          39119
          Member since
          25 Nov 2022
          Notices
          18
          Daily average
          0

          Feeds

          • Atom
          • Help
          • About
          • FAQ
          • TOS
          • Privacy
          • Source
          • Version
          • Contact

          GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.

          Creative Commons Attribution 3.0 All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.