Programming.Dev is going to be going down starting at around 00:01 UTC Monday Dec 2 to attempt to fix the database corruption that happened relating to communities.

This has been affecting attempting to look at the pages for certain communities (it would end up loading forever instead of actually showing the posts for the community) and was affecting federation in some cases.

This is the second attempt to fix the issues with the first one needing to be rollbacked from due to the fix not working.

This fix could take several hours, so the outage window is going to be from 00:01 - 06:00 Dec 2.

  • snoweMA
    link
    fedilink
    arrow-up
    8
    ·
    2 months ago

    The fixes are done for tonight, but are not 100% complete. I estimate that 99.4% of cases have been fixed though. You might see problems with individual posts, but every community should be fixed. I’ve seen at least 1 issue with the [email protected] community and I’ll be investigating that if it doesn’t self resolve (stuff could still be catching up).

    IF YOU SEE ANY MORE ISSUES PLEASE REPORT THEM HERE.

    • ALostInquirer@lemm.ee
      link
      fedilink
      arrow-up
      2
      ·
      27 days ago

      I think [email protected] may still be having some issues? From programming.dev it seems to infinitely load, but when clicking a link to visit from another instance, or refreshing the page, it shows an error of, “couldnt_find_community”.

      • Scoopta
        link
        fedilink
        arrow-up
        1
        ·
        edit-2
        2 months ago

        Wait [email protected] is broken for you? For me in Jeroba that’s one of the communities that’s been working just fine this whole time, unless the recent maintenance broke it the last post I have is from yesterday.

        EDIT: Ok, so clicking your link, or even my link gives an error saying community can’t be found but I can still pull it up from my subscriptions, so it was either broken by this maintenance or something weird is going on

        • Draconic NEO
          link
          fedilink
          arrow-up
          2
          ·
          2 months ago

          I can access many of the broken communities from external links and I might even be able to create posts in them from the UI but I cannot access their links directly from the UI, and they also don’t load from my mobile App.

        • Blaze (he/him)@feddit.org
          link
          fedilink
          English
          arrow-up
          2
          ·
          2 months ago

          For me in Jeroba that’s one of the communities that’s been working just fine this whole time, unless the recent maintenance broke it the last post I have is from yesterday.

          That’s strange indeed, the community has been inacessible from the web UI for weeks

    • my_hat_stinks
      link
      fedilink
      arrow-up
      1
      ·
      edit-2
      2 months ago

      Edit: Definitely not relevant, it was a problem with the link I clicked

      original

      Not sure if it’s related but I’ve noticed a couple of smaller communities from other instances which aren’t accessible from programming.dev any more, Imaginary Witches and Imaginary Fairies. They give the very helpful error “unknown”, this might be a fun one to try to track down. They’re accessible on the source sites (1, 2) and other communities from those instances seem to work.

      Screenshot displaying error page with text "The server returned this error: unknown."

  • Lodra
    link
    fedilink
    English
    arrow-up
    3
    ·
    2 months ago

    I thinki founda small issue though I can’t be sure it’s related to the corrupted data and yesterdays maintenance. It looks like a few of my community subscriptions are a little broken? FYI I’m using voyager on IOS.

    As the server catches up, more content from the last two months has been showing up in my home feed. Fantastic! I decided that I don’t want to subscribe to [email protected] anymore. So I looked at my subscriptions, selected dailygames, tapped the “3 dots” menu button, and tried to unsubscribe. However, I subscribed again rather unsubscribed. I then looked at my subscribers again and saw daily games listed twice. Then, I unsubscribed using the same method but this only removed one instance of my subscription to dailygames.

    So the content showed up in my home feed. My subscription was seemingly working fine. But I couldn’t unsubscribe as the subscription was not recognized in some contexts. It really feels like some crossed data in the database. Thankfully, this is a very small issue.

    I have at least one more subscription that’s showing the same problem.

    Lastly, I did find a solution. Voyager’s list of community subscriptions allows you to swipe on a specific community to unsubscribe from it. That worked just fine!

  • RustySharp
    link
    fedilink
    arrow-up
    1
    ·
    2 months ago

    I’ve not logged in for a couple of weeks. Is old. deprecated now? Or is the fact that it won’t load any comments part of the corruption issue?

    • AtegonOPMA
      link
      fedilink
      arrow-up
      2
      ·
      2 months ago

      old is still supported, what comment section are you trying to look at

      • RustySharp
        link
        fedilink
        arrow-up
        3
        ·
        2 months ago

        Huh. Turns out I had to clear cookies and site data, and now old is working again.

  • First Majestic Comet
    link
    fedilink
    English
    arrow-up
    1
    ·
    2 months ago

    Aww man I just joined here recently, hopefully it’ll be fixed soon or won’t end up effecting newer communities.

    • AtegonOPMA
      link
      fedilink
      arrow-up
      1
      ·
      2 months ago

      Newer communities arent affected

      • First Majestic Comet
        link
        fedilink
        English
        arrow-up
        1
        ·
        2 months ago

        I’m glad to hear that, I was a bit worried since I just started a community here. Do we know how it happened so we can avoid it in the future, or was it just bad luck?

        • AtegonOPMA
          link
          fedilink
          arrow-up
          2
          ·
          2 months ago

          Was an issue with the upgrade script when upgrading to 0.19.5