Unfortunately we had an issue updating to the latest version of Lemmy. This caused database corruption and we had to roll back to a backup that was unfortunately from 2 days ago!

An abundance of caution in the wrong error resulted in me wanting to push this change out to help with the exploit relating to custom emoji. As other instances were attacked, I didn’t want reddthat to fall to the same issue.

I am so sorry !

I have since updated our deployment scripts to automatically take a back up before all updates, as well as investigating why our cron backups didn’t work as we expected them too.

We also lost all new accounts that were created over the past 2 days, all new posts, and all subscriptions. I’m very sorry about that and hope that you will reregister with those account details again, repost the memes and pictures that make our server great.

The good news is that we have updated to the latest version of Lemmy and everything is back to “normal”.

Fixing Deployments

I’m going to use some of our money for a dedicated development server as our current deployment testing is clearly lacking.
Unfortunately that will have to wait until the weekend when I have some extra time to put into developing these new processes.

Tiff

  • MentallyExhausted@reddthat.comEnglish
    5·
    1 year ago

    That sucks, but thanks for the transparency, Tiff. Good call on the auto-backup and dev environment, cool that you have the resources to do that now.

    • Tiff@reddthat.comOPMEnglish
      3·
      1 year ago

      When you have now… 500! active users and over 2.8k user accounts I feel somewhat responsible for people having a good experience. When have you used a service which lost data twice in a week! I’m annoyed in that these things are easily avoidable but they are causing issues. Moving forward I’ll be putting in better more checks around deployments to make sure these cannot happen again.
      I keep putting too much trust into the upgrade process when new versions come out expecting it to work without issues.

      • burgundymyr@reddthat.comEnglish
        4·
        1 year ago

        I really appreciate the apology, and every web admin and sys admin has either made the same mistake, or watched someone else make it. I could absolutely see my younger self creating an instance of Lemmy and doing the exact same.

        The fact that you acknowledged the mistake, learned from it, and communicated the improvements are all great signs that we’re in the right place!

        Thanks for all you’re doing!

  • Otakat@reddthat.comEnglish
    3·
    1 year ago

    Two days is nothing compared to the misery that would come from either a compromised server or a botched upgrade.

    • Tiff@reddthat.comOPMEnglish
      2·
      1 year ago

      Yes. Unfortunately it should never have happened. But at least it was only 2 days, and not weeks!

  • TraditionalMuslim@reddthat.comEnglish
    3·
    1 year ago

    Appreciate the effort. Can we get a pinned post next time we have an update incoming and an ETA for how long the server will be down? Thanks.

  • ZagTheRaccoon@reddthat.comEnglish
    1·
    1 year ago

    Very sad, was having some great conversation that was wiped. Hope this doesn’t happen too often but I understand.