• Lodra
    link
    fedilink
    English
    arrow-up
    5
    ·
    1 year ago

    So if you don’t mind sharing, what happened?

    • snoweOPMA
      link
      fedilink
      arrow-up
      19
      ·
      1 year ago

      yeah in addition to that I’ll do a full writeup, but had several problems

      1. lemmy-ansible had pict-rs set to 0.4.0, which is quite an old version, meaning migration couldn’t occur concurrently (it was copying files to object storage sequentially)
      2. upon stopping the migration at one point and attempting to restart (on a newer pict-rs version) the server had completely filled up, so the migration could no longer continue on either version
      3. therefore I had to mount more block storage, copy the entire drive over to the block storage, in order to perform the migration.

      there were lots more issues, but copying hundreds of gigs many times over is a cause to lot of the downtime.

    • AtegonMA
      link
      fedilink
      arrow-up
      15
      ·
      edit-2
      1 year ago

      Site ran out of storage due to pict-rs storage growth (280GB) (pict-rs is used for images on lemmy) so was being migrated over to object storage today. Theres some posts about it over on the mastodon account https://mastodon.social/@programming_dev

      • burtek
        link
        fedilink
        arrow-up
        5
        ·
        1 year ago

        Wish I knew about this mastodon account earlier tbh

      • Lodra
        link
        fedilink
        English
        arrow-up
        3
        ·
        edit-2
        1 year ago

        Thanks for the info! Thanks also for the link! I wasn’t aware of the mastodon account 🙂

      • cmeerw
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        So that’s 280 GB in roughly 5 months? At what rate is it growing now?

        • AtegonMA
          link
          fedilink
          arrow-up
          1
          ·
          1 year ago

          that would have to be something for snowe to answer since I dont know the numbers on that