I know there’s been a few posts about this before, but it’s been a month since the last one and it’s still ongoing. It doesn’t seem that any of the LW admins responded to Zag’s post on their help community, and the last response from lodion/Nath I’m aware of was from 4 months ago when there were outright federation failures as opposed to just lengthy delays.

@[email protected] posted a comment on the post from last month about the delays stating that it’s an issue on our end as our server isn’t keeping up. I’m not sure whether this is the case or not, and I’m not sure how to interpret the Grafana dashboard they linked to, but as it’s a new reply on an old post, I wanted to note it.

Current federation delays seem to be around 7 days. It doesn’t seem to be affecting posts themselves on Lemmy.world communities, but does affect all replies to them (even from users on other instances), and all upvotes on the posts. [Edit: on further investigation, this isn’t the case. The current delays are at least 13 days, and this does actually affect posts too]

I don’t want to sound too pushy, since the LW admins and Lodion/Nath are all volunteers, but I was hoping we might be able to get an update on what the cause is, and if it’s an issue in Lemmy itself, if anybody’s opened an issue on GitHub and the developers are aware.

(NB: I don’t interact that much with LW, so all of my testing has been on the Boost for Lemmy community.)

  • Zagorath@aussie.zoneEnglish
    5·
    6 months ago

    the “best” solution at the time would be moving the server to Europe

    I believe the NZ instance basically set up a proxy right next to the LW server, which catches the ActivityPub messages and forwards them in batches to another proxy located in NZ, which translates back into a Lemmy-legal synchronous format. Rather a convoluted process, but it seems to work:

    • Nothing4YouEnglish
      9·
      6 months ago

      yes, that’s about the second best option for the time being.

      it’s currently used by reddthat.com and lemmy.nz.

      disclaimer: i wrote that software.

      • Lodion 🇦🇺@aussie.zoneM
        5·
        6 months ago

        Is that software available for others to use? I’ve been busy lately, but Nath messaged me about this… I was traveling and forgot about it by the time I returned.

        I’m open to the idea of setting up an EU VPS to proxy LW traffic temporarily. But the best fix is of course for Lemmy to operate in a way that doesn’t cause this.

        • Nothing4YouEnglish
          4·
          6 months ago

          it’s open source: https://github.com/Nothing4You/activitypub-federation-queue-batcher

          I strongly recommend fully understanding how it works, which failure scenarios there are and how to recover from them before deploying it in production though. not all of this is currently documented, a lot of it has just been in matrix discussions.

          I also have a script to prefetch posts and comments from remote communities before they’d get through via federation, which would make them appear without votes at least, and slightly improve processing speed while they’re coming in through regular federation. this also doesn’t require any additional privileges or being in a position to intercept traffic. it is however also not enough to catch up and stay caught up.
          this script is not open source currently. while it’s fairly simple and straightforward, i just didn’t bother cleaning it up for publishing, as it’s currently still partially integrated in an unrelated tool.
          I previously tried offering to deploy this on matrix but one of my attempts to open a conversation was rejected and the other one never got accepted.