Between 19:45 UTC and 19:50 UTC, there was a mistake in how information was stored temporarily (cached) on Beehaw. This mistake could have allowed some people to see and use other people’s accounts without permission.
If you were using the website during that time, please check that your account settings and email address are still correct. Also, make sure that any posts or actions you made during that time are still connected to your account.
It’s important to note that we don’t have any proof that this error was actually used by anyone to do anything bad during the short time it happened.
Do we know what could have caused this cache error?
Yes; configuration settings for the web server involving improving performance. Those settings have been changed back to the previous, non-issue ones. So this should not occur at this time, or again.
Is this a mistake that’s easy to do for an inexperienced instance admin or just a consequence of too much fiddling and shouldn’t be an issue for other instances?
Was a result of too much fiddling. Attempting to gain even better performance from a bottleneck issue due to recent user influx. It was not an error in the Lemmy instance or Lemmy-UI but rather the web server front-end misconfiguration.
Thanks for working through these issues and improving performance of the website! Very appreciated. I’ve been tempted recently to create my own Lemmy instance, was this a problem with an nginx configuration option? How much does Beehaw deviate from a standard Lemmy deployment?
Feel free to answer vaguely if you don’t feel comfortable with giving away the details : )
Basically, this was
proxy_cache_key
being configured incorrectly. If you don’t use theproxy_cache
you should be fine.The only thing we changed from the norm is
ulimit
s and some nginx settings. If we figure out what works well, we’ll probably create a post about how to host lemmy. If you stick to the defaults, you’ll be mostly fine if your instance isn’t as big as Beehaw’s.Im also considering to setup nginx caching for lemmy.ml. Did you find a configuration which works?
Not yet. Session tracking in Lemmy is pretty hard to proxy, I’ll have to dive into the code to figure out why.