Intermittent site slowness

Post Header

Published:

2019-11-15 23:54:27 UTC

Tags:

Over the last few weeks, you may have noticed a few brief periods when the Archive has been slow to load (or refusing to load at all). This is because our Elasticsearch servers are under some strain. We've made some adjustments to our server setup, but because this is our busiest time of year, we expect the problems to continue until we're able to have new servers delivered and installed in a few months.

We've been planning this server purchase for a while now, but the machines we wanted -- AMD's Epyc Rome CPUs, which have an increased core count and are cheaper than the Intel equivalent -- didn't come on the market until August. Now that they've been released, we're working on finding the best price to help us make the most of your generous donations. We expect to order them very soon.

While we're waiting for our new servers, we plan to upgrade the Elasticsearch software to see if the newer version offers any performance improvements. We hope this upgrade and the changes to our server setup will keep things from getting much worse during our end-of-year traffic influx.

Thank you for your patience, and for all the donations that allow us to buy new hardware when these situations arise!

Update 27 November: The servers have been ordered, but it will still be a few months before they are delivered and installed.

Actions

Comments

Sorry, this news post doesn't allow comments.

Pages Navigation

Swirl Sun 17 Nov 2019 02:04AM UTC

Thank you for the update! It's good to know what's going on. :)

Comment Actions
- Thread
Kilodelta Sun 17 Nov 2019 08:06AM UTC

Glad to know you're on top of the issue, though sad it's Elasticsearch that's the problem - not least of which because _I work for Elastic_ and I am a massive user of your site. I admit it, I'm a weeb.
It sounds like you have some decent plans moving forward, but I wanted to at least let you know that Elastic has a special program for non-profit orgs like OTW that would get you a 16GB Cloud Standard Elastic Cloud cluster for a year at a time with the opportunity to renew. AFAIK there's not currently a program for granting discounts to support subscriptions, but perhaps having the cloud cluster would let you test upgrading versions, or other enhancements you could make to improve site performance?
In any case, if you're interested, let me know! I sent you a contact us message with more details, as well.

Comment Actions
- Thread
1. zz9pzza Sun 17 Nov 2019 09:03AM UTC
  
  Elasticsearch is not the problem, elasticsearch is great piece of software which makes the tagging possible. The issue is that we are more and more often hitting the cpu ceiling of the machines which are in our cluster. It doesn’t help that the machines we originally bought for our cluster are now only capable of supporting replicas and not running the main query load. The machines running our query load presently are not identical and spreading load is hard to do well ( they are mostly old mysql servers that have been repurposed ).
  We currently have 128 cores/256 hyperthreads of cpu in our fast area and we have about 100GB of primary data in the production indexes. We have talked to Elastic in the past about support however the price was quite expensive for the number of nodes we want to have. We do pay for support for some software we rely upon however we do receive large discounts and as you know we rely on donations ( In the past some people have complained about us buy support on any piece of software ). We do have a staging instance of the whole Archive stack and we do experiments there but as I am sure you are aware performance is a really hard area to test.
  I have asked that your message is sent through to the systems team.
  
  Last Edited Sun 17 Nov 2019 09:04AM UTC
  
  Comment Actions
  - Thread
  - Parent Thread
sovara Sun 17 Nov 2019 09:36PM UTC

Honestly, I didn't notice anything going slower than usual. But it's really nice to know what's happening. Thank you •~•

Comment Actions
- Thread
bad_pony_bellboots Mon 18 Nov 2019 05:02AM UTC

y'all take your time and dont feel bad about any site slowness! we can all cope on our end.

Comment Actions
- Thread
neinyagukie Mon 18 Nov 2019 11:16AM UTC

oh i thought it was bec of our internet connection lol. but that may also be the case too since we have a slow ass connection :( fighting tho! thanks for ur hardwork 💜

Comment Actions
- Thread
TabethaRasa Mon 18 Nov 2019 04:57PM UTC

Thank you for the update and the explanation! Definitely appreciate everything you all do to keep the site going as it grows.

Comment Actions
- Thread
verushka70 Mon 18 Nov 2019 08:33PM UTC

Oh, thank dog! I thought it was my stupid Comcrap 'high speed' Internet service. I'm just glad to know it's not on my end.
Thank you kindly for all you do. I couldn't contribute this last time, not in a good financial place now/for a while, but I have in the past, and I will in the future.
This site is so wonderful. The best place to find fanfic I want to read without worries. (It even slipped past my job's adult content blocker for 2 years, heh!) If I bookmark fic (on AO3, or on my browser, or both!), that fic will still be there when I come back to it.
Plus with AO3, we never have to deal with advertisements (omg, it's like websurfing with Netscape in 1994. *loves* <3<3<3; fic-purging by new corporate who change sites' TOS (*eyes LJ* *then eyes Tumblr*); periodically enforcement of usually unenforced site rues resulting in huge purges of fic (*side-eye's fanfiction.net, the last site I would ever go for fic*); loss of individual fen's pages (geocities); ; a site/fic archive with code that falls out of compatibility/compliance with new versions of OSes/browsers/web standards until its usability is a fraction of what it once was (*sighs, thinking of dSA* *smiles with happy sigh and gratitude, thinking of Open Doors dSA import to AO3*).
AO3 does none of these things, and I know it never will. And, oh yeah, AO3 has imported whole other near-defunct fanfic archives and saved them. AO3 is like the Wayback Machine of fandom, or trying to be, except without the fear that even Wayback no longer has a site archived. AO3 is amazing. I came here countless times, all the time, long before I ever got an invite, because there's just so much here. And it keeps growing. It's a staggering amount. And even small fandom fen with no central location for their few fanworks and maybe a few individual sites/blogs and/or an inactive group comm, can find each other and the fic they want, here.
AO3 is a wonderful little island fandom paradise, isolated from a polluted ocean of pop-ops; bots; trackers; click-bait; dozens of irritating, annoying, distracting animated or gify ads, including automatic video pop-ups (that scroll as you scroll *grrrrr!s like Lana on Archer*); social media misanthropy and trolling; and 'contextual advertising' that follows you all over the web (unless your browsers are set to private/delete history/cache every time you exit). AO3 is, for me anyway, a current embodiment of the way the web used to be, in those very early years. Fen don't understand why I use ad-/tracker-blockers, Ghostery, Noscript, etc. I know I seem paranoid to some, but that's only because they've never known any Internet experience without all that crap. AO3 takes me back there. I love, love, love it. Again, thank you for all you do. It IS appreciated.

Comment Actions
- Thread
1. Jeslieness Mon 25 Nov 2019 04:31AM UTC
  
  Well said!
  
  Comment Actions
  - Thread
  - Parent Thread
KestrelHill Tue 19 Nov 2019 07:38PM UTC

Thanks for the update! Next time some goober wants to know "what AO3 does with all that moeney when they're supposed to be non-profit" (oy veh) I'll point them here.

Comment Actions
- Thread
elentiriel Fri 22 Nov 2019 02:09AM UTC

Thanks for letting us know! I know nothing about the hardware and software components, but I'm not surprised this time of year is the busiest and therefore things can be a bit slow. NaNoWriMo, Yuletide, all the holiday exchanges, people rereading past fics vfor inspiration, etc... For the most part I haven't had much trouble, just the occasional slow loading.

Comment Actions
- Thread
HerschelLopez Sat 23 Nov 2019 07:20AM UTC

I think AO3 is a very good software,it methodically handles a variety of things.While I was waiting for the invitation nervously,it was still kind enough to give me any hints that might reassure me.All English allows me to learn English well in the process of creation.AO3 has done well since its establishment,and it will get better and better in the future

Comment Actions
- Thread
JayTDawgzone9999 Mon 25 Nov 2019 01:20AM UTC

Thank you for doing what you do and keeping everyone updated on what happens.

Comment Actions
- Thread
Account Deleted Tue 26 Nov 2019 08:40PM UTC

Thank god, I thought something was wrong with my service plan.
We will all wait patiently and graciously.
<3
Right people? Pls no bitching

Comment Actions
- Thread
1. LOL OTW stans (Guest) Thu 28 Nov 2019 10:26PM UTC
  
  Who appointed you kindergarten director? Fuck off.
  
  Comment Actions
  - Thread
  - Parent Thread
2. 1. Account Deleted Fri 29 Nov 2019 04:36AM UTC
    
    I’m sorry if I offended you. I was just getting tired of seeing people bitching about their internet while these people are actually trying to fix it.
    Have a good day.
    
    Comment Actions
    - Thread
    - Parent Thread
  2. V (Guest) Sat 30 Nov 2019 12:07PM UTC
    
    Go outside or something Jesus Christ
    
    Comment Actions
    - Thread
    - Parent Thread
scifigeek14 Thu 28 Nov 2019 03:35AM UTC

Thank you for the clarity and explanations

Comment Actions
- Thread
hydrogenbetty Thu 28 Nov 2019 03:37AM UTC

come to think of it, i have been having some trouble loading. thanks for acknowledging things and fixing them!

Comment Actions
- Thread
izukillme Thu 28 Nov 2019 03:38AM UTC

That's amazing to hear! I personally haven't experienced any problems, but that may be because of my time zone and being on when most others aren't as someone said in the comments section ahah. This website is the single best thing to happen to fandom - it has created a beautiful community which I'm so proud to be part of :33 None of us have a single right to complain when we know how hard you all are working to keep this amazing Archive up and running. Thank YOU for your hard work; you don't need to thank us for being patient, it's the least we can do for you <3

Comment Actions
- Thread
lumos (Guest) Thu 28 Nov 2019 03:38AM UTC

Thanks for all your efforts to provid us a safe place of our special article！

Comment Actions
- Thread
cathrheas Thu 28 Nov 2019 03:39AM UTC

F

Comment Actions
- Thread
sheron Thu 28 Nov 2019 03:40AM UTC

<3

Comment Actions
- Thread
swazao3 Thu 28 Nov 2019 03:43AM UTC

Thank you for your hard work!! Super interesting hearing the behind the scenes of AO3. Would love to know the reasoning behind owning physical servers vs moving into AWS/other cloud services?

Comment Actions
- Thread
1. foolishgames Sat 30 Nov 2019 07:20AM UTC
  
  If someone else owns the servers, then that someone else gets to dictate what content is and is not allowable. "The cloud" just means "someone else's servers". A core, driving principle of the archive - from literally the first pie-in-the-sky lj post going "wouldn't it be cool if" - was we need to own the servers. We need to be completely independent.
  Anyway, the size of this site means hosting in the cloud would be a nightmare.
  
  Comment Actions
  - Thread
  - Parent Thread
AFWebster Thu 28 Nov 2019 03:49AM UTC

Thanks for the update. I haven't noticed any problems myself, but I'm glad to see you being transparent about the issues and the solutions.

Comment Actions
- Thread

AO3 News

Post Header

Actions

Comments

Pages Navigation

Swirl Sun 17 Nov 2019 02:04AM UTC

Comment Actions

Kilodelta Sun 17 Nov 2019 08:06AM UTC

Comment Actions

zz9pzza Sun 17 Nov 2019 09:03AM UTC

Comment Actions

sovara Sun 17 Nov 2019 09:36PM UTC

Comment Actions

bad_pony_bellboots Mon 18 Nov 2019 05:02AM UTC

Comment Actions

neinyagukie Mon 18 Nov 2019 11:16AM UTC

Comment Actions

TabethaRasa Mon 18 Nov 2019 04:57PM UTC

Comment Actions

verushka70 Mon 18 Nov 2019 08:33PM UTC

Comment Actions

Jeslieness Mon 25 Nov 2019 04:31AM UTC

Comment Actions

KestrelHill Tue 19 Nov 2019 07:38PM UTC

Comment Actions

elentiriel Fri 22 Nov 2019 02:09AM UTC

Comment Actions

HerschelLopez Sat 23 Nov 2019 07:20AM UTC

Comment Actions

JayTDawgzone9999 Mon 25 Nov 2019 01:20AM UTC

Comment Actions

Account Deleted Tue 26 Nov 2019 08:40PM UTC

Comment Actions

LOL OTW stans (Guest) Thu 28 Nov 2019 10:26PM UTC

Comment Actions

Account Deleted Fri 29 Nov 2019 04:36AM UTC

Comment Actions

V (Guest) Sat 30 Nov 2019 12:07PM UTC

Comment Actions

scifigeek14 Thu 28 Nov 2019 03:35AM UTC

Comment Actions

hydrogenbetty Thu 28 Nov 2019 03:37AM UTC

Comment Actions

izukillme Thu 28 Nov 2019 03:38AM UTC

Comment Actions

lumos (Guest) Thu 28 Nov 2019 03:38AM UTC

Comment Actions

cathrheas Thu 28 Nov 2019 03:39AM UTC

Comment Actions

sheron Thu 28 Nov 2019 03:40AM UTC

Comment Actions

swazao3 Thu 28 Nov 2019 03:43AM UTC

Comment Actions

foolishgames Sat 30 Nov 2019 07:20AM UTC

Comment Actions

AFWebster Thu 28 Nov 2019 03:49AM UTC

Comment Actions

Pages Navigation