Intermittent site slowness

Post Header

Published:

2019-11-15 23:54:27 UTC

Tags:

Over the last few weeks, you may have noticed a few brief periods when the Archive has been slow to load (or refusing to load at all). This is because our Elasticsearch servers are under some strain. We've made some adjustments to our server setup, but because this is our busiest time of year, we expect the problems to continue until we're able to have new servers delivered and installed in a few months.

We've been planning this server purchase for a while now, but the machines we wanted -- AMD's Epyc Rome CPUs, which have an increased core count and are cheaper than the Intel equivalent -- didn't come on the market until August. Now that they've been released, we're working on finding the best price to help us make the most of your generous donations. We expect to order them very soon.

While we're waiting for our new servers, we plan to upgrade the Elasticsearch software to see if the newer version offers any performance improvements. We hope this upgrade and the changes to our server setup will keep things from getting much worse during our end-of-year traffic influx.

Thank you for your patience, and for all the donations that allow us to buy new hardware when these situations arise!

Update 27 November: The servers have been ordered, but it will still be a few months before they are delivered and installed.

Actions

Comments

Sorry, this news post doesn't allow comments.

Pages Navigation

Seamy_Infamy Sat 30 Nov 2019 12:50AM UTC

Thanks for making us all aware of this and your plans to try and rectify it. You guys do an amazing job keeping this gem of a site running and your hard work is hugely appreciated!

Comment Actions
- Thread
PennamePersona Sat 30 Nov 2019 06:25AM UTC

This is all really good to know...I appreciate AO3 being upfront about what this issue is and their plan + current situation! Thanks for the transparency, and I hope everything goes well with the servers!! I appreciate the work that the archive and OTW does so much, and I'm definitely glad (though not surprised) to know that serious consideration goes into server maintenance.

Comment Actions
- Thread
TrippNessa (Guest) Sat 30 Nov 2019 06:23AM UTC

I appreciate you explaining what's going on! I totally understand, thank you for the info =) You guys have always been very forthcoming with information about any situations going on. I often find out about a problem from reading about it before I even experienced it myself lol. That's the case with this even! Thanks again.

Comment Actions
- Thread
LawlessQ Sat 30 Nov 2019 07:06AM UTC

I mean it’s a free site, one of the only sites that actually consistently allows people to give feed back, shows statistics over which is your most popular fanfics, gives you an email right on time to let you know when you can create your account/ lets you check your positioning while you are waiting. This is one of the only places that seems to help people gather a consistent following and has easy editing capabilities/ tag systems. Even better you don’t have to sign up to comment on people’s shit. So yeah I don’t care, keep up the work. I can wait. I’m not going to have an aneurysm over this. Shit breaks that’s what happens and I appreciate having a place to write my fanfics. I mean I’m not complaining no way can I do a better job than the people running this place, if it weren’t for AO3 staff there wouldn’t be a site. Otherwise I mean where else would I be/go? Do what you gotta do, my dudes.

Comment Actions
- Thread
MooredMermaid Sat 30 Nov 2019 07:14AM UTC

Thank you for the update and for explaining the issue to the users. I appreciate the work going into keeping the AO3 as maintained as possible in the meantime.
I hope that you all have a happy holidays and for the Americans a belated happy Thanksgiving.

Comment Actions
- Thread
justina (Guest) Sat 30 Nov 2019 10:12AM UTC

nice

Comment Actions
- Thread
justina (Guest) Sat 30 Nov 2019 10:11AM UTC

I think it's great

Comment Actions
- Thread
justina (Guest) Sat 30 Nov 2019 10:12AM UTC

🌚🌚nice

Comment Actions
- Thread
AstaraelWeeps Sat 30 Nov 2019 10:17AM UTC

Good luck you guys! Love you for all the hard work that keeps this running. Owe you my happiness, I really do.

Comment Actions
- Thread
hera_not_juno Sat 30 Nov 2019 01:48PM UTC

Thank you all for doing everything you do to keep the site running!
What you guys do for free is so awesome, and I think the community here can do it's best to be understanding and encouraging though the rough spots. I saw a post on Tumblr reminding us to download our fics to keep off the site so it works better for those who can't, for anyone scrolling through the comments.

Comment Actions
- Thread
Ryan Scott Sutherland (Guest) Sat 30 Nov 2019 03:52PM UTC

www to the moon and back

Comment Actions
- Thread
Sophrederick Sat 30 Nov 2019 05:11PM UTC

I love that I have such a great site to read on that I don’t even have to pay for. Thank you so much for all that the AO3 and OTW volunteers do for us!!

Comment Actions
- Thread
Kaitie J (Guest) Sat 30 Nov 2019 05:26PM UTC

Thank you for doing your best and for continuing to provide this service to us!!! I would rather have it load at 6 kb for eternity than go back to what I was doing before this. Best holiday wishes !!

Comment Actions
- Thread
heineken (Guest) Sat 30 Nov 2019 07:16PM UTC

hey! I'm Heineken, if you prefer just Hei, i don't know how to start but... i'm a writter who lost a lot of my works and i want to them be read for people like and if i don't post this and nobody know what i do in my stories, be hard to published this. i really want share this with the world. If is soooo hard to understand what is cuz' i am a brazilian and don't know so much of your language. thanks for your attetion!

Comment Actions
- Thread
wildmachinery Sat 30 Nov 2019 08:38PM UTC

Thank you for the update and for everything you do! I really appreciate all the hard work you put into keeping this site up and running. 💕

Comment Actions
- Thread
aProgrammer (Guest) Sun 01 Dec 2019 02:37AM UTC

using es for storage only is costy.
are there any other options than using es only to store all these articals?
what about store tags + indexes + titles etc in es, and articals and comments in kv into hbase?

Comment Actions
- Thread
1. james_ (AO3_ADT) (Official) Sun 01 Dec 2019 07:53AM UTC
  
  Hi, aProgrammer!
  We use Elasticsearch for searching and filtering, not storage. For storage we have a cluster of 3 MySQL servers running at 1 million queries per minute as well as a Memcached cluster that runs at about the same rate.
  james_
  Archive development team
  
  Last Edited Sun 01 Dec 2019 07:54AM UTC
  
  Comment Actions
  - Thread
  - Parent Thread
2. 1. aProgrammer (Guest) Sun 01 Dec 2019 08:52AM UTC
    
    :D
    
    Comment Actions
    - Thread
    - Parent Thread
  2. aProgrammer. (Guest) Thu 05 Mar 2020 01:33AM UTC
    
    Hi, james_
    Thanks for answering my questions about storage.
    I am a backend developer in China,I have been seeing AO3 from 2009 but I haven’t donated anything and did not contribute to the opensource code… Perhaps this is a good chance to start contributing….
    Fangirls in China have been discussing start running another website on their own. There have been suggestions given about running a proxy with a new domain that is not blocked by that firewall, but these fangirls insists for a new website, some user end needs still needs to be more clarified.
    I can see GitHub project https://github.com/otwcode/otwarchive as AO3’s backend project. I think there are not too many problems if someone use this project and start another website with another domain? Like permissions from OTW?
    
    And I see AO3’s public account posts something like servers are installed, does this mean that all the storage and proxies are running on actual servers instead of on something like AWS or Azure? I think it might because it is more convenient to maintain some machines yourself?
    And I’d like to ask about these database structures and elasticsearch data structures? Now that China’s firewall blocks Chinese users, what’s the query/qps like now?
    Is AO3 using CDN for JS files and pictures?
    I am thinking maybe letting advertisement in and asking for donation is two different ways, that these two things cannot happen on one website at the same time? Does AO3 publish all the donation incomes and usage for these money?
    …I will be grateful for you answering all these questions, perhaps the answer is already published on fanhackers page…
    
    Thanks,
    aProgrammer, Mischa
    
    Comment Actions
    - Thread
    - Parent Thread
  3. 1. james_ (AO3_ADT) (Official) Wed 18 Mar 2020 06:12AM UTC
      
      Hi, Mischa!
      If you want to explore developing on the Archive, I would recommend using the Vagrant image. Our contributing guidelines have more information to help you get started.
      
      We are an open-source project, so you do not have to ask permission to start a new website using our code. (However, you'll need to avoid using the same color, overall look, name, and logos.) Please note that we'll be continuing to work on the otwarchive code, so you may want to pull in our updates from time to time. The fewer changes you make to the code, the easier this will be. You're welcome to submit pull requests for changes you make, but we generally only merge pull requests for the issues listed in our bug tracker.
      
      We chose to use our own servers instead of a service like AWS for two reasons:
      
      1) We believed we would be more resilient and less liable to be deplatformed if we owned our servers rather than using someone else's.
      2) Using a cloud provider is cheaper when you are small, as the amount it costs to host a website is small, compared to the minimum needed to own your own servers. As the number of page views goes up, hosting your own servers is more economic if you have a free (or cheap) sysadmin. For example, a db.r5.24xlarge MySQL server on AWS costs US$552 per day. We have three MySQL servers with twice the RAM of a db.r5.24xlarge. Each of these servers cost about US$35,000.
      
      Looking at a Sunday evening, we are seeing about 22,000qps for MySQL, which is about the same for memcached and about 2,500 for Redis. Elasticsearch queries are more complex and sit around 170/second.
      We use AWS for user icons, but all CSS and JavaScript comes from our own frontend servers (we have four).
      We do not use advertisements, both because of our philosophy and also because being dependent on advertisers opens up to pressure from those companies or their customers to disallow certain content.
      We do publish all the donations and income and those can be found on our budget posts. (Note that we spent a fair amount on servers after the last report, which will be visible in the next budget post.)
      
      I posted a talk I gave a number of years ago which may still be useful to you, although it is a little outdated: The Technical Architecture of the Archive of Our Own.
      
      Last Edited Wed 18 Mar 2020 06:13AM UTC
      
      Comment Actions
      - Thread
      - Parent Thread
ShinriBakumu Sun 01 Dec 2019 03:30AM UTC

It's completely alright! These things happen and we understand! For a non-profit website, the quality is amazing! Don't stress, just relax and have a nice holiday season! :)

Comment Actions
- Thread
shelikesmyspark Sun 01 Dec 2019 04:57AM UTC

This may sound weird but my dad developed and now runs the Elasticsearch application/company. I would totally ask him for any help for you guys but it would come at the cost of my moral standings with him. RIP

Comment Actions
- Thread
DragonLady_WinterFlame Sun 01 Dec 2019 05:05AM UTC

I didn't notice any slowness on my end, like ever. So I guess it's just for some people, but hopefully it gets updated so that those people's internet starts running fast again. Plus I do remember seeing donations for something a few years ago, so I guess that's gonna be there for us users to help pay off the hardware purchase? Cause when I saw someone's comment say that this site has always been "free", I was afraid we users would lose that freedom and start paying everytime we go on this site, but I'm sure what thought would be stupid and that it would piss off a lot of people. So I assume by "purchase" you mean you guys need the hardware to update the site, and hence the "donations" to pay off the update purchase, correct? Cause I like this fanfiction site, man, even though I hardly ever write fanfiction I do do it when I have an interesting story in my head and want to express it. Excuse the long comment, but just thought I'd post my opinion.

Comment Actions
- Thread
1. Lady_Bellatrix Sun 01 Dec 2019 11:59PM UTC
  
  Well, this site has always been free, and will always be free. This is not-for-profit afterall. And AO3's philosophy is always non-commercial since it's inception.
  But buying servers obviously needs money. Other AO3 philosophy is actually-own-the-server. So AO3 needs to buy new servers as this site expands in size and usage. The donation is mostly used for the operations. Buying, mantaining, and running the servers.
  
  Comment Actions
  - Thread
  - Parent Thread
2. 1. DragonLady_WinterFlame Tue 03 Dec 2019 09:23AM UTC
    
    Oh okay, thanks for clearing that up with me. I feel more calm about the situation now. ;3
    
    Comment Actions
    - Thread
    - Parent Thread
B (Guest) Sun 01 Dec 2019 06:52AM UTC

The site will likely go down during the upgrading.

Comment Actions
- Thread

AO3 News

Post Header

Actions

Comments

Pages Navigation

Seamy_Infamy Sat 30 Nov 2019 12:50AM UTC

Comment Actions

PennamePersona Sat 30 Nov 2019 06:25AM UTC

Comment Actions

TrippNessa (Guest) Sat 30 Nov 2019 06:23AM UTC

Comment Actions

LawlessQ Sat 30 Nov 2019 07:06AM UTC

Comment Actions

MooredMermaid Sat 30 Nov 2019 07:14AM UTC

Comment Actions

justina (Guest) Sat 30 Nov 2019 10:12AM UTC

Comment Actions

justina (Guest) Sat 30 Nov 2019 10:11AM UTC

Comment Actions

justina (Guest) Sat 30 Nov 2019 10:12AM UTC

Comment Actions

AstaraelWeeps Sat 30 Nov 2019 10:17AM UTC

Comment Actions

hera_not_juno Sat 30 Nov 2019 01:48PM UTC

Comment Actions

Ryan Scott Sutherland (Guest) Sat 30 Nov 2019 03:52PM UTC

Comment Actions

Sophrederick Sat 30 Nov 2019 05:11PM UTC

Comment Actions

Kaitie J (Guest) Sat 30 Nov 2019 05:26PM UTC

Comment Actions

heineken (Guest) Sat 30 Nov 2019 07:16PM UTC

Comment Actions

wildmachinery Sat 30 Nov 2019 08:38PM UTC

Comment Actions

aProgrammer (Guest) Sun 01 Dec 2019 02:37AM UTC

Comment Actions

james_ (AO3_ADT) (Official) Sun 01 Dec 2019 07:53AM UTC

Comment Actions

aProgrammer (Guest) Sun 01 Dec 2019 08:52AM UTC

Comment Actions

aProgrammer. (Guest) Thu 05 Mar 2020 01:33AM UTC

Comment Actions

james_ (AO3_ADT) (Official) Wed 18 Mar 2020 06:12AM UTC

Comment Actions

ShinriBakumu Sun 01 Dec 2019 03:30AM UTC

Comment Actions

shelikesmyspark Sun 01 Dec 2019 04:57AM UTC

Comment Actions

DragonLady_WinterFlame Sun 01 Dec 2019 05:05AM UTC

Comment Actions

Lady_Bellatrix Sun 01 Dec 2019 11:59PM UTC

Comment Actions

DragonLady_WinterFlame Tue 03 Dec 2019 09:23AM UTC

Comment Actions

B (Guest) Sun 01 Dec 2019 06:52AM UTC

Comment Actions

Pages Navigation