Archive of Our Own ^beta

We're super-excited to announce that we've just taken delivery of the brand-new servers for the Archive of Our Own! \0/ Our new, vastly expanded server setup gives us more power, more memory, and more room for expansion, and we are THRILLED to be able to have them. They have been purchased with the generous support of fandom - thanks to all our awesome members and donors!

What we're buying

Check out Meet the Machines! for full details on our server family. Our original server setup had just two servers: a main database server and a backup. We're adding three more servers to this, plus a storage server and a switch, so our new setup will look like this:

Money matters

We bought our servers from an awesome company iXSystems, who specialise in open source applications and were really responsive to our needs. They gave us a great deal: our DB server was US$5,554, and the Rails twins were US$4,193 each. The storage server was US$3,014, and the switch that lets them all talk to each other was US$280. In total we spent US$17,234 on new hardware, which is a great price for the kind of machines we've invested in.

Buying new servers also means paying more hosting costs - we own the machines, but we hire space for them with a colocation company who hook them up to the internet. When we had just two servers, we were paying US$228 per month for our 2U of rack space and bandwidth. With the new servers, we'll be paying US$800 per month for 8U of rack space plus extra bandwidth. Because our new servers are such powerful beasts, the company are running a brand new power cable just for us!

All of this investment was paid for by donations from fans! The AO3 is funded by donations to our parent org, the Organization for Transformative Works; the success of our last membership drive, which raised US$18,308.46, gave us the financial confidence to make this investment. Thank you to all our supporters!

If you'd like to support the ongoing development of the AO3 and the OTW, you can donate at any time. A donation of just US$10 makes you a member of the OTW (if you so desire) and confers voting rights in Board elections. Donations over US$50 also qualify for a variety of nommy premiums.

What this means for Archive users

More speed! More features! More fanworks! Once we're up and running with this setup, we'll be much better able to handle high loads, so you'll be seeing much less of our sad 502 page! We'll also be able to start thinking seriously about hosting multimedia fanworks - we still have work to do before we can start hosting fanart, but it's definitely in sight now! We'll also be able to start moving towards implementing Subscriptions on the Archive - one of our most-requested features.

Server installation: Archive downtime

In order to install our new servers and move to the new setup, we will have extended Archive downtime. We're still coordinating with our colocation company, but we're expecting to have two periods of downtime in the week beginning 30th January. All Archive user accounts will be emailed as soon as we know the exact timing (now is a good time to check that your registered email is still correct) and we'll be posting on our news blogs and to our Twitter accounts @ao3_status and @ao3org as more news arrives.

Server naming festival

These servers belong to fandom, so it's only right that fandom should get to name them! During our downtime, we'll be running a server naming festival and soliciting suggestions for names for our servers (and their trusty switch). Meet the Machines to find out what they are like and start thinking about what they should be called! More information on the competition coming soon!

SERVERS YAY!

We are EXCITED! Thank you, fandom!

Thanks to the wonderful generosity of fandom, we've been able to buy a beautiful new set of servers! Meet the machines below!

The DB Server - aka "The Beast"

Our new database server is a serious badass. Built for us by open-source server specialists iXSystems, the DB server is an iX1204 powered by 2 latest-generation Intel 6-Core X5650 processors, and 48GB of RAM. This server will also have 2 Intel X25-M 80GB SSDs (solid-state drives, which have no moving parts) to make reads off the database crazy fast.

The Rails Twins

A set of identical twins, our speedy and strong new Rails servers will split the load of running the Archive app. These two are from the same family as the DB server: iX1204 models, each with 2x Intel Quad Core E5620 processors, 24GB of RAM, and 4x146GB 15k RPM hard drives.

The Originals

Our trusty veterans! The original servers are another set of identical twins: Hewlett Packard ProLiant DL360 G5 models each powered by 2 Quad-Core Intel Xeon E5420 processors. They each also have 16GB RAM, and 4×72GB 15k RPM hard drives. For the last year, these two have had to carry the entire load all by themselves, but now they will both be moving into new roles:

1. One of these machines will now serve as our friendly memcache server and it runs background and other odd jobs.


2. The other one will become the DB's slave. *koff* (Of course the more vanilla-minded among us can think of it as being the DB's loyal sidekick!) It will also run our search engine.

The Storage Server

The QNAP TS-809U-RP is a 2U rackmount Network Attached Storage (NAS) appliance, intended for what the manufacturer describes as "massive data sharing" -- mmmhmm! We're loading this baby to capacity, with 16TB (8x2TB Hitachi Deskstar 7200RPM HDDs) in hot-swap bays to give us greatly increased data storage/backup capability.

The Switch

All of our fabulous servers will communicate with one another through an HP Procurve V1910 Gigabit Layer 2+ smart-managed network switch. The switch has 16 auto-negotiating 10/100/1000 ports and 4 true Gigabit SFP ports.

Es ist wieder jene Zeit des Jahres, in der Fans sich am Geschichtenwichteln beteiligen, Glühwein trinken und das Eigene Archiv verfluchen, weil es wieder mal so verdammt langsam ist. Denn nach einem anstrengenden Tag in der Innenstadt, in der man sich durch Menschenmassen drängeln und an der Kasse ewig warten musste, gibt es nichts Nervigeres als mit der Fehlermeldung Error 502: Page did not respond in a timely fashion konfrontiert zu werden, wenn man zum Entspannen auf seine liebste Fanfiction klickt.

Bitte glaubt uns: Wir wissen, wie das ist. Wir sind auch Fans und benutzen das Eigene Archiv, und nach einem Tag voller Schlangestehen oder dem Programmieren einer neuen Funktion stört uns auch nichts mehr als ein Fehler 502, wenn wir zum Entspannen auf unsere liebste Geschichte klicken.

Deshalb möchten wir euch wissen lassen, dass wir etwas dagegen tun. Wir arbeiten gerade an einem neuen Systemaufbau (dank eurer Unterstützung mitsamt schicker neuer Server) der die Leistung deutlich verbessern wird.

Wir haben eine Menge Vermutungen angestellt, bevor wir die letzte große Aktualisierung durchgeführt haben. Ein wichtiger Teil dieses Updates war die Einführung von Rails 3, das die Grundlage für die Umgestaltung des Systemaufbaus bildet. Sowohl währenddessen als auch kurz danach haben wir schon einige Leistungsverbesserungen auf den Weg gebracht, aber letztendlich gibt es leider keinen Ersatz für die Erkenntnisse, die wir aus den tatsächlichen Nutzungsdaten des Archivs gewinnen.

Wir haben nun 11.000 Nutzer und Nutzerinnen und 120.000 Werke und sind damit wesentlich schneller gewachsen, als wir uns jemals erträumt haben.

Was genau wird sich durch den neuen Aufbau verändern?

Derzeit läuft das Archiv auf zwei Servern – einer für die Datenbank, der andere für die Rails-Anwendungen.

Die neue Systemarchitektur wird in etwa so aussehen:

Zum Vergrößern auf die Grafik klicken

Kurz gesagt werden wir auf einen Systemaufbau umsteigen, der es uns erlaubt, in der Zukunft leicht zusätzliche Hardware zur Leistungsverbesserung anzuschließen.

Wir kaufen gerade die neuen Server für diesen Aufbau, aber im Moment der Zeitpunkt für eine so drastische Veränderung ungünstig. Der Umstieg auf einen neuen Systemaufbau ist keine triviale Sache, und wir verschieben ihn deshalb lieber aufs neue Jahr, wenn wir mehr Zeit für Ausfälle haben und nicht mehr so viel los ist. Im Eigenen Archiv herrscht derzeit reger Betrieb (nicht nur durch Yuletide, auch wegen anderer Wichtelaktionen) und wir wollen nicht riskieren, dass uns das alles um die Ohren fliegt.

Daneben ist Yuletide, unser erster und größter Testfall, in diesem Fall Teil einer langfristigen Lösung. Tausende Nutzer und Nutzerinnen, die alle zur selben Zeit auf dieselben Teile des Archivs zugreifen wollen, werden unsere Systeme bis zum Anschlag belasten und uns helfen herauszufinden, wo die größten Engpässe liegen. Das wiederum zeigt uns, welche Prioritäten wir beim Kauf der Hardware setzen müssen.

Das heißt aber auch, dass im Moment alles etwas langsam läuft. Das tut uns sehr leid. Auch aus diesem Grund ist das AO3 immer noch in der Betaphase. Wir werden das “Beta”-Schild auch nicht herunternehmen, solange wir nicht einigermaßen sicher sind, dass das System mit möglicherweise Millionen von Nutzern und Nutzerinnen und Geschichten umgehen kann, ohne dass uns alles um die Ohren fliegt oder nur im Schneckentempo läuft. Und soweit sind wir noch nicht.

Wir müssen euch also um Geduld bitten. Yuletide und andere Geschichten-Tauschaktionen – und die daraus resultierende intensive Nutzung – sind für das AO3 und diejenigen, die es programmieren und verwalten, auf lange Sicht sehr wertvoll. Es ist alles Teil des Betaprozesses – habt also bitte Verständnis dafür, dass solche Hänger nötig sind, um das Eigene Archiv langfristig voranzubringen.

Esta es la época del año cuando los fans participan en en intercambios de fics, beben sidra fermentada y maldicen a Un Archivo Propio por ser tan @#$!% lento. Porque después de un día de lidiar con multitudes o esperar en las cajas de las tiendas departamentales, nada te frustra más que ver un Error 502: Page did not respond in a timely fashion en tu fic favorito para confortarte.

¡Creénos, lo sabemos! También somos fans y usamos el AO3, y después de un largo día luchando contra las multitudes festivas o programando, nada nos frustra más que nos salga un Error 502 en nuestro fic favorito de confort, tampoco.

¡Queremos que sepas que estamos haciendo trabajando en esto! Actualmente estamos en el proceso de armar una nueva arquitectura de sistema — incluyendo la inminente adquisición de nuevos y brillantes servidores, gracias a tu ayuda — la que ayudará a mejorar nuestro desempeño de forma considerable.

Podemos (y lo hemos hecho) hacer montones de presunciones al realizar nuestro último lanzamiento, el cual incluyó la actualización a Rails 3 que era necesaria antes de proceder con la remodelación de los sistemas, y dimos lo mejor para mejorar el desempeño tanto en el proceso como inmediatamente después del lanzamiento. Pero al final, no hay substituto a observar resultados en el uso real cotidiano del sistema.

En este momento contamos con 11.000 usuarios y 120.000 trabajos hospedados: ¡estamos creciendo más rápido de lo que nunca habíamos soñado!

¿Qué es lo que va a suceder con este nuevo diseño?

Actualmente estamos trabajando con dos servidores — un servidor para una base de datos individual y otro servidor para la aplicación Rails.

La nueva arquitectura se verá más o menos así:

Haz click en la imagen para ampliar

Básicamente, estamos moviéndonos a un diseño donde podríamos, en un futuro, fácilmente conectar máquinas adicionales cuando se requieran, para mejorar el desempeño.

En este momento estamos comprando los nuevos servidores para implementar esta nueva arquitectura, pero no creemos que sea una buena idea implementar un cambio de sistema tan grande como este en este preciso momento. La implementación de esta nueva arquitectura no es trivial y planeamos hacerla en una forma más controlada y con mucho más espacio para tiempo fuera de línea cuando las cosas se calmen entrando el año. Están sucediendo muchas cosas en el AO3 – no sólo Yuletide, sino montones de otros intercambios festivos de fic – y realmente no queremos que todo nos explote.

Pero también, en este proceso, Yuletide – nuestro principal y más grande caso de prueba – es parte de una solución a largo plazo. Tener miles de usuarios, todos golpeando a las mismas secciones del archivo al mismo tiempo y estirando nuestros sistemas al límite, nos ayudará a identificar donde están los cuellos de botella más grandes y eso nos dirá cuales deberán ser nuestras prioridades al comprar el nuevo hardware.

Esto significa que las cosas están lentas ahora. Y estamos muy apenados por eso. Es parte de las razones por las que AO3 aún está en proceso de prueba o beta. No quitaremos el aviso de “Beta” hasta que no estemos razonablemente confiados de que el sistema puede manejar millones de usuarios e historias, sin caerse, explotar o tener grandes estancamientos. ¡Aún no llegamos a eso!

Así que les pedimos paciencia. Yuletide y otros intercambios – y el resultante uso intenso – son tremendamente útiles a largo plazo para AO3 y sus diseñadores y administradores de sistema. Es parte del proceso de prueba – ¡por favor entiendan que todo esto es necesario para que nosotros llevemos a AO3 a donde necesita estar!

This post is also available in Deutsch and Español.

This is the time of year when fans participate in fic exchanges and drink mulled cider and curse out the Archive of Our Own for being so @#$!% slow. Because after a day of fighting holiday crowds or waiting on lines in department stores, nothing bums you out like getting Error 502: Page did not respond in a timely fashion on your favorite comfort fic.

Believe us: we know!  We're fans too, and we use the AO3, and after a long day fighting holiday crowds or coding a new feature, nothing bums us out like getting Error 502 on our favorite comfort fic, either.

We want you to know we're doing something about it! We're currently in the process of putting together a new systems architecture -- including the imminent purchase of several very shiny new servers thanks to your support -- that will boost performance substantially.

We could (and did) make lots of guesses going in to our latest big deploy, which included the Rails 3 upgrade we needed to make before we proceeded with the systems revamp, and we did our best to put in performance improvements both on the way and immediately after the deploy. But there is no substitute in the end for looking at actual results from real-world usage.

We're now at 11,000 users and 120,000 works: we've grown much faster than any of us ever dreamed!

So what exactly is going to happen with this new design?

We currently are running on two servers -- a single database server and a single Rails app server.

The new architecture will look something like this:

Click image to see larger

Basically, we are moving to a design where we can easily in future plug in additional machines as needed for performance improvements.

We are currently buying the new servers to implement this architecture, but we don't think it's a really good idea to implement a major system change like this right now. Implementing this new architecture isn't trivial, and we plan to do it in a more controlled way and with a little more leeway for downtime once things slow down in the new year. There's a lot going on at the AO3 - not just Yuletide, but loads of other holiday fic exchanges - and we don't really want the whole thing to go boom.

But also, in this process, Yuletide, our first and biggest test case, is part of the long-term solution. Having thousands of users all banging on the same parts of the archive all at once and stressing our system to the limits helps us identify where the biggest bottlenecks are, and that in turn tells us what our priorities need to be when we buy our hardware.

That means things are slow now. And we're very sorry about that. It's part of why the AO3 is still in beta. We won't be taking the "beta" sign off until we have reasonable confidence that the system can handle potentially millions of users and stories, without falling down and going boom or having major slowdowns. But we are not there yet!

So we need to ask for your patience. Yuletide and other exchanges - and the resulting intense usage - are tremendously useful in the long run for the AO3 and its designers and sysadmins. It is all part of the beta process - so please understand it is all necessary for us to get the AO3 to where it needs to be!

Happy birthday to us, happy birthday to us, happy biiiiirthday dear servers, happy birthday to us! The beautiful, beautiful servers of our own are one year old! We're incredibly proud of the fannish energy and investment that enabled us to buy our beautiful servers - thank you fandom!

Birthday presents

As our birthday gift to fandom, we've launched an AWESOME code release. Release 0.8.1 includes skins, mobile exports, and a very special guest logo!

As part of our birthday goody bag, we're also giving every Archive user an invitation so that everyone can invite a friend to the party! If you go to your user home and check out 'Manage invitations' you'll be able to see and use your invitation. You can also request more if you need them!

If you don't already have an Archive account, you don't have to be left out! Just add your email address to our Invitations Queue and you should receive an invitation within a few days. Everyone who was in the queue after we released our birthday code was issued an invitation right away!

Birthday party!

We're having a party to celebrate, starting now! Drop into our Public Discussion chatroom anytime today to join the party - Just head for this URL and type in the name you'd like to represent you, and you'll be in: https://fanarchive.campfirenow.com/28473.

Singing happy birthday!

All birthday girls deserve some praise! Throughout the party, we'll be talking about what Archive staffers and users love the most about the Archive -- we'll round up a sample of these, and post them for everyone to enjoy and add their own thoughts to! Everyone's killer features in one place!

Also posted on the OTW News Blog