Archive of Our Own ^beta

2018 was another productive and busy year for the OTW Finance team. We continue to work in the background to make sure that the bills are paid, the record keeping is accurate, and standard accounting procedures are met. Preparations for the 2018 financial statements and audit are also underway!

And now, we present the budget for 2019 (access the 2019 budget spreadsheet for more detailed information):

2019 Expenses

Archive of Our Own (AO3)

• US$12,242.60 spent so far out of US$418,104.06 total this year, as of February 28, 2019.
• 81.2% of the OTW's expenses go towards maintaining the AO3. This includes the bulk of our server expenses—both new purchases and ongoing colocation and maintenance—website performance monitoring tools, and various systems-related licenses (access all program expenses).
• This year's projected AO3 expenses include US$110,000 in contractor costs for upgrades and security testing.
• Additionally, AO3 expenses include a significant server overhaul and expansion of roughly US$177,000. New virtualisation servers will replace the current equipment, making testing and deployment of code for the Archive much smoother. The old servers will be repurposed as storage for the new servers. Other new servers will expand the Archive's capacity to serve pages and search results to users with fewer connection issues.
• Other anticipated costs include US$33,000 for a Systems contractor, who will do some work for the Archive's hardware infrastructure.

Fanlore

• US$329.34 spent so far out of US$6,738.97 total this year, as of February 28, 2019.
• Fanlore expenses are mostly its allocation of server purchases, maintenance and colocation costs, in addition to Fanlore's web domain renewals (access all program expenses).
• The new virtualisation servers mentioned above will also serve Fanlore pages and store Fanlore wiki backups.

Transformative Works and Cultures

• US$275.00 spent so far out of US$1,833.00 total this year, as of February 28, 2019.
• Transformative Works and Cultures' expenses are the journal's website hosting, and publishing and storage fees (access all program expenses).

Open Doors

• US$91.82 spent so far out of US$936.17 total this year, as of February 28, 2019.
• Open Doors' expenses consist of hosting, backup, and domain costs for imported fanwork archives (access all program expenses).

Legal Advocacy

• US$0 spent so far out of US$5,000.00 total this year, as of February 28, 2019.
• Legal's expenses consist of filing fees and travel expenses for conferences and hearings (access all program expenses).

Con Outreach

• US$0 spent so far out of US$500.00 total this year, as of February 28, 2019.
• Budgeted expenses include US$100 for printing flyers and US$400 for other convention outreach activities on behalf of the OTW (access all program expenses).

Fundraising

• US$2,296.25 spent so far out of US$31,460.00 total this year, as of February 28, 2019.
• Our fundraising expenses consist of transaction fees charged by our third-party payment processors for each donation, thank-you gift purchases and shipping, and the tools used to host the OTW's membership database and track communications with donors and potential donors (access fundraising expenses).

Administration

• US$2,785.99 spent so far out of US$50,471.80 total this year, as of February 28, 2019.
• The OTW’s administrative expenses include hosting for our website, trademarks, domains, insurance, tax filing, and annual financial statement audits, as well as communication and accounting tools (access all admin expenses).

2019 Revenue

• The OTW is entirely supported by your donations—thank you for your generosity!
• We receive most of our donations each year in the April and October fundraising drives, which together should account for about 67% of our income in 2019. We also receive donations via employer matching programs, royalties, Amazon Smile, and PayPal Giving Fund, which administers donations from programs like Humble Bundle and eBay for Charity. If you'd like to support us while making purchases on those websites, please select the Organization for Transformative Works as your charity of choice!
• Given your generosity in previous years, we have a healthy amount in reserves, which we plan to keep in store for a rainy day. Thanks to this, we can explore alternative revenue sources to supplement your donations. The Finance team has made progress in its search for a suitable investment method for a small non-profit organization like the OTW and aims to fully set up a low-risk, conservative investment portfolio by the end of 2019.
• The reserves also help bolster us in years when we’re planning larger than usual purchases. As mentioned previously, we have plans to replace old server hardware over the next few months as well as purchase new server hardware, which significantly increase our expenses for this year. Some of the forecasted expenses outlined in this budget may not take place before the end of the year, due to various constraints. To account for this uncertainty, even though all expenses we anticipate are described in the budget spreadsheet, we have projected a withdrawal of US$96,000 from our reserves to cover the amount corresponding to the less likely expenses. This additional amount may be withdrawn from reserves on an as-needed basis during the year. Our budget update, which will be published in October, will contain more precise information about which expenses will take place in 2019.
• US$23,840.99 received so far (as of February 28, 2019) and US$419,475 projected to be received by the end of the year.

Got questions?

If you have any questions about the budget or the OTW's finances, please contact the Finance committee. We will also be hosting an open chat to answer any questions you may have. This chat will take place in our public chatroom on April 6 at 8pm UTC(what time is that in my timezone?).

Here is the link to the chatroom! Come chat with us, and bring your questions!

To download the OTW's 2019 budget in spreadsheet format, please follow this link.

The OTW is happy to announce that the Archive of Our Own (AO3) will appear as a finalist in its category at the 2019 Hugo Awards. The Archive is nominated under "Best Related Work." This is the first time that an OTW project will appear on the final Hugo ballot.

The Hugo Awards were founded in 1953 and are the world’s pre-eminent science fiction awards. Each year, members of the World Science Fiction Convention (WorldCon) vote for the best sci-fi literature, film, television, and fanworks of the preceding year. Last year’s winners included N K Jemisin (for The Stone Sky), Ursula LeGuin (for No Time to Spare) and the creative team behind the Wonder Woman movie.

The "Best Related Work" award recognizes works in the fields of science fiction, fantasy, or fandom that do not fit into the other Hugo categories. AO3's continual evolution makes it eligible for this annual award. We would like to take this opportunity to thank those voters who included AO3 amongst their nominations this year. It means a lot to us as an organization and, in particular, to all the OTW personnel who work so hard to develop and run AO3. We hope that you will continue to keep us in mind when it comes to casting your final votes!

This year’s WorldCon will be held in Dublin from the 15-19 August 2019. Hugo winners will be announced on August 18.

Soon we'll be replacing our current method for generating work downloads with the Calibre command line interface. Our current patchwork system of third-party tools and custom code comes with a number of bugs and limitations, and moving to a well-maintained and documented application like Calibre will enable us to better maintain and even enhance the download feature going forward.

Because this is an entirely new system, there will be some differences between old and new files. We've provided an overview of these changes and some previews below, but we're still testing these changes, so there may be some minor differences between the previews and the final versions. Additionally, exactly how a file looks and behaves may vary depending on your device, software version, and settings.

HTML & PDF

HTML files serve as the basis for all of our download files, so it was necessary to make a few tweaks to help Calibre create tables of contents for other download types. Most of these tweaks are invisible, but we did make one visible change: the work title is now included at the top of the file, above the URL. Aside from these minor changes, HTML downloads will remain the same.

Because generating PDF downloads with Calibre would require us to install additional software on our servers, we will continue to use wkhtmltopdf to create PDFs, although we have upgraded it to a newer version. Like HTML downloads, PDFs will now include the title at the top of the first page but otherwise remain the same.

EPUB & MOBI

The biggest changes will be to MOBI and EPUB files, which are currently very different from each other due to limitations that existed when our system was created nearly ten years ago. The new code will give these files a more consistent look and feel and allow readers to take advantage of some features ebook readers provide.

EPUB

The new EPUB files will do away with the two-column display of work meta (i.e. tags and statistics), which doesn't work well for small screens, lots of tags, or large font sizes.

This update will also fix the bug where EPUB files were missing chapter endnotes.

You can download an example of the new EPUB files to preview on your device.

MOBI

Our current system provides a list of links on the first page as a makeshift table of contents, but Calibre will create files designed to work with your device or app's built-in table of contents function. (As you navigate with the shiny new table of contents, you may also notice each chapter begins on a fresh page.)

Calibre will also add margins to prevent the text from running against the edge of the page. Additionally, we've switched from using indented book-style paragraphs to using vertical margins to create blank lines between paragraphs, bringing MOBI files in line with how text is formatted on the site and in other download files.

This update also brings a major improvement for internationalization: MOBI files of works written in non-Latin alphabets will no longer be automatically transliterated! Please note, however, that MOBI offers only limited support of right-to-left languages like Hebrew and Arabic, so Kindle users will need to use AZW3 to read works in those languages.

You can download an example of the new MOBI files to preview on your device.

AZW3

We'll also be adding a new download format: AZW3, which was created by Amazon for use with Kindle devices and apps. It's a newer format than MOBI, and it will allow us to offer Kindle users downloads of works in right-to-left languages, but it will otherwise look the same.

AZW3 downloads will be available on all works, but if you rely on Send to Kindle to transfer files to your device, you'll need to stick with a different format: Amazon doesn't allow you to send this file type. However, you can still transfer AZW3 files to your Kindle by downloading them to your computer and connecting your device.

If you're using the Kindle app on an Android device, you can move AZW3 files you've downloaded to your SD card's Kindle folder in order to access them. Unfortunately, there doesn't appear to be a workaround for the iOS Kindle app, so iPhone and iPad users will need to use another download format or a different, non-Kindle app.

Deploy date

While this upgrade won't immediately resolve all of the existing issues with downloads, it's an important part of the process, and it will ultimately make the Archive more secure and more maintainable. (After the deploy, our release notes will have more detail on specific bugs this change has fixed.)

We're still in the testing stages, but we plan to deploy these changes sometime after February 11. When the time comes, we'll announce it on our @AO3_Status Twitter account.

No downtime is expected, but if you have a work already loaded in your browser at the time of the update, you'll need to refresh the page to download it.

In the coming weeks, we will be updating the code that powers our login system. When we make this change, all Archive users will be logged out of their accounts.

To avoid permanently losing access to your account, make sure you know your AO3 password and can access the email address associated with your account. If you forget your password and your email address is not up to date, you will not be able to request a password reset! (Edit, December 25: If you're sure you know your password and you regularly receive emails from us, you don't have to do a thing.)

Please take a few moments and follow these steps to check or update your account's email.

We will not deploy these changes until after December 13 to give users time to update their information. Additionally, we will give at least 48 hours notice on the AO3_Status Twitter account and our Tumblr, as well as on the Archive itself.

If you find yourself logged out and need help accessing your account, please do not leave your email address in public comments! Instead, you can contact Support about technical issues (e.g. password resets not arriving), or get in touch with the Policy and Abuse team, who can assist you in verifying that an account belongs to you.

Our previously announced search engine upgrade will not only provide some much-needed stability improvements, but also bring exciting changes to the way you search, filter, and browse for works and bookmarks on the Archive! These changes range from making current functionality more user-friendly to adding new options and overhauling existing features. We hope you enjoy these improvements to the way you find new things to read or look at or listen to!

Work search and filtering

Over the years, you've sent us a lot of suggestions for enhancing work search and filters, and as fellow Archive users, we're excited to finally implement some!

Detail of the new work filters, focusing on tag exclusion, crossover handling, completion status, and word count. The full screenshot (linked) shows the whole list of tag types to include or exclude, and additional options such as a date range and work language.

• Exclusion filters: Now you can filter things out of work listings the same way you filter them in: just enter their names in the "Other tags to exclude" field or choose them from a list of the most popular tags. This might have been our most requested feature, and we think our solution combines familiarity, flexibility, and accessibility. The new fields are only available in the filters for now, but you can keep using the old tag exclusion method on the Work Search page -- we'll be updating that interface soon!

• Crossovers: You can now search for crossovers, or choose to exclude them. Crossovers are defined here as works with at least two unrelated fandom tags, as determined by how tags are wrangled.

• Work Status: Previously, you could opt to show completed works only. Now you can also search specifically for works in progress, if you're into that kind of thing.

• Date Updated: To limit your results to works posted or updated in a particular time period, the filters now include a date picker to help you choose start and end dates for your query.

• Word Count: Lastly, we've added a simple way to look for works in a particular word count range. \o/

Everything you select in the filtering sidebar will continue to narrow down your set of results (i.e. AND filtering), as opposed to including more works that might have one or more of your selected tags (i.e. OR filtering).

Bookmark search and filtering

With the old bookmarking code, wrangling changes to large tags could put a strain on the Archive's servers. Thankfully, Elasticsearch 6 provides a much better way to handle our over 74 million (!) bookmarks, so we've given the code an extensive rewrite. However, in addition to giving bookmarks shiny new exclusion filters, we've had to make some changes to searching, browsing, and filtering bookmarks.

A truncated list of bookmarks, all of the same work, listed under the Veronica Mars tag. The full screenshot (linked) shows the five most recent bookmarks of the work, with bookmarker's notes and tags, and a link to access all bookmarks.

• Redesigned bookmark listings for tags and collections: No more scrolling past hundreds of bookmarks for the same popular work or series when you're browsing bookmarks in a collection or tag -- each bookmarked item will now be listed only once, with the details of its five most recent bookmarks beneath it. (If an item has more than five bookmarks, there will be an "All Bookmarks" link so you can check out the rest.) User bookmark pages and Bookmark Search results will continue to list bookmarks the way they always have.

• Bookmarker's tags: With the new code, using the "Angst" tag on a bookmark will no longer add that item to the bookmark page for "Angst," nor will the item be included in the results when filtering a list of bookmarks for the Additional Tag "Angst." Now if you want to see everything bookmarkers have tagged with "Angst," you'll need to use the "Bookmarker's tags" field on the Bookmark Search page.

  If you use the tags for a personal rating system or reminders to comment on a bookmarked work, for example, you can still filter your bookmarks just as before! There will also be "Bookmarker's tags" checkboxes and autocomplete fields on the bookmark pages for collections and tags.

• Searching bookmarks: While it's still possible to search bookmarks, the performance improvements we've made required us to divide the search into two fields: one for information on bookmarked items (e.g. work tags, titles, and summaries) and another for information on the bookmarks themselves (e.g. bookmarkers' tags and notes).

Detail of new bookmark filters, showing exclusion options for tags on the work and tags added by the bookmarker. Further options include search fields for work and bookmark information, and checkboxes to only look for recs or bookmarks with notes. The full screenshot (showing all filterable tags) is linked.

People search

You can now search for fellow users who have created works in your favorite fandoms! \o/ The new "Fandom" field in our People Search form suggests canonical tags as you start typing, so you don't have to worry about getting the name of your fandom exactly right. The more fandoms you put in, the more you'll narrow down the set of results.

The new People Search form, focusing on a search by fandom. The pseud testy has 970 works and 66 bookmarks, including 1 work for the fandom in question. Additionally, one can also search by username or profile content.

To make your pseud stand out in the search results, you can follow these instructions on editing pseuds to add a little blurb and user picture. (You can also add links and information about yourself, your fannish history, transformative works policy, or social media presence to your account profile, as laid out in our Profile FAQ.)

Limitations

Since even a state-of-the-art search engine like Elasticsearch suggests some gentle limits to the number of search results displayed (by throwing an error after a point), you will only be able to sift through 5,000 pages at a time, which means 100,000 works or bookmarks.

Keep in mind, however, that the Work Search form lets you change how results are sorted. So if you want to access the oldest works in a fandom -- something that wouldn't show within the first 100,000 results in a big fandom -- you can sort by Date Updated in Ascending order to make sure you get the results you’re looking for! (To sort a filtered list of works, use sort:>posted in the "Search within results" field.)

Some things to try

Once the new search is enabled for everyone, here are a few suggestions for getting to know the new options:

• Rating: For a list of works rated General Audiences or Teen and Up, don't select a rating to include -- instead check "Explicit", "Mature", and "Not Rated" to be excluded, and all your results should be safe for work! (You can reverse this for your bedtime reading.)

• Word Count: To get all the actual drabbles in your chosen fandom, enter 100 in both the "From" and "To" fields. (#DrabblePurist)

• Crossovers: Are you a big fan of canon-verse fic? You can eliminate all AUs by excluding the "Alternate Universe" tag, and crossovers or fusions by selecting the "Exclude crossovers" option.

• Date: Working on your list of favorite podfics of 2017? Go to the Podfic tag and put 2017-01-01 and 2017-12-31 in the "From" and "To" fields under "Date Updated", to refresh your memory!

• Status: You've consumed all the completed works for your shiny new OTP? You have skipped the works in progress, because you needed a happy ending now? Nothing left to read? Well, gird your loins and check the "Only incomplete works" ticky to take a second look at those!

At long last, we have upgraded our search engine, Elasticsearch, from version 0.90 aaaaaaall the way to 6.2, which had obviously been long overdue. A lot of other emergencies kept cropping up over the years, and there were never enough volunteers around to handle such a massive code rewrite while also managing the day-to-day upkeep of the Archive. With the help of our contractors (thanks to your generous donations!), we are now getting ready to deploy these changes to the Archive.

For a short period of time, the new search will only be available to a few volunteers for some final testing; everyone else will still see the current search and filtering interface. You might experience some issues in the days before we switch everyone over, as we had to tweak our old code a bit to get both versions of Elasticsearch running. Please bear with us, this will be temporary.

Once we are ready, we will move all accounts to the new system in batches, while monitoring how it holds up under increasing pressure. We currently expect this process to take at least a week, and longer if we run into any problems we have to stop and fix.

As always, we will keep you updated on progress here and through our @AO3_Status account on Twitter. (We'll have a bigger post with more details coming, but you can find a tiny preview below.) Many, many thanks to all the coders and testers who helped carry this over the finish line!

As people get ready to celebrate the end of 2017 in one way or another, we'd like to thank our users for their patience and supportive comments as we navigated downtimes, spam problems, and bumpy infrastructure upgrades together. We accomplished a lot of what we had hoped to tackle this year, and added a lot more to the to-do list for next year. Thanks for sticking with us!

As you might know, we've had to disable invite requests for existing users (due to abuse by spammers) and decrease the number of invites we automatically send out from our queue (ditto). As a result, people have had to wait to create an AO3 account for longer than we'd like. So, for the holidays, we're giving 1 shareable invite to every existing account that:

• currently doesn't have any invites, and
• is older than half a year, and
• has left at least 10 comments, or posted at least 1 work

(Sorry, we had to ensure we don't accidentally let the spammers invite all their spammer friends, so some restrictions apply.)

Check out our FAQ (available in a whole lot of languages) to read up on how to send an invite. You can either email the invite code, or copy-paste the code to share it with people through other means. In that case, our FAQ contains some information on how to use an invite code to create an account.

Happy gifting!

For the past several weeks, the Archive of Our Own has been dealing with an unusually high volume of spam works created to advertise live streams of sporting events. We've taken a number of steps to deal with these spam works, and in the next few days, we'll take one more: automatically hiding works that our spam detection service believes are spam.

We've been using this spam detector behind the scenes for a while now, and it has a 99.3% accuracy rate when it comes to identifying spam works and comments on the Archive. This means there is a small chance some non-spam works will be incorrectly marked as spam.

If your work is marked as spam, we'll send you an email to let you know. Our Abuse team will review your work as soon as possible and unhide it if it isn't a spam advertisement. Works will not be reviewed for other Terms of Service violations while in the spam queue.

We apologize to any users whose works are incorrectly marked as spam and to everyone who is currently waiting for us to review and fulfill their invitation request. We realize this situation is frustrating, but these steps are necessary to help us deliver a spam-free experience to all our users.

Thank you for bearing with us, and if you see any spam that has escaped our net, please report it to Abuse!

Update on November 24, 00:40 UTC: Automatic hiding of spam works is now enabled.