AO3 News

The Archive now supports a whole lot more characters!

Post Header

Published:
2018-12-14 00:52:04 UTC
Translations:
Tags:

🎁

Thanks to a long-planned database upgrade, the AO3 now lets you use all the latest emoji, as well as extended character sets for several languages (such as some of the rarer characters used for Chinese and Japanese) in your works, comments, or profile. 🎉🍾🎈🎊🎇. In fact, all text fields now support the UTF8MB4 character set, except for a few we've limited to Latin characters and some punctuation marks, like pseuds or collection names.

🔧

UTF8MB4 encoding only became available a few years after the Archive's inception, so we already had a pretty sizeable database using the older encoding when the upgrade became inevitable. After our recent server work, which included an upgrade to MySQL 5.7, people trying to post a work or comment with an unsupported character (e.g. one of the newer emoji) would be served an unhelpfully blank error page, exacerbating the problem.

However, migrating our entire 617GB database would have been a massive undertaking, likely necessitating several days of downtime. Fortunately, we were able to come up with a game plan that allowed us to keep downtime to a minimum by making changes to a new, empty copy of a database table while the "old" table with all the data was still in use. Once the new table was ready, we copied the data over and put the new table to use, then moved on to the next table.

😱

Thanks to the diligent efforts of our volunteer sysadmin james_, we completed the work in a little over a week and with no data lost. However, we ran into an unexpected problem with one of the constraints we put on our database to prevent the creation of multiple tags with identical names.

Due to a change in the way data is compared, some characters such as ss and ß are now treated as identical. That means that when we copied a tag like "Weiss Kreuz" to the new system, and then a few seconds later tried to copy "Weiß Kreuz," the second tag would be considered a duplicate and get quietly dropped. This made works originally posted to the dropped tag hard to find for a few days, since the tag didn't appear on the work anymore.

Once we discovered the issue and tracked down the cause, we were able to use a database backup to reconstruct the changes we'd made up to the point of the final swap, so we could compare the two tables and restore the lost tags.

👍

Together with our tag wranglers, we have put all the affected tags back into the system, so they will now appear on their works again. We are sorry if your works or bookmarks were affected by this issue!

Even though our upgrade is now complete and our database allows all of these new characters, keep in mind that some devices and browsers, as well as apps and e-readers relying on the MOBI format, might not display them properly. However, we hope you'll enjoy all your new emoji options and leave some extra-colorful love for creators in your comments! ❤🧡💛💚💙💜