Comment on An Update from the OTW Board of Directors and Chairs

  1. Still, deafening silence!

    Despite not being a member, I have emailed the Board in my capacity as a stakeholder. I'd encourage others to do the same.

    Good afternoon,

    I am an AO3 user who first registered in 2009 and has posted 1.9m words of fic to the site in that time. I care about AO3's existence very much, and though I have criticized it in the past, I continue to use and value the site; as a web developer, I also understand the social and technical effort that goes into maintaining the site.

    I am writing to you to say that I find it unconscionable that you have not addressed the credible allegations, made by azarias, that the Board and Legal have behaved in a negligent manner, the net effect of which has been endangerment and deterioration of quality of life for azarias and other volunteers.

    According to allegations made public, the Board:

    1. Was informed of at least one CSAM (child sexual abuse material) attack prior to the org-wide attack in 2022, and did not warn volunteers or prioritize the establishment of safeguards to reduce the org's potential attack surface and the trauma of volunteers receiving CSAM in their inboxes
    2. Was repeatedly asked for tools to support PAC's more difficult work and declined to provide them, despite the OTW regularly exceeding its funding goals and operating with a surplus, and despite the tools in question being well-known and industry-standard
    3. Regularly behaves in manners which violate the internal Code of Conduct, including behaving in belittling and xenophobic ways towards Chinese volunteers
    4. Made their BLM statement in 2020 under duress and with no intention of following through on stated commitments
    5. Permits the Legal committee to overrule Policy & Abuse (PAC) on matters which documented delegation states are part of PAC's purview, including permitting the display and distribution of material which may be CSAM, and forbidding PAC from proactively removing material which credibly appears to violate US law regarding CSAM
    6. Permits the Legal committee to arbitrarily remove and reinstate volunteers without oversight or justification into their decisions

    Firstly, I'd like to make it clear that I consider these actions morally abhorrent. They run counter to the OTW's mission statement. They are indicative of a profoundly damaging and abusive culture and reflect entrenched bias at the highest levels of the org, including racism and xenophobia.

    Secondly, I'd like to encourage the Board to take this issue seriously: to immediately move to curb Legal's power, to establish a ramp-off plan for all members of the Board and Legal, to apologize publicly to azarias for defaming her by implication, and to set a deadline (which you then meet) for complete reform of internal and external disciplinary, grievance, and decision-making processes, such that stakeholders and members can be assured that these abuses will not continue.

    The OTW is a non-profit registered in the US. As your own bylaws state, "Subject to the limitations of the Certificate of Incorporation, of the Bylaws, and the provisions of Title 8 of the Delaware Code as to action to be authorized or approved by the Members, and subject to the duties of Directors as prescribed by the Bylaws, the business and affairs of the Corporation shall be controlled by the Board of Directors." This means that regardless of the advice Legal has given you, and regardless of how unfair or inaccurate you may feel this framing of your internal issues is, you as officers of the corporation are legally responsible for the OTW's negligence and malfeasance. You are actively endangering every single volunteer who receives CSAM tickets. Your negligence after the first CSAM attack endangered every single volunteer who might receive CSAM spam and doxxing threats (all of them). This pattern of negligence is extremely alarming, not just because of its ethical and moral implications, but because of its legal implications. You have permitted AO3 to become a poorly moderated website where distribution of possible CSAM is subject to the whims of a Legal committee that lacks any relevant expertise in CSAM, cybercrime, informational security, Trust & Safety, or even contract law. If a volunteer were to sue you, the Board would be asked to explain why they, as officers of the corporation, behaved in such a negligent manner.

    I strongly encourage you to treat this situation with the seriousness it deserves. You have an opportunity to ensure the OTW can continue to serve fans for decades to come. Please don't waste it.

    Thank you,

    impertinence @ ao3

    Comment Actions
    1. While I'm here, anyone still reading this comment section should know that the org does not allow for suspension of a user caught trying to distribute CSAM.

      Relevant quote:

      There were several other cases where I had to carefully comb through the works and comments of a user after banning them for attempting to distribute CSEM, looking for other attempts that hadn't been reported. I had to read some disturbing stuff - usually fictional underage stuff does nothing more than normally squick me, but when I KNOW the writer was trying to pass around real CSEM? It's worse. A better policy would be to just nuke the entire account, which is what most sites do when you're caught with your hand in the CP jar, but "maximum inclusiveness of content" means the violations have to be carefully peeled away from the rest of the content. God forbid we delete one fic we could save.

      So, AO3 is likely harboring people who distribute CSAM/distributing CSAM itself, because their official policy is that distributing CSAM is against the rules but being a known CSAM distributor caught using the site to distribute CSAM does not result in immediate removal of the account and its content.

      Comment Actions
      1. I am utterly disgusted that once again this is proof of OTW personally kneecapping effective removal of harmful irl content such as CSAM and intentional harassment of real users (even when it takes MORE work to do so) and puts the volunteers AND CHILDREN in harm's way in order to follow the whims of the legal team who oversteps MASSIVELY. It's also notable that most of the legal team is focused on IP law, not obscenity or exploitation, meaning that they are not best educated on how to handle these materials properly. So of course, they have a bunch of random volunteers forced to look at unedited images of CSEM on their personal computers (Which could easily step out of some moral and legal bounds). AO3: the reason why your PAC team is overworked and underappreciated is that they're forced to do archaic and nonstandard things like this.

        They can't moderate properly or keep people safe when they're forced to do things like this.

        Comment Actions
        1. Agreed completely. It's appalling. This is a massive legal liability that's also morally disgusting and I genuinely do not understand how the Board and Legal can be so cavalier about it. I have no idea if they're even legally compliant wrt this policy for handling the material itself; they are definitely opening themselves up to serious allegations of negligence on the part of volunteer management. Like, it's just...horrifying.

          Comment Actions
          1. Idk the fullest extent of how the PAC team handles CSEM tickets, but it seems like from what ex-PAC have been saying... it's not well. And I have always wondered about the legality or infrastructure used for the PAC when handling said tickets. Are they mandated to report these activities to authorities, do they keep tabs on the people who are caught if they let them walk? Are they looking up these images on privately owned personal computers? Would OTW legally defend their own volunteers if this ended up causing problems with actual investigators? Because the idea of getting sent entire tickets of CSEM to personally look over with no physical proof of this being my job, and I see it, and remove it, but am not expected to report them widely or remove ALL possible entry points for distribution is... SCARY. It also opens up huge negligence allegation opportunities.

            Because it seems like they don't have an amazing track record of letting volunteers seek out and use external tools. And an even worse track record with defending their own volunteers from internal or external issues.

            Comment Actions
            1. All extremely valid questions that it seems like the Board has no interest in answering!

              In terms of volunteers, I do know they're using a ticketing system and have internal documentation about how picking tickets up works. So they're likely covered in terms of "why on earth did you ever view this material". US law is relatively permissive when it comes to mandating reporting of CSAM by providers, because they want providers to report and not be afraid of getting smacked for procedural violations. It sounds like their reporting process was within the bounds of the law, though really only by chance: if azarias and others in PAC didn't take it seriously, are there any mechanisms of accountability to ensure the material's removal? Doesn't seem like it! And even accounting for a relatively generous law around reporting, I would have questions about whether they are truly removing material "as soon as reasonably possible" given that the org refuses to support PAC with automation tools and more aggressive staffing. Which circles back around to "this really seems like blatant negligence which is a legal liability". Same with the "we'll deal with what's reported & look for more but not ban the distributor" policy. Seems like a pretty extreme failure of due diligence baked into the policy. Would love to know if anyone with actual relevant legal expertise vetted this!

              And then on top of that there's just the simple fact that the Board and Legal are asking volunteers to traumatize themselves for no fucking reason. Which is a liability in addition to being heartbreaking.

              Comment Actions
              1. In this case, it seems like they're putting more time and energy into being archaic. The lack of automation, not banning the distributor, etc. Seems like they're creating more work for themselves as opposed to there being less. This begs the question of "WHY"? Why spend more human unpaid manpower? More time playing whack-a-mole with distributors? Those things don't just seem inefficient it seems genuinely foolish, automation and manpower might not be up to them due to budget, but full suspensions + bans are. That is a choice entirely up to them. The fact that AO3 doesn't roll by what seems to be the wisest choice is actually sick. I would have to do two backflips over common sense to come to the conclusion that PAC should personally crawl this person's entire fic history and individually pick them instead of just reporting and suspending them entirely. The mental gymnastics needed to come up with that process is wild.

                Comment Actions
                1. The really depressing conclusion I'm coming to is that no one making decisions for the org (the Board, Legal) has any real understanding that CSAM is materially and legally very different from garden variety trollfic. It seems like a total context collapse perpetuated by people who wouldn't know accountability if it bit them on the nose.

                  Comment Actions
        2. a stylized AO3 cat

          Hi,

          This is incorrect. Any instance of CSEM reported to us, once investigated by PAC, will be deleted as fast as possible, the account that posted it will be permanently suspended, and an investigation will be conducted to determine if the individual has any sockpuppet accounts. If found, they are permanently suspended as well.

          Best,
          Ran
          AO3 Policy & Abuse

          Comment Actions
          1. Yes, azarias had given me more information later about how the process works. I will delete my comment so as to not cause confusion. However, is it true that said suspended accounts and their nonoffending works stay up and are visible to the public even when the account is permanently suspended? And that PAC are tasked with checking said works to make sure that they are clean to continue to host?

            Last Edited Thu 15 Jun 2023 05:59PM UTC

            Comment Actions
          2. AO3 prevents editing a thread once someone has replied to it, but I corrected myself in replies to others. I maintain that retaining content from the suspended account and requiring you to go through it is appalling negligence and a ridiculous policy.

            You're being taken advantage of by people with absolutely no ethics. You should quit.

            Comment Actions
      2. I think there's a bit of confusion in terminology. PAC absolutely did immediately and permanently suspend users we caught attempting to distribute or solicit CSEM. I can't imagine that has changed. Permanently suspended users are not able to upload content to the site, and may not make a new account to evade their suspension*.

        What we didn't do, when I was on PAC, was just nuke the account's contents. Any content they had uploaded that didn't violate the TOS could stay, which is why we had to review all of the content they had created for violations - we couldn't just nuke everything. This was ... unpleasant, to say the least. And I'm concerned about the liability it might open up the org to; I'm not a legal expert at all, but this really strikes me as a better safe than sorry situation. It makes sense to selectively remove only violations when someone has been suspended for, say, constantly posting find-a-fic spam, but not when they were attempting a felony. This really seems like an area where "maximum inclusiveness of content" should bend. Not for taste or morals, but for safety.

         

        * The tools for preventing suspended users from rejoining could be better. On the other hand, ban evasion is something every site with user-generated content has to deal with, and it's not an easy problem to solve. As a webdev, you can probably speak to this better than I can.

        There were also some holes in what suspension prevented users from doing. I don't know if those holes have been plugged, and I don't want to be more specific because I don't want to encourage exploits.

        Comment Actions
        1. Thanks for the clarification! That is less terrible than I thought but still pretty bad, because ultimately it still means that yeah, the due diligence falls on error-prone humans to ensure all offending content is removed. (And yall get traumatized in the bargain. Great. Would love to hear a justification for how this serves the org's mission whatsoever.)

          Last Edited Thu 15 Jun 2023 12:29AM UTC

          Comment Actions
        2. Oh, I see. Still really awful though. Does this mean that the content posted goes to Anon or does the account just go defunct and people are unable to log in, but the name and everything remains?

          Comment Actions
          1. Name etc remains. Able to login, but not to edit, delete, or upload most content.

            Comment Actions
            1. *Shady Hell's Kitchen Waterphone sound*

              Ummmm... why not just nuke it or hide it from public view until the authorities can see it like most other platforms? Why allow them to log back in? Why allow other people to access and see their account (and possibly find them on other platforms to continue distribution)? Ah naw. That's not right.

              Comment Actions
        3. Let me add some citations here so readers know where these policies come from. I'm going to be quoting the TOS (https://archiveofourown.org/tos) and the TOS FAQ (https://archiveofourown.org/tos_faq).

          The team may also permanently suspend users when it determines that such action is justified. Permanent suspensions for violations other than spam, violation of the Archive's Age Policy by Age-Barred Individuals, or threatening the technical integrity of the site require a majority vote of the team.

          "Age-barred individuals" means users under 13. "Threatening the technical integrity of the site" means uploading malware or something like that. Distributing CSEM is not on the list that lets PAC suspend without a vote.

          Legal gave us a workaround for this after the first couple of times, because obviously requiring PAC to all review and vote on CSEM was not viable. We took a preemptive vote in favor of suspending anyone we determined was distributing or soliciting CSEM, or trying to do so. It was a kludgy workaround, but the TOS is a stunningly naive document in many ways.

          Penalties are not retroactive: a suspended user's nonobjectionable Content will not be automatically removed unless the user is an Age-Barred Individual.

          That means PAC can't just zero out your account contents. Only the TOS violating content will go. (Also! While I was on PAC, we weren't allowed to delete content uploaded by an Age-Barred Individual, but only suspend them until they turned 13, which! Seems unusual! But tbh I had too much on my plate to fight about that one. Ask Legal, they never explained why to me.)

          The team's decisions are final unless overturned by the Board at the Board's sole discretion.

          Oh that's just a plain lie, Legal/Content could and did overturn PAC.

          And this explains the rational behind part of why PAC is like it is:

          What constrains the abuse team's discretion?

          Our commitment is to build a community that welcomes anyone with a willingness to learn the rules but defends itself against people who deliberately flout them. Our discretion is aimed at that objective. Procedurally, permanent suspensions for violations other than spam or threatening the technical integrity of the site require a majority vote of the abuse team. Majority rule builds in checks on individual discretion without trying to resolve every possible situation in advance.

          Basically, the org weights protecting users from PAC overreach more than protecting the site from users. There are historical fannish reasons for why this philosophy took hold, but it's not really a philosophy that can effectively govern a website the size and popularity of AO3. Trust and safety issues are much more complex and impactful than the fannish grudgewank the founders were focused on avoiding.

          Last Edited Thu 15 Jun 2023 12:37AM UTC

          Comment Actions
          1. thank for taking the time to clarify! Out of curiosity, has the TOS ever been updated? Primarily because the state of the internet when AO3 was created....is somewhat different now.

            Comment Actions
            1. The TOS was last updated 23 May 2018. TOS review and revision has been in progress for a couple of years, but the process was mysterious even while I was a volunteer.

              Comment Actions
          2. Thank you for the detail & citations.

            Earnestly, the biggest glaring hole I see in all this is a complete failure to protect yall, PAC. The moment this became an issue they should have moved to update the TOS as quickly as possible, if the TOS in its current form truly prohibits, say, "you have no rights under the TOS if you use our platform to commit a felony", which seems like a big loophole you'd want to close anyway.

            But setting all of that aside, it is breathtakingly awful that this is the environment they expect PAC to operate in. There's just no justification for it. Suspend posting of new works entirely till you push the changes through, who cares, creation of new fan content should never be placed above volunteer safety.

            Last Edited Thu 15 Jun 2023 12:49AM UTC

            Comment Actions
        4. "Any content they had uploaded that didn't violate the TOS could stay, which is why we had to review all of the content they had created for violations"

          Just for clarity's sake: Were you told to go through those accounts' contents and look for more violations, or did you choose to do that of your own accord? Like, were you told it was part your job to do that, or a "you can't nuke everything, we have to look at content on a case-by-case basis"? Cause there's a difference, and only one of these 2 scenarios seems reasonable.

          Comment Actions
        5. Interesting.
          Would you mind confirming that the person signing with your name on DW here and giving an explanation of "your" work slighter further down under a spoiler tag is, indeed, you?

          https://fail-fandomanon.dreamwidth.org/598363.html?thread=3666921307#cmt3666921307

          Cause if that is you, it makes me question (once again, cause you didn't answer me before), whether you really HAD TO review everything. As in, where you TOLD to go through content that hadn't been reported yet? Because, according to this anon claiming to be you, you permabanned people for legal content and removed/edited said legal content from the site, while your chair *looked the other way*. According to this anon, you moderated without permission on a site whose lack of liability hinges on them not moderating (aka looking into stuff that hasn't been reported yet).

          So please clarify. Thanks in advance.

          Comment Actions
      3. 1) TIL Franzeska was involved in writing the TOS and oh god that probably contributes why the board doesn't view racism as an actual problem 😐
        2) forgive me, I'm not a lawyer but doesn't retaining a member who was distributing CSEM and actively choosing to not suspend make AO3 liable for distributing CSEM as well 😬

        Comment Actions
        1. Azarias has corrected my understanding and made it clear PAC does suspend the user, but did not remove/render inaccessible (AFAIK there can be issues with immediately deleting CSAM from a legal standpoint, but that's no reason to leave it up publicly) the material the user posts. Which is still a huge issue, both because of the human cost of requiring volunteers to comb through the material, and because of the potential liability of leaving a vector of distribution just...up, hanging out, ready for the next monster.

          Comment Actions
        2. a stylized AO3 cat

          Hi,

          This is incorrect. Any instance of CSEM reported to us, once investigated by PAC, will be deleted as fast as possible, the account that posted it will be permanently suspended, and an investigation will be conducted to determine if the individual has any sockpuppet accounts. If found, they are permanently suspended as well.

          Best,
          Ran
          AO3 Policy & Abuse

          Comment Actions
          1. Ran, I doubt you like me very much right now, but you know I care about you and the rest of Sharkat. Please, before you say ANYTHING ELSE IN PUBLIC about what PAC does or doesn't do with CSEM data, read Rahaeli's thread about the possible legal requirements: https://twitter.com/rahaeli/status/1669350441971494914

            For example, "deleted as fast as possible" might be the wrong answer. Please, if possible, run this past a lawyer with experience in internet law who is not on the Legal Committee before you say anything else on this topic. You have possibly just admitted that the OTW is violating US law. Please, ask someone qualified.

            Comment Actions
      4. (Previous comment deleted.)

        1. It's not completely accurate, see explanation here: https://archiveofourown.org/comments/660613498

          Comment Actions
          1. The Queen of Shadows from Mirrormask

            Thank you. I have deleted my comment to avoid spreading confusion.

            Comment Actions
      5. A digital shoulders-up portrait of Jonathan Sims from the Magnus Archives. He is a bespectacled, gaunt man with light brown skin, chin-length wavy dark, graying hair, and many assorted scars. There is a cracked golden halo behind him.

        Thank you for sharing this. Absolutely horrifying.

        Comment Actions
      6. This is fucking ridiculous. CSAM constitutes so many crimes I can’t begin to list them all.

        The most important thing here is the children who may be harmed, but there’s no question this also jeopardizes the AO3’s legal standing.

        Comment Actions
        1. Yes, and in more than one way: failure to appropriately moderate content (why the fuck isn't the account's content removed when they are found to be committing a felony) and reckless negligence towards PAC volunteers.

          I would love to know if the members of the Legal committee and the Board who empowers them have ever had to look at the CSAM they're demanding PAC comb through, lest they inadvertently delete a pedophile's valuable contribution to the Archive Of Our Own dot org fanfiction corpus. I doubt it.

          Comment Actions
          1. Yeah, there are some contributions that are just not needed. I understand the urge to save everything, but you do have to ask at what point that becomes detrimental. Like I don’t save spam emails about how I can enlarge my nonexistent manhood in order to have a record of jobs I’ve applied for.

            Comment Actions
        2. a stylized AO3 cat

          Hi,

          This is incorrect. Any instance of CSAM reported to us, once investigated by PAC, will be deleted as fast as possible, the account that posted it will be permanently suspended, and an investigation will be conducted to determine if the individual has any sockpuppet accounts. If found, they are permanently suspended as well.

          Best,
          Ran
          AO3 Policy & Abuse

          Comment Actions
          1. Ran, I doubt you like me very much right now, but you know I care about you and the rest of Sharkat. Please, before you say ANYTHING ELSE IN PUBLIC about what PAC does or doesn't do with CSEM data, read Rahaeli's thread about the possible legal requirements: https://twitter.com/rahaeli/status/1669350441971494914

            For example, "deleted as fast as possible" might be the wrong answer. Please, if possible, run this past a lawyer with experience in internet law who is not on the Legal Committee before you say anything else on this topic. You have possibly just admitted that the OTW is violating US law. Please, ask someone qualified.

            Last Edited Thu 15 Jun 2023 06:26PM UTC

            Comment Actions
          2. You've confirmed in public that you're likely noncompliant with CSAM reporting laws in the US.

            Get out of there. Seriously. You're playing the dupe for people who don't care about you or PAC or the archive itself, it seems. Please pick your head up and recognize what's going on here.

            Comment Actions
          3. That’s nice. And is PAC provided with the necessary tools and support they need to do this, or do y’all just toss them on their asses and call them pedophiles?

            Comment Actions
      7. a stylized AO3 cat

        Hi,

        This is incorrect. Any instance of CSAM reported to us, once investigated by PAC, will be deleted as fast as possible, the account that posted it will be permanently suspended, and an investigation will be conducted to determine if the individual has any sockpuppet accounts. If found, they are permanently suspended as well.

        Best,
        Ran
        AO3 Policy & Abuse

        Comment Actions
        1. Ran, I doubt you like me very much right now, but you know I care about you and the rest of Sharkat. Please, before you say ANYTHING ELSE IN PUBLIC about what PAC does or doesn't do with CSEM data, read Rahaeli's thread about the possible legal requirements: https://twitter.com/rahaeli/status/1669350441971494914

          For example, "deleted as fast as possible" might be the wrong answer. Please, if possible, run this past a lawyer with experience in internet law who is not on the Legal Committee before you say anything else on this topic. You have possibly just admitted that the OTW is violating US law. Please, ask someone qualified.

          Last Edited Thu 15 Jun 2023 06:26PM UTC

          Comment Actions
          1. Magnus Bane and Alec Lightwood from the TV show Shadowhunters pressing their foreheads together

            If what Rahaeli is saying is true and everything under 18 U.S. Code § 2252A has to be reported and preserved, then... I think we did actually break that law. When I was on PAC there was a scandal over on twitter involving a hockey player who had, when they were a minor, had someone film them engaging in sex with someone, and then it got leaked onto twitter years later. It had been taken down pretty quickly but not before someone had made a work or comment on AO3 linking to the video on Twitter. I remember handling the case. Because the video had already been taken down by the time we even saw the reports since we got so many false child porn reports, and since AO3 didn't host the video, we just deleted the post or comment and I believe warned or suspended the person who had posted it. I don’t remember if we contacted Legal about it since obviously I wasn't allowed to contact Legal, that would have been Matty being a go-between. But it's possible we did tell them. This would have been back in 2018 or 2019.

            Comment Actions
            1. Holy fuck. Yeah no we got a problem here if that's the case.

              Comment Actions
            2. The Forest Queen, by @MachineGarden_ on Twitter

              Fingers crossed that procedures have improved since then.

              Comment Actions
        2. The Forest Queen, by @MachineGarden_ on Twitter

          Really hoping that "deleted" here means "hidden from public view for 90 says and THEN deleted."

          Comment Actions
        3. You are literally destroying criminal evidence by deleting the CSAM. Unless you mean by deleting that it is hidden and still preserved in the database to be turned over to investigators.

          Comment Actions
        4. It’s almost unbelievable that you posted this three whole times. Yikes.

          Comment Actions
        5. For those of you worrying about the US law mandating a 90 day preservation period: the data would be in the server backups, of which every major website would have. It's only deleted from the main website, not everywhere for ever.

          Comment Actions
          1. 1. Full backups are rarely retained for 3 months. I have never worked at an organization where this was the case.
            2. There is no evidence in this comment or anywhere that PAC and Systems have worked to ensure the backups are retained for an appropriate time period, much less labeled so that the data is easily accessible. (By this I mean, if you are removing offending content and investigating over a period of days or weeks, you need both the data and some form of log/tracking to ensure you know where the data is. "Oh, we deleted the first post on March 31...but found another April 2" Great, that's 2 separate backups, maybe more given that AO3 is very large and likely does not simply copy everything once per day. The problem compounds when you have more than one offender.)

            Comment Actions
            1. 1. Most sysadmins would retain backups in offline cold storage, but yes, it does not necessarily mean AO3 is doing the same thing. We should ask FrostTheFox.
              2. The negotiation between PAC, Legal and Systems about this is almost certainly a private conversation, so it's perfectly reasonable that you can't find the evidence for it. You don't need the data to be easily accessible by your average volunteer, these information requests don't happen very often (and by the looks of it, it hasn't happened on AO3 over the last 15 years), so it's acceptable for the data to be manually located by the sysadmin every once in a while, therefore they don't need to be labelled.

              Comment Actions
            2. (9 more comments in this thread)

            3. Our primary datastore is a mysql database, we have 3 primary servers and 2 secondary servers. We make a full backup of each of the machines weekly on different days of the week. We store binlogs on the primary server for 90 days, which I felt at the time was more than enough for any emergency. Given the comments here I am happy to change the retention of the bin logs to 120 days on the primary database servers. Note the failed full backups at the start of the year when we were having issues. However you can see a full backup did succeed each week.

              Given then full backups made weekly and the binary logs we could restore the database to any point in the time we have binary logs for.


              root@ao3-db15:~# ls -last /var/lib/mysql_bin_log/| head -5 ; echo ... ; ls -last /var/lib/mysql_bin_log/| tail -5
              total 1781496116
              126792 -rw-rw---- 1 mysql mysql 129833951 Jun 16 14:15 ao3-db15.002303
              72 drwxrwx--- 2 mysql mysql 69632 Jun 16 14:04 .
              1048632 -rw-rw---- 1 mysql mysql 1073792016 Jun 16 14:04 ao3-db15.002302
              72 -rw-rw---- 1 mysql mysql 66261 Jun 16 14:04 ao3-db15.index
              ...
              1048588 -rw-rw---- 1 mysql mysql 1073746632 Mar 18 20:01 ao3-db15.000609
              1048588 -rw-rw---- 1 mysql mysql 1073746948 Mar 18 18:58 ao3-db15.000608
              1048584 -rw-rw---- 1 mysql mysql 1073741885 Mar 18 17:57 ao3-db15.000607
              1048584 -rw-rw---- 1 mysql mysql 1073744393 Mar 18 16:39 ao3-db15.000606
              1048776 -rw-rw---- 1 mysql mysql 1073942182 Mar 18 15:11 ao3-db15.000605

              Full backups are kept for a year.


              james_@va-nfs02:/export/mariadb-backups$ ls -l ao3-db14/*/*
              -rw-r--r-- 1 mysqlsec otwmysql 456694521583 Jan 3 06:19 ao3-db14/01/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 163492698847 Jan 9 18:21 ao3-db14/02/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 36 Jan 16 09:30 ao3-db14/03/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 36 Jan 23 09:30 ao3-db14/04/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 468320400211 Jan 31 06:55 ao3-db14/05/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 6636748800 Feb 6 21:12 ao3-db14/06/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 564111910609 Feb 20 14:59 ao3-db14/08/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 566714288410 Feb 27 15:27 ao3-db14/09/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 569025655471 Mar 6 16:52 ao3-db14/10/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 571672167509 Mar 13 15:31 ao3-db14/11/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 574276182358 Mar 20 15:33 ao3-db14/12/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 576392200459 Mar 27 15:35 ao3-db14/13/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 578894256784 Apr 3 15:38 ao3-db14/14/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 553940497880 Apr 24 18:40 ao3-db14/17/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 559877638499 May 1 18:38 ao3-db14/18/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 566257329182 May 8 18:42 ao3-db14/19/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 571954267295 May 15 18:48 ao3-db14/20/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 577891972415 May 22 18:54 ao3-db14/21/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 583880046885 May 29 18:59 ao3-db14/22/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 588010512850 Jun 5 19:03 ao3-db14/23/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 592798965253 Jun 12 19:07 ao3-db14/24/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 453507570967 Dec 29 09:36 ao3-db14/52/full.xtream.gz

              james_@va-nfs02:/export/mariadb-backups$ ls -l ao3-db13/0{3,4,6}/*
              -rw-r--r-- 1 mysqlsec otwmysql 458481881196 Jan 17 11:59 ao3-db13/03/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 461994523791 Jan 24 12:34 ao3-db13/04/full.xtream.gz
              -rw-r--r-- 1 mysqlsec otwmysql 522464735677 Feb 6 20:39 ao3-db13/06/full.xtream.gz

              james_@va-nfs02:/export/mariadb-backups$ ls -l ao3-db15/02/*
              -rw-r--r-- 1 mysqlsec otwmysql 457405815353 Jan 12 09:33 ao3-db15/02/full.xtream.gz

              james_@va-nfs02:/export/mysql-backups$ ls -last ao3-db12/*/*
              478220832 -rw-r--r-- 1 mysqlsec otwmysql 489698077848 Jan 4 23:44 ao3-db12/01/full.xtream.gz
              475547560 -rw-r--r-- 1 mysqlsec otwmysql 486960646969 Dec 28 23:44 ao3-db12/52/full.xtream.gz
              472887620 -rw-r--r-- 1 mysqlsec otwmysql 484236868533 Dec 21 23:42 ao3-db12/51/full.xtream.gz
              470573096 -rw-r--r-- 1 mysqlsec otwmysql 481866793148 Dec 15 2022 ao3-db12/50/full.xtream.gz
              468250364 -rw-r--r-- 1 mysqlsec otwmysql 479488318844 Dec 8 2022 ao3-db12/49/full.xtream.gz
              465785060 -rw-r--r-- 1 mysqlsec otwmysql 476963846852 Nov 30 2022 ao3-db12/48/full.xtream.gz
              463497968 -rw-r--r-- 1 mysqlsec otwmysql 474621862523 Nov 23 2022 ao3-db12/47/full.xtream.gz
              461436380 -rw-r--r-- 1 mysqlsec otwmysql 472510799856 Nov 16 2022 ao3-db12/46/full.xtream.gz
              459621464 -rw-r--r-- 1 mysqlsec otwmysql 470652324061 Nov 9 2022 ao3-db12/45/full.xtream.gz
              457557216 -rw-r--r-- 1 mysqlsec otwmysql 468538534252 Nov 2 2022 ao3-db12/44/full.xtream.gz
              455515016 -rw-r--r-- 1 mysqlsec otwmysql 466447321750 Oct 26 2022 ao3-db12/43/full.xtream.gz
              453495824 -rw-r--r-- 1 mysqlsec otwmysql 464379668161 Oct 19 2022 ao3-db12/42/full.xtream.gz
              451362780 -rw-r--r-- 1 mysqlsec otwmysql 462195433451 Oct 12 2022 ao3-db12/41/full.xtream.gz
              449289724 -rw-r--r-- 1 mysqlsec otwmysql 460072622255 Oct 5 2022 ao3-db12/40/full.xtream.gz
              447218724 -rw-r--r-- 1 mysqlsec otwmysql 457951920038 Sep 28 2022 ao3-db12/39/full.xtream.gz
              445111404 -rw-r--r-- 1 mysqlsec otwmysql 455794020870 Sep 21 2022 ao3-db12/38/full.xtream.gz
              443087144 -rw-r--r-- 1 mysqlsec otwmysql 453721179978 Sep 14 2022 ao3-db12/37/full.xtream.gz
              272689648 -rw-r--r-- 1 mysqlsec otwmysql 279234165734 Aug 31 2022 ao3-db12/35/full.xtream.gz
              495174360 -rw-r--r-- 1 mysqlsec otwmysql 507058486085 Aug 24 2022 ao3-db12/34/full.xtream.gz
              493882216 -rw-r--r-- 1 mysqlsec otwmysql 505735329634 Aug 20 2022 ao3-db12/33/full.xtream.gz
              464664020 -rw-r--r-- 1 mysqlsec otwmysql 475815902745 Aug 11 2022 ao3-db12/32/full.xtream.gz
              460855380 -rw-r--r-- 1 mysqlsec otwmysql 471915852558 Jun 16 2022 ao3-db12/24/full.xtream.gz
              458849836 -rw-r--r-- 1 mysqlsec otwmysql 469862177051 Jun 9 2022 ao3-db12/23/full.xtream.gz
              456799944 -rw-r--r-- 1 mysqlsec otwmysql 467763086866 Jun 2 2022 ao3-db12/22/full.xtream.gz
              454025996 -rw-r--r-- 1 mysqlsec otwmysql 464922563369 May 26 2022 ao3-db12/21/full.xtream.gz
              452832312 -rw-r--r-- 1 mysqlsec otwmysql 463700231925 May 19 2022 ao3-db12/20/full.xtream.gz
              450019028 -rw-r--r-- 1 mysqlsec otwmysql 460819428708 May 12 2022 ao3-db12/19/full.xtream.gz
              448011664 -rw-r--r-- 1 mysqlsec otwmysql 458763890506 May 5 2022 ao3-db12/18/full.xtream.gz
              446138396 -rw-r--r-- 1 mysqlsec otwmysql 456845663112 Apr 28 2022 ao3-db12/17/full.xtream.gz
              444256328 -rw-r--r-- 1 mysqlsec otwmysql 454918424149 Apr 21 2022 ao3-db12/16/full.xtream.gz
              442217092 -rw-r--r-- 1 mysqlsec otwmysql 452830250200 Apr 14 2022 ao3-db12/15/full.xtream.gz
              440170952 -rw-r--r-- 1 mysqlsec otwmysql 450735003746 Apr 7 2022 ao3-db12/14/full.xtream.gz
              438180520 -rw-r--r-- 1 mysqlsec otwmysql 448696800082 Mar 31 2022 ao3-db12/13/full.xtream.gz
              436230512 -rw-r--r-- 1 mysqlsec otwmysql 446699991296 Mar 24 2022 ao3-db12/12/full.xtream.gz
              434470716 -rw-r--r-- 1 mysqlsec otwmysql 444897964006 Mar 17 2022 ao3-db12/11/full.xtream.gz
              432298860 -rw-r--r-- 1 mysqlsec otwmysql 442673979804 Mar 10 2022 ao3-db12/10/full.xtream.gz
              430553256 -rw-r--r-- 1 mysqlsec otwmysql 440886481577 Mar 3 2022 ao3-db12/09/full.xtream.gz
              428741732 -rw-r--r-- 1 mysqlsec otwmysql 439031483312 Feb 24 2022 ao3-db12/08/full.xtream.gz
              426696880 -rw-r--r-- 1 mysqlsec otwmysql 436937555536 Feb 17 2022 ao3-db12/07/full.xtream.gz
              424473628 -rw-r--r-- 1 mysqlsec otwmysql 434660944094 Feb 10 2022 ao3-db12/06/full.xtream.gz
              422397752 -rw-r--r-- 1 mysqlsec otwmysql 432535246023 Feb 3 2022 ao3-db12/05/full.xtream.gz
              420166576 -rw-r--r-- 1 mysqlsec otwmysql 430250523608 Jan 27 2022 ao3-db12/04/full.xtream.gz
              418137108 -rw-r--r-- 1 mysqlsec otwmysql 428172346709 Jan 20 2022 ao3-db12/03/full.xtream.gz
              381888852 -rw-r--r-- 1 mysqlsec otwmysql 391054135759 Sep 9 2021 ao3-db12/36/full.xtream.gz
              372169620 -rw-r--r-- 1 mysqlsec otwmysql 381101643736 Aug 5 2021 ao3-db12/31/full.xtream.gz
              205063632 -rw-r--r-- 1 mysqlsec otwmysql 209985130034 Jul 28 2021 ao3-db12/30/full.xtream.gz
              334079920 -rw-r--r-- 1 mysqlsec otwmysql 342097796158 Jul 22 2021 ao3-db12/29/full.xtream.gz
              332217684 -rw-r--r-- 1 mysqlsec otwmysql 340190865502 Jul 15 2021 ao3-db12/28/full.xtream.gz
              330263024 -rw-r--r-- 1 mysqlsec otwmysql 338189291779 Jul 8 2021 ao3-db12/27/full.xtream.gz
              328337300 -rw-r--r-- 1 mysqlsec otwmysql 336217352428 Jul 1 2021 ao3-db12/26/full.xtream.gz
              326446440 -rw-r--r-- 1 mysqlsec otwmysql 334281109545 Jun 24 2021 ao3-db12/25/full.xtream.gz

              Comment Actions
              1. Yeah, this is roughly what I'd expect but then my question would be why on earth you'd have to do full restores etc rather than having tools which allow you to selectively preserve content in compliance with regulations. Which I KNOW is a question you can't answer but it's yet another layer of ridiculousness to all this. There is risk here in terms of specific individuals needing to execute essentially DR features for compliance reasons. Not great!

                Comment Actions