Work Text:
DEDICATION
This one is dedicated to saathi1013 for nudging me about popularity metrics, and to anonsally, llamapunk, and anyone else who is a fan of LBD/AoJE and stats. :)
TL;DR: SUMMARY
I wanted to better understand whether and how much the AO3 popularity metrics I've discussed previously vary across different types of fanfic on AO3. This is not intended to be a comprehensive study, but further exploratory work before I dive into large-scale analyses (e.g., looking at all fandoms on AO3). A few highlights:
- Different popularity metrics paint very different pictures. Looking at number of fanworks written in a given category on AO3 gives you a very different measure of "popularity" than looking at the average number of readers per fanwork, or the percentage of readers who leave accolades.
- M/M and Multi get more readers on average than other categories. Some fandoms also get more hits than others, but not in any predictable patterns so far.
- The median hits and kudos charts follow extremely similar patterns. This would be consistent with a similar percentage of readers leaving kudos in most of the categories I evaluated, but it doesn't guarantee that.
- I don't have an easy way to get average kudos/hits rate per category, so I did some exploration of whether median kudos / median hits is at all useful in evaluating fan interactivity in different fandoms. Results are not super conclusive so far. Would love input from statisticians.
UPDATE: Please see analysis about word count and popularity for AO3 ratings categories -- I haven't yet redone the above analyses to see how much of a difference word count makes here, but it could explain some or all of these effects.
METHODOLOGY
Number of fanworks: For relationship categories, I did a works search on AO3 with the relevant checkbox checked, then recorded the number of works returned. These categories are not mutually exclusive (the same fanwork can be placed in more than one of these categories).
For fandoms, I chose two large fandoms (>30K works), two medium fandoms (5K-10K works), one small fandom (500-1000 works), and one tiny fandom (6 works). 6 works is obviously not sufficient for most statistical work -- all stats will be very volatile with so few data points -- but I was curious to look at an active fandom that is fairly young and see how it compared to fandoms that have been around longer. (I also admit to being biased by the fact that I have several friends in both of the smallest fandoms I chose :) ). I looked at the works for each of the following fandom tags:
- Sherlock (TV)
- Marvel Cinematic Universe
- Transformers - All Media Types
- A Song of Ice and Fire - George R. R. Martin
- The Lizzie Bennet Diaries
- The Autobiography of Jane Eyre
Number of hits: I used median instead of mean for two reasons: first, it is less influenced by outliers, and second, it is much easier to obtain from AO3.
I sorted each of the above sets of search results by hits. Then I looked at the number of pages of search results and chose the middle page (rounded down). I recorded the number of hits for the first work on the page (this number is slightly larger than the actual median in most cases, but the result is extremely close to the median).
The exception is Autobiography of Jane Eyre, where I calculated the exact median.
Note: I use "number of readers" and "number of hits" as loosely equivalent in this post, even though there are generally more hits than people who actually stick around to read.
Number of kudos: I did the same thing as I did for hits, but sorting by kudos.
Median kudos / median hits: Yeah... here's where things get iffy. :)
My ideal goal is to compare how many people who read the fic leave feedback. Ideally, I would calculate this by looking at the kudos/hits ratio for each individual fanwork, and then take the average of all those numbers.
However, to calculate the kudos/hits for each fanfic is a formidable task -- it would mean scraping the stats for every fanwork on AO3 (which would take weeks or maybe months). Instead, I used the numbers I had, and I divided the median number of kudos by the median number of hits (then multiplied by 1000 for nicer numbers).
This gives us a number, but it's not really clear what this number means. It's the ratio of two averages, but it might be very different from the average of the ratios. I do a little more thinking about this down in the discussion section.
Bookmarks and comments: I did analyze these as well, but the median number was always extremely low (0 or 1 for every relationship category), so these numbers weren't much use and are not included in the graphs.
RESULTS AND DISCUSSION
The number of fanworks produced in a given category clearly don't tell the whole story about how popular that type of fanfiction is. Neither does any single metric, in all likelihood -- but these graphs show interesting differences in how much work is written vs. how much each type of work is read vs. how many readers leave kudos.
The M/M category gets the most readers per work as well as producing the most fanworks. But Multi gets many readers on average yet has relatively few works. The number of readers per fandom varies in ways that I can't predict/see a reason for.
The median hits and kudos charts follow extremely similar patterns (R=0.95 -- very high positive correlation). This would be consistent with about the same percentage of readers leaving kudos in most of the categories I evaluated, but it doesn't guarantee that.
The median kudos/median hits graph illustrates this more clearly -- there is very little variance between the relationship categories especially. The fandoms show a little more variance; the AoJE fandom is the big outlier. (As I said, we can't take that fandom's results very seriously with only 6 data points. It does seem possible that when there are only a few works in a fandom, they're likely to get a lot more love on average from the people who are looking for those fanworks. I'd have to test this hypothesis in other small fandoms, though, and there's a counterexample below.)
Like I said, it's not really clear whether median kudos / median hits tells us anything useful. It's quite possible that the graphs would look very different for median (kudos/hits), for instance -- my original desired metric.
I did do a few tests of how the two metrics compare (which was labor intensive and involved my hand-collecting the metadata from a bunch of fics, as I don't have code to do that yet). I tested it for three different cases:
Test #1: 100 random Sherlock fanworks (100 from the middle of the alphabetized list):
median (k/h): 30.0
median k/median h: 28.7
Test #2: Kid Icarus (all 19 fanworks in the "Kid Icarus" fandom, which I know nothing about and selected because it was small):
median (k/h): 10.4
median k/median h: 16.7
Test #3: AoJE (all 6 AoJE fanworks):
median (k/h): 46.2
median k/median h: 53.7
Test #2 and #3 are particularly small groups, so not the best test cases. But it's worth noting that two metrics don't differ in predictable ways -- sometimes one is lower and sometimes the other one is. And the difference can be relatively large compared to the differences between the categories in Figures 4 and 8. So basically, we can't conclude anything about the relative average rates of kudos/hits from my graphs.
OTOH, the two metrics are at least vaguely in the same ballpark for these test cases. So it also might be the case that median K/median H is mildly indicative of the rates at which fans interact in different categories -- big differences may in fact be meaningful. But we don't have any way to measure the degree of uncertainty.
FUTURE WORK
This was, as I said, an exploratory effort. I welcome thoughts about improvements to make going forward in looking at popularity metrics on AO3... I would like to do more cross-category comparisons, but I don't yet feel satisfied with my metrics.
I also want to think more about how to measure confidence intervals (the degree of uncertainty) more in all my metrics. I haven't put error bars into my graphs because, in most cases, I don't have any sampling data to help indicate what those should be. But it would be nice to have a better sense of which differences are actually reliable differences for a lot of these comparisons.