My AO3 reading history

[superborb]

~Inspired~ by the end of year memes, I was curious what my actual AO3 reading history was like. These data were directly scraped from the history page, so they're only as accurate as that. Namely, AO3 only gives you the date you last opened a fic and dynamically updates things like word count; also, some fics I might have opened without reading all the way through etc etc. So pretty fuzzy here.

Some interesting graphs for my own edification:

The number of works that I've read this year looks like a substantial increase over previous years, BUT remember that only the most recent visit "counts," so if I had a long fic that I most recently read in 2020... (I say to myself to make myself feel better.)


If you look instead at word count, what the actual fuck. Apparently I've opened / read 60 million words this year GOODBYE.


Word count by fandom:
I divided the word count equally among fandoms when multiple fandoms were present.
My top 10 list was:
1.9M words: 魔道祖师 - 墨香铜臭 | Módào Zǔshī - Mòxiāng Tóngxiù
1.3M words: 陈情令 | The Untamed (TV)
405K words: 人渣反派自救系统 - 墨香铜臭 | The Scum Villain's Self-Saving System - Mòxiāng Tóngxiù
140K words: Hockey RPF
136K words: 人渣反派自救系统 - 墨香铜臭 | The Scum Villain's Self-Saving System - Mòxiāng Tóngshiù
126K words: 琅琊榜 | Nirvana in Fire (TV)
124K words: Animorphs - Katherine A. Applegate
120K words: 魔道祖师 | Módào Zǔshī (Cartoon)
103K words: Harry Potter - J. K. Rowling
94K words: Yuri!!! on Ice (Anime)

Actually, I'm very surprised by this, because I didn't think I had read a lot of Hockey RPF this year? But I looked at the fics, and they were all longer fics that I had first read years ago, so I probably didn't remember it bc I didn't reread the whole thing. Ditto with Harry Potter.
However, the top slots all belonging to MXTX and related fandoms makes sense haha.

Word count by character and relationship:
For these, I didn't divide the word count, so numbers are not going to add up. I felt like it was a more fair comparison point though!
27.4M words: Lán Zhàn | Lán Wàngjī/Wèi Yīng | Wèi Wúxiàn
4.4M words: Jiāng Chéng | Jiāng Wǎnyín & Wèi Yīng | Wèi Wúxiàn
3.4M words: Jiāng Yànlí/Jīn Zǐxuān
3.2M words: No relationship
2.8M words: Luò Bīnghé/Shěn Yuán | Shěn Qīngqiū
2.6M words: Lán Huàn | Lán Xīchén/Mèng Yáo | Jīn Guāngyáo
2.0M words: Jiāng Chéng | Jiāng Wǎnyín/Lán Huàn | Lán Xīchén
1.8M words: Lán Huàn | Lán Xīchén & Lán Zhàn | Lán Wàngjī
1.5M words: Liǔ Qīnggē/Shěn Yuán | Shěn Qīngqiū
1.3M words: Original Shěn Qīngqiū/Yuè Qīngyuán

27.0M words: Wèi Yīng | Wèi Wúxiàn
26.8M words: Lán Zhàn | Lán Wàngjī
18.8M words: Jiāng Chéng | Jiāng Wǎnyín
17.2M words: Lán Huàn | Lán Xīchén
13.6M words: Lán Yuàn | Lán Sīzhuī
11.3M words: Niè Huáisāng
11.1M words: Jiāng Yànlí
9.7M words: Wēn Níng | Wēn Qiónglín
8.2M words: Jīn Líng | Jīn Rúlán
7.9M words: Wēn Qíng

No surprises there, based on the top fandoms! I'm surprised there's so much Xicheng though, since I don't think I read very much of it.

Since 2013, I've encountered 539 deleted works. Feels like a lot, it's really a tiny fraction!

Technical notes:
I forked my AO3 code from @/regretsonmain, who also answered my extremely confused cookie questions (thanks again!). I added a bunch of parsing to get more info out of the history, and you can peek at the code here: https://github.com/superborb/ao3.

It turns out the AO3 timeout, when you load too many works at once (aka AO3 jail, thanks @/musikologie for that excellent term) is not as much a problem if you are just looking at your reading history! I thought this would be the main issue with scraping, but it was not. The code does handle it correctly and sleeps for a bit before trying again.

Anyway, I'm not sure what other interesting tidbits might be in this data... I'm very much not a data scientist haha. I got a bit bored of the data and didn't finish up the tag canonization checker which might be where more interesting stuff lies? But my conclusion from this is really that the reading history is not very informative about my reading habits, since it doesn't track what I reread most often, just what I click on.

Comments

[superborb]

Yeah, the original author has dropped the project, but it was a good base for updating the code for the reading history! And I haven't used requests or beautifulsoup before, so it was kind of fun to learn.

Do tell if there's any fun insights! I had trouble coming up with interesting questions...

[momijizukamori]

Haha, I work in web application security (and like 3/4ths in Python) so I am familiar with both. Beautifulsoup is really handy for doing transforms on stuff - I used it to put together an epub of the Exiled Rebel translation of MDZS because I was not gonna remove all the WP markup from 100+ pages by hand.

Unfortunately there appears to be something wrong with the way I was trying to add together word counts in the script I wrote at uhhhh.... 10:30 last night, so I think I may write a quick JSON dumper before I rerun the fetch so that I don't have to run it a third time if I mess up again.

[superborb]

Lol yeah, I can see how it'll be a useful tool in the toolkit!

Mmm I fetched and stuck it into a pandas dataframe and pickled that bc I had been worried about the AO3 timeout being an issue

[momijizukamori]

I contemplated pickle but I have had issues with it before trying to read and reopen data (which I am 100% sure are user error and not something wrong with pickle, but wasn't up for sorting it out). Of course, I forgot that datetime objects don't have a native serialization method so I had to do it all a third time when writing the data to file caused it to crash orz

I haven't used pandas before - perhaps today is the day to learn something new.

[superborb]

Oh yeah, I also had an issue with the pickling when I was doing it dflksjk I was using the wrong conda environment and there was a different version of pickle....

I wouldn't have made it a datetime object bc of the serialization issues, but it was in the code before I forked it. The object types are kind of a mess, oops

I also hadn't used pandas before! It was actually super handy for this, bc this is basically a use case that's right up its alley haha.

[momijizukamori]

I really wish they'd just add isoformat/fromisoformat as the default json seralization for datetime objects, because that's pretty 'standard' at this point, particularly in JSON representations. Or epoch time. I guess because there's no 'official' JSON datetime format we're stuck though (and I apparently use datetime enough to have Opinions about this, lol)

pandas definitely seems useful in this case, though I'm still trying to figure out how to 'unpack' the fields that are lists (like fandom, pairings, characters, etc)

[superborb]

Lol probably it should have been returned as a string like everything else and then cast to datetime when actually analyzed...

Oh, I made a copy of the dataframe and then used "explode". It was very handy! For those where I wanted to divide the wordcount, before using explode, I created an adjusted wordcount column so it wouldn't double count.

[momijizukamori]

Oooh, I hadn't come across explode, thank you for the tip!

Somehow I have read 14mil words of WWX/LWJ since... August.....

[superborb]

The number of words of fic I've read... is horrifying...