My AO3 reading history

[superborb]

~Inspired~ by the end of year memes, I was curious what my actual AO3 reading history was like. These data were directly scraped from the history page, so they're only as accurate as that. Namely, AO3 only gives you the date you last opened a fic and dynamically updates things like word count; also, some fics I might have opened without reading all the way through etc etc. So pretty fuzzy here.

Some interesting graphs for my own edification:

The number of works that I've read this year looks like a substantial increase over previous years, BUT remember that only the most recent visit "counts," so if I had a long fic that I most recently read in 2020... (I say to myself to make myself feel better.)


If you look instead at word count, what the actual fuck. Apparently I've opened / read 60 million words this year GOODBYE.


Word count by fandom:
I divided the word count equally among fandoms when multiple fandoms were present.
My top 10 list was:
1.9M words: 魔道祖师 - 墨香铜臭 | Módào Zǔshī - Mòxiāng Tóngxiù
1.3M words: 陈情令 | The Untamed (TV)
405K words: 人渣反派自救系统 - 墨香铜臭 | The Scum Villain's Self-Saving System - Mòxiāng Tóngxiù
140K words: Hockey RPF
136K words: 人渣反派自救系统 - 墨香铜臭 | The Scum Villain's Self-Saving System - Mòxiāng Tóngshiù
126K words: 琅琊榜 | Nirvana in Fire (TV)
124K words: Animorphs - Katherine A. Applegate
120K words: 魔道祖师 | Módào Zǔshī (Cartoon)
103K words: Harry Potter - J. K. Rowling
94K words: Yuri!!! on Ice (Anime)

Actually, I'm very surprised by this, because I didn't think I had read a lot of Hockey RPF this year? But I looked at the fics, and they were all longer fics that I had first read years ago, so I probably didn't remember it bc I didn't reread the whole thing. Ditto with Harry Potter.
However, the top slots all belonging to MXTX and related fandoms makes sense haha.

Word count by character and relationship:
For these, I didn't divide the word count, so numbers are not going to add up. I felt like it was a more fair comparison point though!
27.4M words: Lán Zhàn | Lán Wàngjī/Wèi Yīng | Wèi Wúxiàn
4.4M words: Jiāng Chéng | Jiāng Wǎnyín & Wèi Yīng | Wèi Wúxiàn
3.4M words: Jiāng Yànlí/Jīn Zǐxuān
3.2M words: No relationship
2.8M words: Luò Bīnghé/Shěn Yuán | Shěn Qīngqiū
2.6M words: Lán Huàn | Lán Xīchén/Mèng Yáo | Jīn Guāngyáo
2.0M words: Jiāng Chéng | Jiāng Wǎnyín/Lán Huàn | Lán Xīchén
1.8M words: Lán Huàn | Lán Xīchén & Lán Zhàn | Lán Wàngjī
1.5M words: Liǔ Qīnggē/Shěn Yuán | Shěn Qīngqiū
1.3M words: Original Shěn Qīngqiū/Yuè Qīngyuán

27.0M words: Wèi Yīng | Wèi Wúxiàn
26.8M words: Lán Zhàn | Lán Wàngjī
18.8M words: Jiāng Chéng | Jiāng Wǎnyín
17.2M words: Lán Huàn | Lán Xīchén
13.6M words: Lán Yuàn | Lán Sīzhuī
11.3M words: Niè Huáisāng
11.1M words: Jiāng Yànlí
9.7M words: Wēn Níng | Wēn Qiónglín
8.2M words: Jīn Líng | Jīn Rúlán
7.9M words: Wēn Qíng

No surprises there, based on the top fandoms! I'm surprised there's so much Xicheng though, since I don't think I read very much of it.

Since 2013, I've encountered 539 deleted works. Feels like a lot, it's really a tiny fraction!

Technical notes:
I forked my AO3 code from @/regretsonmain, who also answered my extremely confused cookie questions (thanks again!). I added a bunch of parsing to get more info out of the history, and you can peek at the code here: https://github.com/superborb/ao3.

It turns out the AO3 timeout, when you load too many works at once (aka AO3 jail, thanks @/musikologie for that excellent term) is not as much a problem if you are just looking at your reading history! I thought this would be the main issue with scraping, but it was not. The code does handle it correctly and sleeps for a bit before trying again.

Anyway, I'm not sure what other interesting tidbits might be in this data... I'm very much not a data scientist haha. I got a bit bored of the data and didn't finish up the tag canonization checker which might be where more interesting stuff lies? But my conclusion from this is really that the reading history is not very informative about my reading habits, since it doesn't track what I reread most often, just what I click on.

Comments

[momijizukamori]

Ooh, thanks for the github link! I had vaguely considered writing some scraper code myself and then like... forgot about it again with the twenty other projects I have going. I suspect my 2020 counts will similarly be absurd because 1) MDZS fandom has SO MUCH longfic and 2) 2020 is stressful and I have been escaping to fantasy ancient China a lot.

[superborb]

Yeah, it's pretty easy to do (bc it's just html parsing haha)!

Someone asked if there was a monthly trend, and the Mar-May time when I first encountered MDZS + right when quarantine started was uh... a lot more than the surrounding months

[momijizukamori]

Yeah, it wouldn't have been hard to do, just kind of tedious, so I'm glad someone else has put forth the effort. I ran a fetch on my history last night, though I need to sit down and whip the data into something graphable later.

[superborb]

Yeah, the original author has dropped the project, but it was a good base for updating the code for the reading history! And I haven't used requests or beautifulsoup before, so it was kind of fun to learn.

Do tell if there's any fun insights! I had trouble coming up with interesting questions...

[momijizukamori]

Haha, I work in web application security (and like 3/4ths in Python) so I am familiar with both. Beautifulsoup is really handy for doing transforms on stuff - I used it to put together an epub of the Exiled Rebel translation of MDZS because I was not gonna remove all the WP markup from 100+ pages by hand.

Unfortunately there appears to be something wrong with the way I was trying to add together word counts in the script I wrote at uhhhh.... 10:30 last night, so I think I may write a quick JSON dumper before I rerun the fetch so that I don't have to run it a third time if I mess up again.

[superborb]

Lol yeah, I can see how it'll be a useful tool in the toolkit!

Mmm I fetched and stuck it into a pandas dataframe and pickled that bc I had been worried about the AO3 timeout being an issue

[momijizukamori]

I contemplated pickle but I have had issues with it before trying to read and reopen data (which I am 100% sure are user error and not something wrong with pickle, but wasn't up for sorting it out). Of course, I forgot that datetime objects don't have a native serialization method so I had to do it all a third time when writing the data to file caused it to crash orz

I haven't used pandas before - perhaps today is the day to learn something new.

[superborb]

Oh yeah, I also had an issue with the pickling when I was doing it dflksjk I was using the wrong conda environment and there was a different version of pickle....

I wouldn't have made it a datetime object bc of the serialization issues, but it was in the code before I forked it. The object types are kind of a mess, oops

I also hadn't used pandas before! It was actually super handy for this, bc this is basically a use case that's right up its alley haha.

[momijizukamori]

I really wish they'd just add isoformat/fromisoformat as the default json seralization for datetime objects, because that's pretty 'standard' at this point, particularly in JSON representations. Or epoch time. I guess because there's no 'official' JSON datetime format we're stuck though (and I apparently use datetime enough to have Opinions about this, lol)

pandas definitely seems useful in this case, though I'm still trying to figure out how to 'unpack' the fields that are lists (like fandom, pairings, characters, etc)

[superborb]

Lol probably it should have been returned as a string like everything else and then cast to datetime when actually analyzed...

Oh, I made a copy of the dataframe and then used "explode". It was very handy! For those where I wanted to divide the wordcount, before using explode, I created an adjusted wordcount column so it wouldn't double count.

[momijizukamori]

Oooh, I hadn't come across explode, thank you for the tip!

Somehow I have read 14mil words of WWX/LWJ since... August.....

[superborb]

The number of words of fic I've read... is horrifying...

[rekishi]

Huh interesting. That wouldn't work if I did it but very interesting!

[superborb]

Yeah, it only worked bc I like my AO3 skin so I'm always logged in when reading!

[rekishi]

Ah no, because I don't read online. I download it and then read it offline, because I'm already staring at the screen all day at work (and when writing), reading fic online too... So my results would be faaaaaaaalse. I have 150 unread CQL fics ^^;;;

[superborb]

Ahh, it would get the dates you found the fics I guess haha.

But don't most ebook readers track stats for you automatically?

[rekishi]

Not a kindle that's uh 10 year old XD

The battery is apparently finally going the way of all batteries, so I will have to replace it eventually (soon). But I abuse these things, it lives in my work bag and is carried around a lot and gets taken (in a baggie) to the bathtub and stuff, so I wouldn't buy an expensive one again either.

[superborb]

I guess I don't use my really old kindle anymore (it has trouble charging), but it theoretically was supposed to sync last read pages? But also, I never thoroughly tested that functionality...

[rekishi]

Haha. My kindle is an offline device. It lives in flight mode. It gets managed via calibre. It doesn't even have a touch screen, it's the last generation that didn't have any special features (4th generation?). I don't like any company being able to track what I read, especially not Amazon. I basically own the kindle to read fanfic on the go (or in the bathtub ^^;;; ). :D

[superborb]

Aha, yeah, I was using the kindle app to read some library books and it freaked me out that they show you the most popular highlights. Like, no thank you.

I have a kindle 2nd gen, which I would use so much more if the screen refresh was faster. The eink is so much more comfortable to my eyes... I've been contemplating getting another ereader, but I want to be fully USB-C which only leaves the really high end ereaders...

[rekishi]

Yeah, no, no. I fully agree. dnw.

Hmmmm yes. Or maybe wait another year... I've just recently been seeing that the battery drains a lot faster than before so I guess maybe next year (when I have to be back in the office more?) it's time for another. I've been shopping around, but I don't really like any of them. So idk yet.

[momijizukamori]

It may be possible to pry it open to replace the battery - a lot of the eink models just have some glue at most holding them together, and you can generally get replacement rechargable batteries for under $30USD. iFixit looks to have some Kindle guides, but you'd have to check for your exact model.

My current ereader is a secondhand Kobo Glo I bought on ebay for like $40 - the Kobo devices seem to offer the most 'hackability' of anything other than the like, $500 specialty devices.

[rekishi]

It's probably possible, but at this point... I bought this already used a decade ago, I think the battery would set me back more than I paid for it. But either way, right now I'm not going to the office anyway so it doesn't matter if I need to charge it more often. I might disassemble it and maybe just disconnect and reconnect the battery since sometimes that helps.

We don't have real Kobo's in my country. *sigh* Here, Rakuten bought up the local version of ereaders and now there's some sort of hybrid OS on the devices (which are also not quite the kobo ones). I've been at the store and none of them sit quite right in my hand, aside from functionality. I can, of course, import it from UK or Spain or whatever (or just take a cross-border trip next time there's not a pandemic raging across the globe?). Or I'll go back and maybe play around with the local ones some more. Or stick with kindle since I only read fic on it anyway.

Ah well.

[momijizukamori]

Ah yeah after a certain point the hardware upgrade is worth it - I replaced my B&N Nook when it died because 1) I'm not actually sure why it died (wasn't the battery) and 2) there was a damaged spot on the screen from where it collided with the tip of a pencil in my bag. But if Kindle fits your usecase, those are certainly plentiful secondhand - sometimes I forgot not everyone is all 'ah yes let me rewrite part of this device's operating system myself', heh

[rekishi]

I mean. I am a woman of many talents but "ah yes let me rewrite part of this device's operating system myself" is not one of them. (Don't tell my colleagues, they think I can do everything with technology.)

Mostly I want that thing to be usable with Calibre and be able to do collections and stuff (because my calibre is a very organized). That excludes certain kindles already.

So we'll see. I didn't feel like queuing up at the bookstore today.

[lirazel]

Fandom stats are so fun!

Apparently I've opened / read 60 million words this year GOODBYE.

WHAAAAAT. That's impressive!

[superborb]

I mean, by its nature, it's a ceiling on the number of unique words I read, but uhhhhhhhhhh