Fun With Numbers: Torne DVR Rankings Predicting the Summer 2013 Disk Over/Unders

Quick recap: I’m taking a look at various no-cost indicators of popularity for anime and their related goods. First, I’m checking how well they correspond to disk sales by checking whether different applications of that statistic beat the null “every v1 will sell less than 4000 disks” accuracy criterion for a given season (66% for Fall, 59% for Summer). Later I’ll check how well these indicators corresponded to boosts in manga/LN source popularity (for works that were originally LN/manga), to contrast their predictive abilities at both high-cost and low-cost levels of interest.

I’ve already looked at the torne DVR ranks for Fall, now I’m doing the same for summer, where the accuracy of null hypothesis test is 59%. The results are listed below, with the lists and detailed breakdowns here. Less than 15 shows made the rankings for August and September, so I didn’t do a top 15 check for those two.

Summer2013-torne-compDespite some changes in what the top 5/10 were, none of these selection criteria did better than the null hypothesis, which is a pretty bleak sign for the rankings as a disk-sales picker given its middling Fall results (though obviously the jury’s still out on manga/LN boosts).

Fun With Numbers: Torne DVR Rankings Predicting Fall 2013 Disk Over/Unders

Note: The original version of this article incorrectly stated that there were 39 shows in the Autumn list, not 38. This has been corrected.

This is the first proper article in what will hopefully be a few individual examinations of the power of various indicators to predict disk sales in a rough manner. This segment deals with Torne rankings, a list ranking shows based on how often they were recorded on PS3 DVR. The original data can be found here, and is collected on this doc. I’m comparing these ranks with the rank of v1 disk sales to see how predictive the former were of the latter, based on an over/under 4000 accuracy model.

See this post for details, but basically, one can get 66% accuracy just by predicting that every v1 will sell under 4000 disks (25 of 38 shows in the season were under). Hopefully, good models will do better than that. In order to test how well each metric performs, I take the top 5, 10, and 15 seasonal shows from the metric and guess they will be over 4000 while everything else will flop, then see how many of those “guesses” end up being right. Clerical note: I’m excluding shows from different seasons (eg Monogatari, HxH) that end up in the Torne rankings, and my top 15 shows are the top 15 to rank from Fall 2013 only.

Since Torne rankings are available monthly, I also compared the accuracy of different months; one might expect later months to be closer to the final sales total, as people have had more time and information (read: episodes of the show) with which to make an informed decision. Assuming the metrics are indicative at all, of course.

Anyway, here’s how accurate the top 5/10/15 guess models were for October/November/December Torne Rankings (numbers outperforming the null hypothesis in green):

torne-correctedThe top 15 model doesn’t seem to be particularly precise in general, but accuracy improved each month, and the December rankings were actually notably accurate. We’ll see how these indicators stack up against others (and how well they compare with source boosts) later.