Rotten Tomatoes, IMDB and the Wisdom of Crowds

In the Introduction of James Surowiecki’s The Wisdom of Crowds, the author writes that “under the right circumstances, groups are remarkably intelligent, and are often smarter than the smartest people in them”. This prescient book, written in 2004, was describing the crowd-sourcing, data driven world that we live in today. If you want information, you type a couple of words into Google and you find exactly what you were looking for on the first page of links. If you are visiting a new city and you’re looking for a good restaurant, you check Yelp to identify the highest rated restaurants. And, if you want to go to the movies, you check Rotten Tomatoes and IMDB to see which of the movies you are considering is the highest rated.

The “right circumstances” for groups to be intelligent, according to Surowiecki, is that the group has to be big enough, diverse, and individual decisions within the group need to be made independently. Rotten Tomatoes is independent enough, most of the critic reviews are made prior to the release of the movie without knowledge of how other critics are rating the movie. Diversity is an interesting question. They are all movie critics after all and most of them are men. Still, they certainly bring a diverse set of life experiences. So, diversity isn’t optimal but still exists. The biggest question mark is whether the group is big enough. Star Wars: The Force Awakens is the most reviewed movie I’ve come across on Rotten Tomatoes with a little more than 335 critics reviews counted in the rating. My database average is 104 reviews. That is not a big sample size for statistical analysis. While, logically, movies rated Certified Fresh 95% should be better than Certified Fresh 75% movies, my data doesn’t support that.

“Really Like” Don’t “Really Like” Total % “Really Like”
CF > 88% 284 155 439 64.7%
CF < 88% 283 154 437 64.8%

There is virtually no difference between movies rated higher than Certified Fresh 88% and those less than Certified Fresh 88%. On the other hand, when you just look at Certified Fresh vs. Fresh vs. Rotten movies, the group allocates the movies intelligently.

“Really Like” Don’t “Really Like” Total % of Total Database % “Really Like”
 CF 567 309 876 44.6% 64.7%
F 324 399 723 36.9% 44.8%
R 91 272 363 18.5% 25.1%

It turns out that crowds of critics are pretty smart.

IMDB certainly meets the criteria for an intelligent group. It is big enough, Star Wars: The Force Awakens has over 450,000 votes, for example. While not as diverse demographically as one might like, it is much more diverse than a crowd of critics. And, moviegoers who vote on IMDB cast their vote independently (how influenced they are by other ratings is a subject for another day). When I rank the movies in my database by Avg. IMDB Rating and allocate them in groups identical to the Rotten Tomatoes table, you get the following results:

Avg. IMDB Rating “Really Like” Don’t “Really Like” Total % of Total Database % “Really Like”
> 7.4 552 324 876 44.6% 63.0%
6.7 to 7.4 361 362 723 36.9% 49.9%
< 6.7 69 294 363 18.5% 19.0%

Crowds of moviegoers are pretty smart as well.

Let’s go one step further. What would these results look like for movies that Rotten Tomatoes rated Certified Fresh and IMDB rated 7.4 or higher:

“Really Like” Don’t “Really Like” Total % of Total Database % “Really Like”
370 156 526 26.8% 70.3%

How about if Rotten Tomatoes rated the movie Rotten and IMDB had an average rating of 6.7 or less:

“Really Like” Don’t “Really Like” Total % of Total Database % “Really Like”
24 193 217 11.1% 11.1%

This is the basis for my rating system. When you combine movie recommender systems together, you improve your chances of selecting movies that you will “really like” and avoiding movies you won’t “really like”. It turns out that crowds of critics and moviegoers are the wisest crowds of all.


IMDB…and the Oscar Goes To

On Sunday, the 2016 Academy Award for Best Picture will be announced. The pundits expect a close race among Spotlight, The Revenant,  and The Big Short, with Mad Max: Fury Road a possibility for an upset. Six weeks ago the Las Vegas odds makers had set the odds for each movie as follows:

  1. Spotlight                                    4:5
  2. The Revenant                           6:5
  3. The Big Short                            8:1
  4. The Martian                               8:1
  5. Mad Max: fury Road              20:1
  6. Bridge of Spies                        30:1
  7. Room                                          40:1
  8. Brooklyn                                    50:1

To determine a winner, voters from the Academy membership, representing a variety of film disciplines, vote for the movie that represents the highest cinematic achievement of 2015. The discipline with the highest representation in the voting is acting. Actors make up 22% of the Academy voters and presumably have the greatest influence on the ultimate winner.

What if IMDB voters chose the Academy Award winner for Best Picture? While IMDB voters don’t represent a variety of film disciplines, they do represent different demographic perspectives. If each of these demographic slices of the IMDB voters chose the Best Picture winner, the results for each group would be:

  • Age Under 18                              The Revenant
  • Age 18 – 29                                   Room
  • Age 30 – 44                                   Room
  • Age 45+                                          Spotlight
  • Males                                              Room
  • Females                                         Room
  • United States                               Room, Spotlight (tie)
  • Non-United States                     Room

And, after combining the votes for all of these IMDB voter groups, the Oscar, in an upset, goes to Room.

The average IMDB ratings (as of February 22, 2016) for the eight nominees reflect a tight race:

  1. Room                                            8.3
  2. Mad Max: Fury Road                8.2
  3. The Revenant                             8.2
  4. Spotlight                                      8.2
  5. The Martian                                8.1
  6. The Big Short                             7.9
  7. Bridge of Spies                           7.7
  8. Brooklyn                                      7.6

Although the co-star of Room, Brie Larson, is the favorite to win Best Actress, I don’t believe Room will win Best Picture on Sunday. Academy voters and IMDB voters are very different. Just as actors will have the greatest influence over who wins the Oscar for Best Picture tomorrow, there are demographic segments that have heavily influenced our IMDB voting for Best Picture. Here are the three primary groups influencing the IMDB vote with their percentage of the aggregate IMDB vote for all eight movies displayed alongside:

  • Voters Aged 18 – 29             52% of total vote
  • Non-US Voters                     79% of total vote
  • Male voters                            84% of total vote

Although the IMDB voting for Room reflects a pretty strong consensus across almost all groups, the vote is dominated by Young, Male, Non-US IMDB voters.

Sunday night, as you watch the Oscars, the lack of diversity among the Academy nominees will be the topic most commented on by Chris Rock, the emcee, the presenters, and the winners. But if you really want to know why  a particular actor or actress didn’t get a nomination, or why a particular movie didn’t win IMDB Best Picture, check out who voted. It’s all there.




IMDB: The Ultimate Word of Mouth

Before the internet, one of the ways people decided what movies to watch was through “word of mouth”. Family, friends, neighbors, work associates etc. would talk about a movie they had seen recently that they really liked. If enough people mentioned the same movie it became a movie that you wanted to see as well.

Today, IMDB (Internet Movie DataBase) is the ultimate “word of mouth” source of feedback on a movie.  From its 60 million registered users, ratings are generated for over 3.4 million movies, TV shows, and episodes of TV shows. (Since this is a movie selection blog I’ll stick to the website’s movie benefits.) After completing the free IMDB registration, users can vote for a movie they’ve seen on a 1 to 10 scale, with 10 being the highest.  From all of the ratings, IMDB compiles an average rating for each movie, with some controls in place to prevent ballot stuffing. So, instead of getting “word of mouth” feedback from a few family and friends, IMDB provides you with feedback from movie watchers from around the globe. If, for example, your movie choice for the evening is between Saving Private Ryan and Life is Beautiful, IMDB provides you with feedback from over 835,000 people for Saving Private Ryan and over 315,000 for Life is Beautiful. But, here is where it gets a little bit tricky. Both of these World War II related movies have an average rating of 8.6. They are both great movies. Which movie will you enjoy more? It depends on how “average” you are.

Because of the volume and diversity of IMDB viewers, the average rating for a movie may not be a demographic fit for you. While the two movies being considered have the same average rating,  the average rating for United States IMDB voters is 8.8 for Saving Private Ryan and 8.4 for Life is Beautiful. The average rating for female IMDB voters is 8.9 for Life is Beautiful and 8.1 for Saving Private Ryan.  Having this information puts a new perspective on which movie you’d prefer to watch.

One of the first of the many useful features available on IMDB that you should become familiar with is the capability to look at a demographic split of the votes that go into a specific rating for a specific movie. This feature is not available directly from the IMDB phone app. It can only be accessed on the website. But, if you go to the bottom of the page for the movie that you pulled up on the phone app, there is a link to the website page for the movie. When you access the movie on the website it will provide you with the average rating for the movie. Right next to the average rating it will show you the number of votes the rating is based on, or how much “word of mouth” feedback you’re getting on this movie. If you click on the number of votes, a page opens up with all of the demographic data behind the feedback population. It tells you how women rated the movie vs. men. It tells you how different age groups rated the movie. It splits US and non-US voters. In a nutshell, it gives you the opportunity to see how the group most like you rated the movie. It’s also a good tool to use when you are trying to select a movie for a group of people to watch.

Will I “Really Like” this Blog?

I love baseball, movies and analyzing data. Since analyzing baseball data is a well-traveled path and analyzing data from baseball movies is too narrow a path, I am left with the intersection of movies and data analysis. Specifically, I analyze data generated by five Movie Ratings websites: IMDB, Rotten Tomatoes, Netflix-DVD, Movielens, and Criticker. There are other sites I could include and maybe will include in the future. For example, Metacritic is a fairly well known website but, for now, I’ve chosen not to use Metacritic because it is similar to Rotten Tomatoes with a less robust volume of movie ratings. I focused on these five sites because they are a good cross section of the methodologies I’ve come across that are used to rate movies.

Over the past few years I have built and maintained a database of all of the movies I have watched in the last 15 years. As of January 31, 2016, my database contains 1,957 movies. For each movie, I have entered the ratings provided by IMDB and Rotten Tomatoes, as well as the personalized ratings generated by Netflix-DVD, Movielens, and Criticker. If you are unfamiliar with these sites, the links at the top of the page will get you to the home page for each site. IMDB and Rotten Tomatoes don’t require any work on your part to see ratings. Criticker and Movielens base their ratings off of the ratings you provide for the movies you’ve seen. Their value as movie guides requires some effort on your part. Netflix-DVD is also based off of your ratings but has the additional requirement that you be a subscriber to their DVD service.

At about this point you are probably asking, “Why is he doing this?”  Initially, I wanted to test which website was the best at leading me to movies that I’d “really like”. Instead, I ended up with an algorithm, using all five websites, that provides me with the probability that I will “really like” a particular movie.  For example, the two Oscar nominated movies for Best Picture this year that I haven’t seen are Bridge of Spies and The Revenant. Based on my algorithm, there is a 98.1% chance that I will “really like” Bridge of Spies while there is only a 49.5% chance that I will “really like” The Revenant. I will watch both movies but it will be with the recognition that Bridge of Spies is close to a sure thing while The Revenant is a 50/50 proposition. Where these probabilities come from is a topic for another day.

 I’ve been using some form of this algorithm to select the movies to watch over the last two years. A comparison of the last two years with the first two years used in my study would suggest I’ve been pretty successful. I watched 165 movies during the 1999 & 2000 calendar year and “really liked” 72 of them. Over the last two years, 2014 & 2015, I watched 182 movies and “really liked” 163.  I’ve gone from “really liking” 44% of my movies in the first years of my study to 90% over the last two years.                                                      

I have a friend of mine who, on occasion, will ask me what I thought of a particular movie. Mostly, I’ll tell him I “really liked” it. He then dismisses my recommendation by saying “but you like everything.” He’s right! I’ve reached the point where 9 out of 10 movies I watch I “really like”. It’s not, however, because I like everything. It’s because I’m able to identify those movies that I probably will “really like” and avoid watching those that I probably won’t like.

As to the question posed in today’s title, “Will I ‘Really Like’ this Blog?” I’ll say this. If you frequently watch movies that you wish you hadn’t, you will “really like” this blog. You won’t have to build your own personal movie selection algorithm. You will, though, gain a better understanding of various movie websites and how they can help you pick a movie to watch that you will “really like”.