Rating Movies: If You Put Garbage In, You’ll get Garbage Out

Netflix, MovieLens, and Criticker are predictive models. They predict what movies that you will like based on your rating of the movies you have seen. Just like the predictive model discussed above, if the ratings that you input into these movie models are inconsistent from movie to movie, you increase the chances that the movie website will recommend to you movies that you won’t like. Having a consistent standard for rating movies is a must.

In my prior life, I would on occasion find myself leading a training session on the predictive model that we were using in our business. Since the purpose of the model was to help our Account Executives make more effective business decisions, one of the points of emphasis was to point out instances when the model would present them with misleading information that could result in ineffective business decisions. One of the most basic of these predictive model traps is that it relies on data input that accurately reflects the conditions being tested in the model. If you put garbage into the model, you will get garbage out of the model.

Netflix, MovieLens, and Criticker are predictive models. They predict movies that you might like based on your rating of the movies you have seen. Just like the predictive model discussed above, if the ratings that you input into these movie models are inconsistent from movie to movie, you increase the chances that the movie website will recommend to you movies that you won’t like. Having a consistent standard for rating movies is a must.

The best approach to rating movies is a simple approach. I start with the Netflix guidelines to rating a movie:

  • 5 Stars = I loved this movie.
  • 4 Stars = I really liked this movie.
  • 3 Stars = I liked this movie.
  • 2 Stars = I didn’t like this movie.
  • 1 Star = I hated this movie.

When I’ve used this standard to guide others in rating movies, the feedback has been that it is an easily understood standard. The primary complaint has been that sometimes the rater can’t decide between the higher and lower rating. The movie fits somewhere in between. For example, “I can’t decide whether I “really like” this movie or just “like” it. This happens enough that I’ve concluded that a 10 point scale is best:

  • 10 = I loved this movie.
  • 9 = I can’t decide between “really liked” and “loved”.
  • 8 = I really liked this movie.
  • 7 = I can’t decide between “liked” and “really liked”.
  • 6 = I liked this movie.
  • 5 = I can’t decide between “didn’t like” and “liked”.
  • 4 = I didn’t like this movie.
  • 3 = I can’t decide between “hated” and “didn’t like”.
  • 2= I hated this movie.
  • 1 = My feeling for this movie is beyond hate.

The nice thing about a 10 point scale is that it is easy to convert to other standards. Using the scales that exist for each of the websites, an example of the conversion would look like this:

  • IMDB = 7  (IMDB uses a 10 point scale already)
  • Netflix = 7 /2 = 3.5 = 4 rounded up.  (Netflix uses 5 star scale with no 1/2 stars)
  • Criticker = 7 x 10 = 70 (Criticker uses 100 point scale).
  • MovieLens = 7 /2 = 3.5 (MovieLens has a 5 star scale but allows input of 1/2 star)

Criticker, being on a 100 point scale, gives you the capability to fine tune your ratings even more. I think it is difficult to subjectively differentiate, for example, between an 82 and an 83. In a future post we can explore this issue further.

So from one simple evaluation of a movie you can generate a consistent rating across all of the websites that you might use. This consistency allows for a more “apples to apples” comparison.

So throw out the garbage. Good data in will produce good data out, and a more reliable list of movies that you will “really like”.

 

Rotten Tomatoes: When Fresh is Not So Fresh

All of these movies have been graded Fresh by at least 75% of the critics who have reviewed the movies. They fail the Certified Fresh test because not enough of Rotten Tomatoes’ critics have reviewed these movies. They fall into a hole in the Rotten Tomatoes ratings system.

What do Mary Poppins, The African Queen, Sleepless in Seattle, In the Heat of the Night, An Officer and a Gentleman, The Natural, and Mr. Smith Goes to Washington all have in common? They are a few of the many Academy Award nominated movie classics that do not earn the Rotten Tomatoes Certified Fresh designation. The reason they aren’t Certified Fresh isn’t because movie critics didn’t like these movies. All of these movies have been graded Fresh by at least 75% of the critics who have reviewed the movies. They fail the Certified Fresh test because not enough of Rotten Tomatoes’ critics have reviewed these movies. They fall into a hole in the Rotten Tomatoes ratings system.

Rotten Tomatoes established their website in 1998. Movie critics for the most part review today’s movies. So the movies with the most reviews used in the Rotten Tomatoes ratings are for those released after 1997. Some of the original Rotten Tomatoes critics, Roger Ebert for example, had been writing reviews for a number of years and had also reviewed some of the classic movies. These few critics gave some of the older movies some reviews in Rotten Tomatoes. But over the years, there haven’t been enough critic reviews of the pre-1998 movie releases to meet the 40 review minimum for limited release movies and 80 review minimum for wide release movies. The movies that would otherwise be graded Certified Fresh except for their failure to meet the minimum critic reviews are graded as Fresh.

There are legitimate reasons for having these minimums. Having 75% of the critics grade a movie as Fresh could be as much about randomness as quality. There is no denying, though, that the movies mentioned above and many more like them would be Certified Fresh if they had been released after 1997.

Does it matter that these movies fall through the Rotten Tomatoes cracks? Yes, if Certified Fresh is your threshold for watching a particular movie. There is a whole universe of movies of higher quality that you are missing, many of much higher quality than some of today’s Certified Fresh movies. Because I use Certified Fresh as criteria for a recommended movie, it biases my movie selections to more recent films.

What can we do about it? How do we determine if the pre-1998 movie is of Certified Fresh quality and not some movie that is randomly Fresh? One idea I’ve come up with is to treat pre-1998 movies graded as Fresh, that were also nominated for an Academy Award, as a Recommended Movie. There are 418 movies released since 1998 that have been nominated for an Academy Award that are in my database. 80% of those nominated movies have been graded by Rotten Tomatoes as Certified Fresh. In contrast,  the remaining 718 non-nominated post-1998 movies in my database were Certified Fresh only 37.5% of the time. There is clearly a strong correlation between an Academy Award nomination and a Certified Fresh rating. When I change my definition of a Rotten Tomatoes Recommended Movie to include pre-1998, Academy Award nominated movies that were graded 75%+ Fresh by Rotten Tomatoes, the probability that I will “really like” the Rotten Tomatoes Recommended Movie goes from 82% to 86%.

This seems to me to be a viable alternative to the not so Fresh Rotten Tomatoes conundrum. What do you think?

 

When It Comes to Movie Rating Websites, There is Strength in Numbers.

If you can only use one website to help you select movies that you will “really like”, which should you choose?

If you can only use one website to help you select movies that you will “really like”, which should you choose? That’s a tougher question than you might think. Because I have used all five of the websites recommended here to select movies to watch, my data has been heavily influenced by their synergy. I have no data to suggest how effective using only one site would be. Here’s what I do have:

Probability I Will “Really Like”
Recommendation Standard When Recommended in Combination with Other Sites When Recommended by This Site Only
MovieLens > 3.73 70.2% 2.8%
Netflix > 3.8 69.9% 8.4%
Criticker > 76 66.4% 10.1%
IMDB > 7.4 64.1% 0.3%
Rotten Tomatoes Certified Fresh 62.7% 4.3%

When MovieLens recommends a movie, in synergy with other websites, it produces the highest probability. When Criticker recommends a movie but the other four sites don’t recommend the movie, then Criticker has the highest probability. Netflix is second in both groups. Which one is the best is unclear. What is clear is that the three sites that recommend movies based on your personal taste in movies, MovieLens, Netflix, & Criticker, outperform the two sites that are based on third party feedback, Rotten Tomatoes and IMDB. When Netflix, MovieLens, & Criticker recommend the same movie there is an 89.9% chance I’ll “really like” it. When both IMDB & Rotten Tomatoes recommend the same movie the probability is 75.8% I’ll “really like” it.

What also is clear is that if four websites are recommending that you don’t watch a movie and one is recommending that you do, the probability is that you won’t “really like” the movie no matter how good that one website is overall. The progression of probabilities in the example below gives some perspective of how combining websites works:

Websites Recommending a Movie Probability I Will “Really Like”
None 3.9%
Netflix Only 8.4%
Netflix & MovieLens Only 31.9%
Netflix, MovieLens, & Criticker Only 50.9%
Netflix, MovieLens, Criticker & IMDB Only 71.1%
All Five 96.6%

Stated Simply, your odds increase with each website that recommends a particular movie. If, for example, you were to only use Netflix for your movie recommendations, the probability of “really liking” a movie might be 69.9% but, in reality, it could be any one of the probabilities in the table above with the exception of the 3.9% for no recommendations. You wouldn’t know if other websites had recommended the movie.

So, if I had to choose one website, I’d choose Netflix-DVD if I were one of their 5,000,000 DVD subscribers. If I’m not already a subscriber I’d go with MovieLens. It would be a reluctant recommendation, though, because the strength in numbers provided by using multiple websites is just so compelling.

***

You’ll notice in the Top Ten Movies Available to Watch This Week that there are a number of movies on the list that are available on Starz. I’m taking advantage of the Comcast Watchathon Week which provides for free Starz, HBO, & Cinemax. Some of my highly rated movies which would ordinarily be unavailable are available for the short duration of this promotion. Bonus movies. Wahoo!!

 

 

Hollywood Has an even Deeper Diversity Problem than it Thinks

According to a study released within the last few weeks by the website Polygraph Cool, actresses have less dialogue than male actors by a significant amount

Children should be seen and not heard. This is a proverb whose origins date back to medieval times. It is a proverb that is rarely used today because, well, it’s so medieval. When it comes to roles for actresses in major motion pictures, however, we aren’t far removed from those medieval times. Actresses are seen in the movies but are not heard as much as their male counterparts. According to a study released within the last few weeks by the website Polygraph Cool, actresses have less dialogue than male actors by a significant amount in 2,000 of the top grossing box office films of all time. The study measures words of dialogue for each character in the screenplays of these movies. Some of the key findings in the study are:

  • Female characters have the most dialogue in only 22% of the films studied.
  • Female characters have two of the top three roles in only 18% of the movies.
  • In all age groups, actresses have less dialogue than male actors in the same  age group.
  • This dialogue discrepancy gets more pronounced as actresses age. Actresses 22-31 have 3 words of dialogue for every 4 words for Actors in the same age group. In comparison Actresses 42-65 have 1 word of dialogue for every 5 words of male dialogue.
  • Even in Romantic Comedies the dialogue is 58% male.

Are there movies out there with greater gender parity? If so, how do you find them? The answer is yes. They do exist but not in great numbers. At the bottom of the Polygraph study linked above, the authors provide a tool that you can use to access the movies used in the study. As I’ve mentioned in a prior article, there is a male bias to both IMDB and Rotten Tomatoes. You might check out ChickFlix.net, which provides movie reviews from a female perspective as a substitute for Rotten Tomatoes.

There is also the Bechdel Test, which is cited in the Polygraph study. This tests movies based on a simple criteria. There must be two main female characters who have a conversation with each other about something other than a man. Based on studies, only about 50% of movies pass the test.

You can also use the personalized movie recommenders that I’ve recommended on my posts. By rating movies on Netflix-DVD, MovieLens, or Criticker, you will generate movie recommendations based on your taste in movies.

The lack of diversity in today’s movies reflect the box office. The first step is being able to identify which movies reflect the diversity that we’d like to see in film. I would like to think that we can push film evolution out of medieval times.

Who is Your Favorite Actor or Actress? Criticker Knows.

Who is your favorite actor or actress?  If you can’t wait for the next Leonardo DiCaprio or Jennifer Lawrence movie, does that make them your favorite actors?  If you have rated on Criticker every movie you’ve ever seen, or in my case every movie seen in the last 15 years, the answer to these questions is just a click away.

Criticker has a number of neat tools on its website. One of my favorites is its Filmmaker List, which can be found by clicking the Explore button that appears along the top banner. You can rank  Actors, as well as Directors or Screenwriters, using a variety of criteria. I like to rank actors based on the average rating I’ve given the movies that they’ve  appeared in. Once you’ve ranked them, you can click on their name and see which of their movies you’ve seen and which ones you haven’t. You can also set a minimum number of movies for your rankings so that you can get the most representative sample that your number of ratings will allow.

For example, I have 1,999 movies rated in Criticker. If I set my minimum at 20 movies for Actors and rank them by average score, my top five favorite Actors are:

Actor Avg. Score # of Movies Seen Best Movie Not Seen
Tom Hanks 85.88 26 Cloud Atlas
Harrison Ford 83.50 24 The Conversation
Morgan Freeman 82.50 22 The Lego Movie
Phillip Seymour Hoffman 81.18 22 Boogie Nights
Samuel L. Jackson 81.00 25 Kingsman: The Secret Service

Based on a 15 movie minimum, my favorite Actresses are:

Actress Avg. Score # of Movies Seen Best Movie Not Seen
Kate Winslet 79.13 15 Hamlet (1996)
Scarlett Johansson 75.52 21 Hail, Caesar!
Judi Dench 74.22 17 Hamlet (1996)
Laura Linney 74.63 16 Mr. Holmes
Natalie Portman 74.35 17 Paris, je t’aime
Meryl Streep 74.28 18 Fantastic Mr. Fox

I included 6 on my Actress list because you just can’t leave Meryl Streep off of any list of the Best Actresses. Also, the Best Movie not seen is based on the highest predicted Criticker Score for movies I haven’t seen, or haven’t seen in the last 15 years.

There are a couple of surprises on my lists. Samuel L. Jackson is a surprise. I can’t say that I watch a particular movie because Samuel L. Jackson is in it. It does, however reflect how many quality movies he’s been in. Scarlett Johansson is another surprise. It’s amazing that I have seen 21 movies of hers and she is only 31 years old.

There are favorite actors of mine who didn’t make the list, such as Paul Newman and Jodie Foster. In Paul Newman’s case, he didn’t meet the 20 minimum and his average movie score wasn’t high enough (79.32 in 19 movies). Jodie Foster would have been the highest on my list with an average score of 79.64 but I’ve only seen 11 of her movies, under the 15 movie minimum I set.

When you first go to the Filmmaker List the default for the minimum movies rated is 3. Under that criteria my favorite actor of all time is Billy Dee Williams (95.67 and 3 movies). “I love Brian Piccolo”, and Lando Calrissian (Star Wars V & VI) as well.

 

 

 

According to IMDB, Do We Get More Conservative as We Age or Just Smarter

We are in the midst of a hotly contested election season in which the pundits attempt to place groups of people in defined boxes. Two of the boxes are the liberal youth vote and the conservative Senior Citizen vote. Some pundits argue that as Senior Citizens die off the country will become more liberal. Others argue that young people will become conservative as they age, maintaining the status quo. Do we become more conservative as we age? There are studies on both sides of the issue as evidenced in this 2012 article and this 2014 article.

I won’t attempt to resolve this issue here but, instead, use it as a backdrop for another interesting finding in my demographic study of IMDB’s Top 250 Movies. Age does appear to factor in IMDB voting. Take a look at these results by Age Group:

Avg. IMDB Rating
Age Group All Male Female
Under 18             8.7             8.8                       8.6
18 to 29             8.5             8.5                       8.4
30 to 45             8.3             8.4                       8.3
Over 45             8.2             8.2                       8.0

As IMDB voters get older, the average rating for the same group of movies is lower. It doesn’t matter whether the groups are male or female. The pattern is still the same. The fact that the avg. ratings for the female groups is consistently lower than the male groups is probably due to the bias towards male-oriented movies in the Top 250. Is this further evidence that we get more conservative as we get older?

I’ll offer up a counter-argument, maybe we get smarter as we get older. There are scientific studies that support this including those cited in this 2011 article. There is some IMDB support for this argument, as well. One of the demographic groups that IMDB captures data for is the Top 1,000 IMDB voters. These are voters who have rated the most movies on IMDB and presumably have watched the most movies. The avg. IMDB rating from this group for the Top 250 Movies is 7.9. Perhaps, the more movies that you watch, the smarter you get at differentiating one movie from another. If so, then maybe the lower average ratings for the older age groups are more representative of the experience gained from watching a greater number of movies. Whether we get more conservative or smarter as we age, it would be wise for the older moviegoer to recognize that the avg. IMDB rating is heavily influenced by males aged 18 to 29. You’ll need to apply a Senior Discount to the rating. What do you think?

 

Screening Room and the Long Shadow of Netflix

A new home video service called Screening Room is being pitched in Hollywood. Here is the idea. For $150, customers would buy a set-top box, probably similar to a cable TV box, which would enable them to rent, for 48 hours, a movie on the day it is released in the movie theaters for $50 a movie, with a share of that $50 going back to movie theaters.

Over the last few weeks, an interesting news item that might significantly impact the future of home entertainment has been making its way into the mainstream media. A new home video service called Screening Room is being pitched in Hollywood. Here is the idea. For $150, customers would buy a set-top box, probably similar to a cable TV box, which would enable them to rent, for 48 hours, a movie on the day it is released in the movie theaters for $50 a movie, with a share of that $50 going back to movie theaters. But wait, the renter of the movie would also get two tickets to see the same movie at the theaters. This is either a crazy idea or the next big thing on the horizon for an industry already in great flux.

My initial reaction was that this is a crazy idea because the $50 cost is out of reach for the average moviegoer. What gives me pause, though, is that the idea belongs to Sean Parker, the co-founder of Napster and former president of Facebook. For moviegoers, he was the character played by Justin Timberlake in The Social Network. Screening Room is backed by a powerful cadre of a who’s who in Hollywood, including producer Brian Grazer and directors Steven Spielberg, Ron Howard, J.J. Abrams and Peter Jackson. Next week it is being demonstrated to industry professionals at the CinemaCon convention in Las Vegas. Stay tuned for further developments. This story could have legs.

For now, let’s consider, is the idea viable and why are powerful moviemakers getting behind it? Many home entertainment customers have bought a streaming device, Roku, Apple TV etc., for $50 – $100. The $150 for the set-top box probably isn’t unreasonable if there is sufficient demand for $50 movies. That’s the big question. Event movies, like Star Wars: The Force Awakens, that are candidates for viewing parties, would be a reasonable target for the service. It would be an extension of the demand that exists today for pay per view boxing matches. Families, that often go to the movies as a family, are another potential market. Couples with young children might also be a target when you add to the movie theater costs the cost of hiring a babysitter. But, for Singles, and Couples with no children, the economic cost benefit doesn’t seem to make sense.

As for moviemakers, the motivations might be complex. There is the possibility that it could add to the box office. For example, young couples with kids who would rather stay home than pay for a sitter, might see more first run movies than they normally would go to the theater to see. The biggest motivation for the movie industry, though, may be the looming threat posed by Netflix and other streaming services.

When Netflix streamed Beasts of No Nation (Idris Elba was a Screen Actors Guild Award winner for the film.) for all of its subscribers on the same day it was released in movie theaters, it sent a shot across the bow of the movie industry, particularly theater owners. The four major theater chains in the US boycotted the film and it ended up getting only a limited release in 200-250 independent and art film theaters. Another Sundance favorite, The Fundamentals of Caring with Paul Rudd and Selena Gomez, was purchased by Netflix and will be streamed on June 24, 2016 on Netflix. It is dawning on the industry that moviegoers may not need to actually go to movie theaters to see first run movies. They can be delivered to them at home.

Screening Room may be seen by the movie theater industry as a financially viable alternative to Netflix, at least until Screening Room also eliminates their payoff to the middle man, making it more affordable to average movie watchers. After all, that’s how Netflix does it.

What do you think? Is it a viable concept?