nettle on Netflix, part 3

by Brian L. Dear

Part Three of an Ongoing Series

Let's see.... where were we? Ah yes.

3. Ratings
One thing a new customer notices right away on Netflix is the ratings. They're everywhere!

It's funny: Netflix does ratings exactly, and I mean exactly, as I proposed we do them for my.mp3.com, oh so many years ago, when I worked at MP3.com. The idea was the same: five stars per song and per album. We'd show five black outlines of stars next to each song and album, and fill only enough stars with red to indicate what the community's overall average rating was for something. And then you as the customer could mouse over the stars and click on one of the five stars, and in so doing, indicate your own rating for that item. Your own rating would then be added to the community average. I'm not mentioning this to say I invented the idea first --- but great minds do think alike. :-) Someone else thought of it long before the web I'm sure. I just find it eerie to see a design idea implemented, exactly as I envisioned it years ago.

During my tenure at MP3.com, I was an advocate for letting users rate more than just music. I argued that users should be able to rate everything on the site. My motto was: "if they can't review it, then at least let 'em rate it, but there better be a good reason why they can't review it and rate it." Amazon went this direction -- their site is full of ways for users to share their likes and dislikes. I wanted MP3.com users to rate the artists, their actual band names, their songs, their CDs, their CD covers, their liner notes, even their page design (artists owned much of the look and feel of their artist pages on MP3, much like sellers own a lot of the look and feel of their item pages on eBay). Rating and reviewing stuff and sharing how well or little you liked something seemed like a good idea at the time --- when "site stickiness" was all the rage.

Michael Robertson, CEO and head-honcho of MP3.com, was violently against ratings of songs and wasn't a big fan of user-created reviews either. Thus, him being CEO and head-honcho and all, you never saw any of that on the MP3.com site. :-) Michael's argument was that if you let people rate stuff, you get community averages of 3.2 stars for every damn thing, and so while the idea sounded cool in theory, in practice it's meaningless information.

Knowing what people think of the music they listen to is kinda important if you're going to do a recommendation engine like Netflix's CineMatch, although in MP3.com's case, it was rating music (hey, we shoulda called it... MusicMatch! Oh, wait, that was taken already.)

Instead of ratings, Michael argued, we'll be able to feed our recommendation engine with data on how well users like songs by noting what songs they listen to, how many times they listen to them, and if we want to get really fancy, how much of each particular song they listen to (in other words, how long into the song did they listen -- just a few seconds? half? the full song?). I was never a fan of this approach because simply recording the fact that they listened to something (and how long they listened) didn't communicate to us exactly what the user felt --- only by letting them rate their music would we know if they loved what they heard or thought it sucked, or something in between. We had no idea if they were really listening, or talking on the phone, or gone to the bathroom, or what.

Now, a few years later, I have to wonder: if we had let users rate songs, would things have been any different? Would the users have enjoyed the service more? Would they have valued it more? Who knows.

So we come to Netflix, which is ratings-heaven, like Amazon. Everything is ratable, everything is reviewable. Just like a proper content/community/ecommerce website should be, right?

Maybe.

Let's take a closer look:

• [Good]: If you're going to do ratings, do them well, and do them everywhere, and Netflix does a good job. There's a lot of JavaScript behind those star ratings (somebody was very very busy, I imagine!) but they work on every browser I've tried, and the behavior is consistent and predictable.

• [Troublesome]: One concern I have with Netflix's implementation of ratings is that they seem too in-yer-face. Too loud. Too big. Take a look at the picture above. The "Rent" button is dwarfed by the screen-space taken over by the five stars and the "Not Interested" button. Put a dozen, two dozen, three dozen of these DVD covers on a single page, and suddenly things get very busy. I wonder if there's been any user testing on Netflix's part to see if all that "star clutter" gets in the way of the bread-and-butter of the business -- the "Rent" button. Futhermore, I wonder if there's been any testing to see if there's any psychologically negative effect by having the phrase "Not Interested" underneath every damn DVD pictured on the page.

  Does the appearance of so many "Not Interested" buttons on the page have any subliminal effect on the user? Do all those "Not Interested" buttons suggest "These aren't the droids you're looking for" to the user?

  What does the data say, I wonder? Maybe Netflix can share their findings with us nettlers.

• [Proposed Enhancement]: Netflix basically has a rating system with values of 0 through 5, where 0 means "not interested" (or zero stars), and 1 through 5 are the star values. Netflix in my opinion is missing one additional rating value: the "I own it" value.

  I happen to own about 100 DVDs, and tons of VHSs and laserdiscs on top of that. In other words, I own hundreds of movies already. I don't want Netflix's recommendation engine to recommend any of these particular hundreds of movies because I own 'em already! But there they are, the Ghosts, the Star Treks, the Die Hards, the David Lean films, the Big Easys, the Clint Eastwoods, the Godfathers, the Shawshanks, the Crimson Tides; they're all there, all over the place: Netflix recommending stuff I already own. This hurts my browsing experience. This lowers my expectations of Netflix's ability to recommend stuff to me. Relevance is the key to having a recommendation engine work. When a recommendation engine recommends stuff I already have, relevance goes down the drain.

  If I discover that browsing is unrewarding, then I leave or revert to searching. Searching implies I know what I want. Sometimes I don't. Sometimes I want the all-knowing virtual video-store-rental-clerk that is Netflix to recommend stuff to me. But it's doing a bad job doing so, what with all these irrelevant recommendations. So I wind up not renting. This is bad for me the customer. This is way bad for Netflix the business.

  What I'm trying to say is:

  Recommendations for movies I already own waste value screen real estate. Seeing recommendations for movies I already own wastes my time. The value of my Netflix browsing experience goes down as the number of "Ughh, I own that one already" DVD recommendations goes up. There ought to be a way for me to tell Netflix what stuff I already own. There ought to be a way for me to RATE the stuff I already own.

  At this point, Netflix (if you're still reading along, and I hope you are), you might say: look, just click on "Not Interested" for those DVDs you already own. To which I say, NO, that would be VERY BAD.

  Here's why: let's say there was a way for me to tell you I owned Star Trek I, II, III, and IV -- the movies -- but not V or VII. If I couldn't rate I, II, III, and IV, you might not know how I feel about them. Perhaps they were all given to me as a gift. Perhaps I'd give all of them only one star each. Wouldn't this be valuable to know? If your recommendation engine knew this about me, then it might keep quiet about recommending V and VI. Or if I'd rated them all four or five stars, it might be loud about recommending V and VI. If instead all I could do was click "Not Interested" on I, II, III, and IV, even though I loved 'em, say, you'd never recommend V and VI and that'd be a shame. (After all, VI is pretty good. :-)

  In the end, I think relevance would be increased by Netflix knowing more about me -- and this is one instance where I'd probably be willing to tell a website about stuff I own.

• [Another Proposed Enhancement]: Let users set preferences to indicate whether they even want to see all these star ratings and not-interested things. Most of the time I simply don't care to look at the ratings stuff -- but I might care a lot on the particular DVD's detail page. But in search results? Blah. While browsing? Eh. Maybe. It'd be nice if the ratings could be turned on or off by the user.

• [Good]: Just to show I'm not entirely unhappy :-), let me wrap up Ratings by saying that Netflix seems to already be on the way to recognizing the above issues -- there are places around the site where you can tell the system, "Don't show me titles I've already rented." That's good. (Of course, maybe customers want to rent stuff again in six months. Perhaps another preference.....)

4. The "Rent" Button
Now we come to the heart of the matter -- the graphical button that causes dollars to flow into Netflix and DVDs to flow out: the red "Rent" button. Some observations and thoughts:

• [Color]: I wonder about the choice of color. Did red test best with users, or...? Red so dominates the look and feel of the site -- red company logo, red envelopes, red rating-stars, red navigational-bar background banner colors, red, red, red. And fire-engine red at that. My own personal opinion only, but I find it red to be kinda harsh. (I also wonder about that segment of the population that's color-blind.)

  My main concern with the color has to do with the unclicked versus clicked state of the button. I'll deal with that issue separately below.

• [Size and Shape]: The "Rent" button is exactly the same width as the DVD-cover thumbnail image above it. Same rectangular shape (harkening back to my earlier comments in this series, about the rigid, rectangular design style throughout the site). I wonder if alternatives would work better with users? Did you try a pill-shape or some other shape whose silhouette would more contrast and stand out from the rectangular DVD box above it?

• [Unclicked vs Clicked Colors: Very Bad]: Take a look for yourself. On the left we have an unclicked "Rent" button. On the right we have one that I've clicked with the mouse, telling Netflix that I wanna rent it:
  
  Can you say "S-U-B-T-L-E"? Good. Now say "T-O-O   S-U-B-T-L-E". See, you agree with me. :-) This one kinda baffles me. Why the button on the right isn't orange or green or some other color is beyond me. Why the label on the button on the right hasn't also changed from "Rent" to "Added" or "Queued" or "Now in Queue" or "In Queue" or "Chosen" or "Renting" or something is beyond me.

That's enough for now. We'll continue this in Part Four.

Continued in Part Four...

Posted by brian at August 18, 2002 09:03 AM