Nettle: One User's Experience

Update on XML Feed

Just a reminder to those who are pinging Nettle's XML feed: the feed location has changed. The old feed was www.nettle.com/feeds/nettle-feed.xml; the new feed is www.nettle.com/index.rdf. The server now has the old XML file symbolically linked to the new file, but consider the old file "deprecated" and eventually it may go away.

Netflix continued -- a mini part 9

Netflix watchers will definitely want to go read a fascinating article called An Analysis of Netflix's DVD Allocation System by someone who only goes by his email address "dvd-rent-test." His argument:

Netflix uses the number of movies you rented in your previous billing period to determine your priority in getting movies. The more movies you rented during your last billing cycle, the less chance you have of receiving a movie versus an individual who has rented fewer movies.

Very interesting. I don't think there's enough data here to confirm that this is indeed the case, and I'm still holding out that there may be other rational explanations for why things happen the way they do for individual Netflix customers. His data does pretty well match my own experience using the service since August 2002. I looked back over all my Netflix automated emails, which are Netflix's "promises" to customers to manage their expectations as to when DVDs have arrived back at the company and when the next DVDs will arrive at my mailbox. The faster the I pushed the rental cycle turnaround (order 2 DVDs, they arrive in the mail, watch 'em both and return to USPS the next day), the slower the turnaround became at Netflix's end. I started noticing regularly that even if I returned two DVDs together, only one would arrive next, even though both were shown as "Now" in the availability column. First the automated emails said the next two DVDs would arrive on the same day. Increasingly, they didn't. Then the emails started indicating the next two rentals were coming on different days. It sure looked like Netflix was staggering the releases, throttling back on the speed at which I was renting stuff.

But it seems that Netflix is saying "a-a-aaa, not so fast" if customers push the rental turnaround cycle too aggressively. One possibility, as dvd-rent-test's data suggests, is that Netflix fine-tunes each customer's rental cycle, not doubt to control costs and balance long-term customer expectations. I'm doubtful that Netflix "punishes" some customers by giving them very long waits and other customers less or no waits at all for the same DVD title. I suspect wait-time discrepancies have more to do with supply and demand at different distribution centers.

By the way, there's a typical debate thrash going on over at Slashdot about dvd-rent-test's report.

UPDATE: Continued in Part Ten...

Coincidence, or What?

Ha! This is too funny! Guess what I just received in an email? A message from "Netflix Shipping" that says "Heavenly Creatures" has shipped and that I should expect it Wednesday. So the Very Long Wait is over.

Now you tell me: within hours of my posting today's long Part Eight review of Netflix, the very article where I talk about how two movies have been on Very Long Wait status for ages, one of these movies being "Heavenly Creatures", I get a notice saying the long wait is over? Hmmmmmm!

Continued in Part Nine...

The Very Long Wait, or, Netflix Part Eight

by Brian L. Dear

I originally signed up for the Netflix service in August 2002. Since then I've posted seven installments in a long series about the service. Time to revisit Netflix and see what's new.

This article is going to cover a lot of ground, including:

• Very Long Waits
  
• Rating Marathon
  
• Questioning the 14,500 number
  
• Various bugs or unexpected behavior seen along the way
  
• Miscellaneous comments about Netflix site changes

Let's start with the very long wait.

1. When the Very Long Wait Gets to be Too Long

"Very Long Wait" is a phrase dreaded by many Netflix users I'm sure. On the Netflix service, you see this message in the Availability column of your rental queue, denoting DVDs that you shouldn't expect to see in your mailbox any time soon.

As I wrote about back in Part 6 of the series, I identified five "phases of the Netflix customer life-cycle". To recap:

• Phase One: Netflix Newbie. Just signed up for the free trial.
  
• Phase Two: The free DVDs rentals for the trial period arrive.
  
• Phase Three: Committment. You pay Netflix for a subscription.
  
• Phase Four: The Borg phase. Total assimilation. Visiting the site less and less, simply populating the queue and going into near-automatic mode.
• Phase Five: The maybe-it's-time-to-quit, there's-more-to-life-than-watching-DVDs phase which when customers enter, may mean they're quitting the service. This is the phase Netflix dreads customers entering, and if the company's smart, it'll do everything it can to continue creating value such that veteran customers stay locked in Phase Four, while more and more customers enter the pipeline of Phases One, Two, and Three.

Phase Four customers, of which I'm one, come to know and dread the "Very Long Wait" message once they start seriously populating their queues. For a long while I had my queue up to about 90 DVD titles, a mixture of 40% mainstream and 60% little-known movies (documentaries, special categories, etc). For example, I added the documentary Hands on a Hard Body back last fall, and to this day it's on "Very Long Wait". In fact the wait is so long I think Netflix ought to just fess up and say "Hopelessly Long Wait" or "Permanent Wait" or "We Only Bought One Copy And We Don't Know Where It Is, And It's Too Expensive to Have One of Our Employees Go Search For It So You're Out Of Luck But If You Really Want to, Go Ahead and Wait".

"Very Long Wait" isn't the only delay message you'll see in a Netflix queue. There's "Short Wait" and "Long Wait" as well. In my experience, it's really quite binary: either a title is available "Now" or it isn't. Anything that doesn't say "Now" means "Forget It" in my book.

On The WELL, the suggestion has come up several times that one way of dealing with these delayed DVDs in your queue is to "flush" your queue of everything except the delayed titles. Now, when you've entered dozens and dozens of titles in your queue, erasing 90% of those titles means a lot of hard work goes down the drain. Nevertheless, I decided to give the ol' flush a try.

One might say that when a Netflix customer reaches this point in the customer life-cycle, it's really what might be called Phase Four-and-a-half. Phase Four-and-a-half being, of course, Borg-level total assimilation but deep in the back of the customer's mind, the customer starts asking questions like, "um, this DVD has been on Very Long Wait status for so long, I'm wondering if I shouldn't just head on over to the Blockbuster and rent it there, because I know they have it because nobody ever rents weird obscure documentaries at Blockbuster." One gets such thoughts because each time one visits one's Netflix queue, and sees more and more Short Waits, Long Waits, and Very Long Waits in the list, one starts to focus on them. At least this particular customer did.

In fact, when the point came that 20% of my queue was populated by DVDs with Wait messages, I decided, ok enough's enough, what would happen if I move all of my Wait movies to the top? Perhaps that'll wake the Netflix engine up and somebody'll see I'm kinda gettin' antsy about having to wait so long.

Alas, moving the Wait titles to the top of my queue did nothing. Netflix blissfully passed right by them, and as rentals came and went in the mail, the next titles fetched off the queue were those that had no "Wait" status. Argh. Time to try The WELL method.

Rather than delete all the "Available Now" titles from the list, I first saved the HTML for the queue page, so I could at least refer back later when I wanted to repopulate my queue. HTML saved, I then proceeded to use the Remove checkbox option on the queue page to remove everything that was available now. Now, that was interesting. Sure enough, within a day, two of the movies in the remaining list went to "Now" status! Ha! It works!

Ok, don't laugh. This is what the queue looked like after I'd condensed it down to Wait titles. I had my own pet theories about why Heavenly Creatures and The Frighteners were on the list (both directed by Peter Jackson, now a superstar thanks to The Lord of the Rings and I was curious to revisit his earlier work, like many other Netflix customers, I'm sure). But Nico Icon, I mean, pretty obscure. Why any wait at all? And Do the Right Thing? I mean, come on, it's a very famous, successful Spike Lee film that every video rental joint has extra copies of. Why delayed by Netflix? No idea.

Anyway, the flush-the-queue-and-leave-only-the-waits technique continued to work. Magically, DVDs in the queue suddenly went into "Now" status. The Frighteners came at one point, and it was so bad, so over-the-top Zemeckis-style, I stopped watching after about 20 minutes and promptly returned the movie in the mail -- a first.

Eventually I widdled the list down to two titles:

And there it stayed. And stayed. And stayed. And stayed. For what seemed like forever. I noticed something interesting.Just above the main queue list is a section Netflix calls "DVDs You Have Out":

Normally, Netflix reports which movies it thinks are in your possession or which movies are on their way to you. But now it was saying "We expect to ship your next available movie by Monday." And when Monday came the messages said "Tuesday". Then "Wednesday". Then "Thursday". Then "Friday." And on and on.

Not a good way to manage a customer's expectations. Netflix generally does a splendid job of managing customer expectations, which is critical if Netflix wants customers to keep in Phase Four. Start screwing around with customer expections, and customers start flocking to Phase Five.

Some additional observations.

Not once during my self-imposed "queue standoff" with Netflix did I receive an email from them noting my queue issues -- the fact that Netflix could not send me any movies. I was curious to see if Netflix, even in automated mode, would make note of this to the point that a) a canned message for "wait-heavy queue" customers would be sent or b) a personal message from a real, live, human from Netflix Customer Support, saying "hey, we just wanted to let you know what the scoop was on those delayed movies in your queue. First, we're having distributor problems with Hands on a Hard Body, and it's prolly not going to be rentable for months; the other titles are just really popular and there are 173 people ahead of you in the backorder log...". Alas, no such messages. Just increasingly wrong messages from the website about when the company expects to ship out the next available movies in the queue. Customers are very sensitive to these messages, and even though they are cushioned with "we expect" rather than "we promise" they'e often, I suspect, considered as promises. And when a company starts failing to deliver on its promises, or over-promising and under-delivering, we wind up knocking on the door of Phase Five again. Not good.

Some suggested recommendations:

• a. Provide some honest explanations to the customer when the queue gets weird. My queue should have sent up a bunch of red flags on the console of some customer support representative at Netflix HQ long ago. I believe the company owes the customer some sort of explanation when things get so out of whack in the queue. I would have been very happy to receive an email explaining the situation with each and every movie in the queue -- why each was delayed so long. OR, instead of an email, add an icon or link within the queue itself saying there's a "Note" available regarding this title. I could click on the note and get the full story, and recommendations on other titles that are available that I might be interested in while I continue to wait.

• b. For heaven's sake, go out and buy more DVDs. If Netflix doesn't already have an employee whose sole mission is to deal with "wait" situations as they arise, then they ought to get someone. If it turns out that customers have been waiting for months for a title, then duh, go out and buy more copies of that title! I mean, Netflix has $100 million in the bank. Surely it can afford to send someone out to scour the cut-out bins at the swap meets and on eBay and elsewhere to get more copies of obscure titles like Hands on a Hard Body. If you're going to be #1 in online DVD rentals, you need to be comprehensive in your inventory -- you need to have everything. (And you should know about everything too -- IMDB-style, even if you don't have the titles, or even if the titles aren't available in DVD format, but that's another story.) Bottom line --- neglecting the needs of your old-time loyal veteran Phase Four customers is a dangerous strategy. You could always open a side business enabling customers to sell DVDs they own but no longer want, to you! I bet out of 1,000,000+ Netflix customers, there are a few who have personal copies of Hands on a Hard Body and never watch it anymore and wouldn't mind selling it to Netflix for the greater good of the customer community. :-)

2. The Rating Marathon
Ever noticed those little blurbs in the headers of Netflix pages where it says "You have X movies in your Rental Queue" and "You've rated N movies. Rate More!"? Well, during my long queue standoff I was down to 2 in the queue and a few hundred movies rated. I've already written a lot and even more about Netflix ratings so I'll try not repeat it here. One assumes that Netflix's reccomendation engine is fueled by data from one's queue and from the movies one's rated. So, back to pet theories again, one might suppose that giving the engine more data might make its recommendations "smarter". After what I went through, I'm not so sure. (More on that later.)

Given how thin my queue had gotten I decided, rightly or wrongly, to fatten up the ratings. Oh boy. That was an adventure. First, I made sure I had good music available, a few hours worth loaded up in an iTunes playlist on my Mac. Next, I just started surfing the Netflix site, rating everything in sight.

Now, admittedly, the vast majority of titles in Netflix's holdings are movies I've never seen. So how can I rate them, you ask? Well, simple. By using the "Not Interested" button located underneath the five rating stars for each title. So I surfed around the site clicking "Not Interested" on hundreds, and then thousands, of titles. Page after page of obscure Tamil movies. Disney movies. Vampire flicks. In fact I built up a list of search keywords that yielded lots of stuff I wasn't interested in:

secrets of the mysteries the ultimate mtv workout best of greatest hits vampire elvis bette midler titanic imax national geographic pokemon

disney blood ninja death fear terror crime combat street fight UFO Volume justice season

complete season murder the crow killer war 2000 world war II wwII vietnam battle horror episode dvd karaoke

Now, here and there within these search results were movies I did eventually want to rent -- so I had to be careful not to rate them lest I never hear about them again on the Netflix site. (I've written before about how I think this is a real flaw in the Netflix rating model: there is no easy way for a customer to communicate that I DO NOT OWN BUT REALLY LIKE THIS MOVIE AND WANNA RENT IT SOME DAY, or, I HAVE SEEN THIS MOVIE IN A THEATRE BUT GUESS WHAT, I DEFINITELY WANNA RENT IT SOME DAY. Netflix in its wisdom seems to think that by rating a movie, you're indicating you have seen it and never ever wish to see it again -- and that includes ever renting it from Netflix. As Homer Simpson would say: "D'oh!")

So I am careful to leave alone those titles I want Netflix to think are potential future rentals.

Kudos to Netflix for adding the checkbox on Genre browsing pages that says, "Don't show movies I've rated/rented." Very handy when you're on a major Ratings Marathon. :-)

Some interesting things turned up in my multi-hour exploration of the Netflix genres --- clicking away at movie after movie after movie, in which for the vast majority I indicated "Not Interested." First, Netflix is very shy about showing you a lot of movies at once. Like in search results. I got the distinct impression there were plenty more movies that had the keyword in the title but Netflix wasn't showing me. And the genre pages and "Rate More!" pages tend to be about 10 movies per page. Now, this is understandable --- the average customer can't be bombarded with an intimidating list of hundreds or thousands of titles on a single page. (Plus there are technical/performance issues both on the client and server, no doubt, in showing such long lists). So the design compromise seems to have been that you get pages instead. Lots and lots and lots of pages. And when you run out of things to rate in a particular genre or subgenre, you go on to the next one. And when that gets old, you do a search on one of the wacky keywords above, and that yields, say, 200 more titles to nuke.

Hours pass. Soon, the genres are drying up, and you find yourself clicking on that "Rate More!" link I mentioned before. The Rate More! link brings up page after page of random movies Netflix knows you have not rated before.

It's pretty cool when you see the header message change from "You've rated 2000 movies. Rate More!" to "You've rated 4000 movies. Rate More!" Eventually that "Rate More!" becomes not just the enthusiastic invitation from Netflix to a happy customer, but rather it becomes a challenge. Eventually you start seeing it as a downright dare. "Go ahead, make our day." So I did. I kept right on rating, rating, rating, while the good tunes kept right on playing.

Oh there was the occasional problem of Netflix suddenly disabling its ratings altogether during my marathon. I would replot a page and suddenly notice no stars or "Not Interested" buttons anywhere --- and at the top, where the "Rate More!" invocation was, a notice appeared announcing that ratings were temporarily unavailable! Eeek! Did I break the site? A half-hour or so later, ratings returned, and my marathon resumed. :-) So it went for few hours on a lazy Saturday.

Eventually it said "You've rated 8000 movies. Rate More!" to which I wanted to respond "Show me more! Prove you really have 'over 14,500 titles'!" I found it increasingly difficult to actually find stuff I'd not rated. I went into each genre and subgenre, and found pages full of titles for which the "Not Interested" button was highlighted yellow (indicating my own rating -- another nice design touch by Netflix... if you give a title a star rating, the stars show as yellow, indicating it's your rating not the general customer base's):

At 8200 movies I really was straining to find anything to rate. At 8220 movies I gave up. I simply couldn't find any more, or was too fatigued to want to bother finding any more. But hey, Netflix proudly states all over the site that there are "13,500+" titles available.

[Before I forget -- let me mention here that Netflix should go do a sitewide search and get its numbers straight. Its press releases say "14,500" titles. The site for the most part still says "13,500" but if you dig deep you still find places such as on the "why do we think you'll like or dislike a movie?" popup window, which says "12,000+". It reminds me of the issue we had at MP3.com during the hypergrowth days: the homepage would say "Over 100,000 songs!" when the press releases said "over 300,000 songs", and by the time we fixed the "100,000" the real number was "500,000". But hey it's a good problem to have in business, so don't complain, just add it to the weekly checklist off stuff to fix on the site.]

I have concluded that Netflix does not in fact have "14,500+" titles on its site because I simply can't find 'em. I looked. :-) Netflix, I could understand there being a thousand or two that I couldn't uncover, but six thousand two hundred and eighty!? Where are they hiding????

Another thing --- getting back to the issue of the recommendation engine and the pet theory that if you feed it more data, if you increase the size of the statistical sample, one supposes the recommendations that spew out of the recommendation engine will be that much smarter. Right? I don't know. It seems to me, when you've rated 8220+ movies, the recommendation engine should have earned its PhD and its ratings to me should reflect that. What they reflect instead, I found, was simply those titles I'd carefully not rated so that Netflix would not "forget" about them. Right or wrong, this particular customer is left with the impression that the recommendation engine isn't doing anything for me that I didn't do better myself (after much hard labor). I will add, however, that it is pretty cool to visit the Netflix site now and pretty much everything it lists is stuff that I want to see. I have finally turned Netflix into the reminder service I wanted --- imagine going into a Blockbuster store and finding every shelf lined with only titles that I had indicated to the store were the titles I wanted to eventually rent. (Of course, I shouldn't have to go through the store title by title, pulling off the shelves everything I know I will never, ever be interested in! :-)

A final note on my Ratings Marathon. I noticed that when I clicked on the "Recommendations" link in the left navigation column, the recomendations I was shown included titles I had explicitly said I was not interested in. For example:

Note that the "Not Interested" is highlighted. That's because I clicked on it before. (I happen to own the laserdisc, so no need to rent it.) It shouldn't be recommended to me, surely. A bug, perhaps?

Now, clearly I am operating under quite a few assumptions here. Some of them are wrong, no doubt. Wrong perhaps because I didn't read all of the explanatory text deep down some help page on the site. Assumptions like the one whereby I intentionally do not rate movies I really wanna see, because I've learned that doing so seems to tell Netflix to never mention them to me again. This is the biggie. I would love to know if my assumption is wrong, partially wrong, partially right, or flat-out right. The fact that I'm a customer operating on assumptions and pet theories suggests to me that the site could stand some additional documentation or design/messaging clarification. (Of course, it also might suggest I'm simply being ignorant. :-) But, seriously, isn't the User Always Right?)

Whew, that's enough. There's more, but it'll have to wait for Part Nine, which I hadn't planned on but I guess will now have to be written!

Comments on this article? Please feel free to share your thoughts below, using my new nifty Movable Type comments tool.

---

UPDATE at 3:23pm Pacific Time ---- got an automated email from Netflix Shipping! This is hilarious. See next blog entry.

Migration Complete

Migrated from plain HTML to a blog engine, Movable Type, this weekend. (No, there's no easy "import" capability -- this was brute force, all done manually. Ugh.) Frequent visitors to nettle will no doubt notice things are a little different around here. Not everything is perfect yet, and some of the CSS specs need tweaking still, but it's up and running. Whew. Glad that's over.

Coming up this week.... nettle revisits Netflix and wrestles the site to the ground. Stay tuned. :-)

Microsoft, Google

Reuters has a story out today about Microsoft. Looks like they're firing the FUD gun up and targeting Google. This quote I found particularly amusing:

We do view Google more and more as a competitor. We believe that we can provide consumers with a better product and a better user experience. That's something that we're actively looking at doing," Bob Visse, director of marketing for Microsoft's MSN Internet services division, said.

Microsoft has wanted to own the web search space for a long time. I remember they had a building full of people as far back as 1997 working on a Google-like service.