The question that is on many people’s minds as the iPhone 4 antenna drama plays itself out is “why didn’t any of the reviewers notice this?” After all, we had reviews of the iPhone from the heavy hitters quite a bit ahead of time — seasoned tech journalists who were ostensibly on the lookout for issues like this. Yet I don’t recall reading a single word about sudden signal drops, proximity sensor issues, screen discoloration, or any of the other launch issues. This may be explained by the fact that a sample size of a dozen or two may easily have avoided the launch issues, which clearly do not affect all units.
But it’s also worth considering that a phone, or media player, or game console, or operating system, really isn’t something you can review over a week or two. A restaurant you can review after a meal. A movie you can review after a viewing. A blog post, apparently, you can review just from the headline. But a device that’s going to be a part of your life for a year, two years, or more? Any review posted before, at, or within a few days of launch should properly be considered a first impression.
Flagship devices like mobile phones and media players remain on sale for such a long time, and are used by so many people, that a serious, comprehensive, real-life review really is necessary. Two objections immediately become apparent: “What about the people who want to buy it right away, don’t they deserve some sort of judgment?” and “Reviewers can’t live with every device that comes their way.” Fair enough.
As for the reviewer: his life is a hard one. Bombarded with products, choices and sacrifices must be made to his well-being in order to effectively do his job. I myself rarely live with just one phone for more than a couple weeks, and the layout of my desk is constantly changing as new mice, keyboards, speakers, headphones, drives, and so on pass my way. Some of these can be reviewed in an afternoon, of course: an external hard drive, once tested, examined, and used for a day or two, can be reviewed with confidence. A mouse or speaker set, however, requires getting used to, and nothing less than a week or two weeks of solid use can bear fruit. And something like a phone or media player must be allowed to completely infiltrate my life before I can say a word. But to address the objection more directly: no, even a mighty reviewer like myself doesn’t have time to make a spouse of every gadget that comes my way. But I can do right by the ones that deserve it, and not make judgments where judgment should be deferred.
Systematic reviews, like those done of video cards and DPReview’s camera reviews, are self-sufficient, and provide valuable data. But not every review can or should be systematic. Sometimes, a review isn’t really even possible. Can I review an Android phone when I know the next version of the OS is coming out in a month? Can I review a media player knowing that key features are arriving in the summer? An ongoing judgment is the best, but it’s not something you can easily put in black and white. If someone asked me to review the G1 at launch, I would do so to the best of my ability, but the G1 I have is not the G1 I had at launch, and I would look back on that review with guilt in my heart, and ask people not to read it. My updated opinion is here, in my head. Ask me.
As for early adopters: what can I say? You are very likely to buy the device anyway, and return it if it’s not good. The early adopter is buying based on features, not based on judgments. And early reviews are essentially just confirming those features. They do occasionally catch glaring problems, but really, their function is not to create an opinion of the product so much as to say “yes, these are the features, and yes, they work, except for this one, which doesn’t.” As an early adopter, these problems are part of the lifestyle. You love it.
Unfortunately, the internet is a race. It’s better for your site’s numbers to be first rather than be complete, well-written, or even correct. The difference of a few minutes means a huge amount, and the pressure on a reviewer when given, say, the iPhone 4, is enormous. I don’t blame people for putting up reviews as fast as possible; it makes sense in a business-first sort of way, and on the web you can always update them. And in case you didn’t know, we do have PR people breathing down our necks the entire time. It’ll go up when it’s done, man! Quit emailing me! But these “reviews” should be given the weight they deserve and look forward to the long, opinionated real reviews that really say something.
Where does that leave us? Nowhere, I guess. We aren’t going to stop racing towards FIRST or penning reviews that are nothing more than a checklist of features. But this fall, when someone’s contract is coming up and they’re thinking about getting an iPhone 4, which review do you think would be more useful to them? One written day-of, in which hasty video is made of its features, or one written five months later, taking into account the updates that have occurred, any ongoing hardware issues, average return rates, new features, apps, and so on — doesn’t that sound like a review you’d like to read?
Just so that this rant doesn’t appear to be completely without a point, I should say this: at CrunchGear and MobileCrunch we are cognizant of this problem with reviews, and generally try to state our purpose openly. “First impressions” or “hands-on” when it’s an early or pre-release post, and “Review” only when we feel a device has been given enough time to prove itself (or when the PR wolves are at the door).