Last week, the Internet was abuzz with talk about a single tweet which caused quite a stir. The head of a PR firm tweeted: “#AlwaysBetOnDuke too many went too far with their reviews…we r reviewing who gets games next time and who doesn’t based on today’s venom.” Working hand in hand with PR firms is something we at AppStorm have to do pretty much every day, but it’s not often we’ve seen one of them speak out like this.
Because of that issue, it got me thinking about how we do our reviews here at AppStorm, and it made me wonder if some of our readers think we might have a bias towards the positive side of things, and therefore, we don’t give “real” reviews. So to address that issue, I figured I’d peel back the curtain a bit and talk about how we at AppStorm review an app, and what that means for you, the reader.
The Input
We get lots of submissions at AppStorm, mostly from PR firms and developers who want to get the word out about their product. There’s a few different ways we take in the stream: There’s our Quick Look feature, email, and Twitter is another popular option. When we get these submissions, I’ll typically mark down the name on a spreadsheet I keep, and put it under a “potential review” section. We won’t review all of these apps (there’s just not enough time, frankly), but this way if a writer is looking for an assignment, we have something in the bank.
The Actual Review Process
As the editor, it’s my job to make sure that each writer is on task and writing about an app that either they’ve chosen themselves or that I’ve sent their way. To make it fair and accurate, I want them to use the app for a few days and get the feel of it. Sometimes, an app looks great on day one, but by day five, it’s delegated to the “Unused Apps” folder on their iPhone. This gives us the most fair and unbiased opinion on each app.
If the app works, then the writer puts up a review, with a rating at the bottom that designates how good the app is on a 1-10 scale. Typically, our reviews are between six to ten points, with most of them being in the eights, nines and tens. So why not anything below a six?
Peering Behind the Curtain
I bought an app the other day, with really high hopes for it. I expected it to replace two or three of my other go-to apps, and I figured that it was absolutely perfect to my workflow. Back then (before I was editor), I emailed the boss and called dibs on the review. He slotted me in, and I started playing with the app. As disappointment crept in, I realized that this app wasn’t going to be a 6 or above, so I told the editor as such. The response? “Fine with me, let’s just skip it and move on to the next one.”
We like to keep things positive here on the site, so we’re not going to publish scathing reviews that tear each developer up with each word — we’re just not that site. What we want to do is collect the best apps that you can get, and put them out for our readers to find. It’s those diamonds in the rough or hidden gems that we want to showcase alongside the big names, because we don’t want you to clutter up your iPhone with junk the way we do. The App Store is a pretty big place, so we like to think we’re helping you narrow down the process.
What We Don’t Review
Which, at the end of this all, means that we often review more apps than we actually publish. I know that between all of the AppStorm sites that I write for, I personally turned down at least four last month because they just weren’t good enough. I spent hours of my time trying to determine if the applications were worth our reader’s time, and in the end, they weren’t. So instead of writing something fake just to get paid, I opted instead to move on to something else, that way the quality of the site was kept up.
And now, as editor, that’s something I continue to push. If one of our writers doesn’t like an app, don’t review it. For me, I find the best type of review is written by someone who uses — and hopefully loves — the program. For example, I’m a huge fan of Instapaper for the iPad. I use it every day — usually multiple times a day — and I really think it’s not only helped my productivity, but also the way I consume websites. As such, I wrote a review about it for our sister site iPad.AppStorm.net, and gave it a 10 out of 10. You may not agree with the score, but at least you know that I have a strong opinion about it, which means that at least one person out there thinks it’s worth the money. Why not listen to someone who’s passionate about the program, instead of hunting and pecking in the App Store to find something that just works OK?
The Results
Because of that, we won’t get into an argument on the web with a PR firm the way other sites will. If a PR company pushes a product our direction and we don’t like it, then it’s just not going to get reviewed. That way, the PR company doesn’t get a negative hit on the product, and our readers don’t waste their time reading a review on a less-than-stellar app. It’s win-win in our book.
This isn’t how most businesses run review sites, and we understand that it’s not for everyone. But this system works out pretty well over here, and I think it keeps a positive spin on things. And frankly, if you’re here reading this now, chances are pretty good you agree.