Vinicius Vacanti
Misadventures in Technology Entrepreneurism

Inexpensive User Testing

thegongshow:

When I talk to startups about the benefits of constant, iterative user testing in product design cycles, many people think that cost is a hurdle.

Booking 10 in-person sessions in a rented listening lab with a one-way mirror and screen recording in NYC can get pricey: $100-$200 per *qualified* candidate, plus ~$1000 fixed-cost of lab rental… or, as an even more deluxe option, an all-included engagement with a user testing consultant can run a startup from $5,000 - $15,000. That kind of bill is not feasible when you’re trying to get ramen profitable. But, cost is no excuse for not doing regular user testing in the early stages of a startup. So, for the bootstrapped startups, here’s five inexpensive ways to start doing user testing right now:

1) UserTesting.com - This is a slightly more upscale solution (relative to the other 4 choices below), but for $29 you can get a user to do a 20-min screencast for you while they talk about their thought process outloud. If this sounds confusing, this demo of a screencast explains the value of UserTesting.com well. UserTesting.com has the advantage of providing candidates that are qualified to your specifications (such as: age, sex, social-economic characteristics) and you can have users do very specific tasks you request.

UserTesting.com was developed by an ex-co-worker of mine at Homestead who did some user testing with me on our site’s splash page. He is a sharp guy and great at iterative design, so I’m glad he’s making a tool he finds useful for user testing available to the public.

2) Camcorders and friends - This is perhaps the most economical solution, but also the most time-intensive solution on this list. Schedule family, friends, or even strangers of craigslist to come in and sit down in front of your computer and record their actions performing tasks with a camcorder and some simple consumer-grade screen recording software. It will take time to schedule, prepare, perform and analyze all these tests, but the value of having testers live, in-person is often underestimated. The best benefit of live, in-person testing is the ability to ask follow-up questions when you see users stumble in order to best comprehend their frame of reference and expectations at the time of the incident. If possible, try to get a friend who does not work at your startup to be an independent facilitator. Users are more honest with their feedback when they don’t think they’re talking to someone who works on the site day-to-day. [Update: wow, forget the camcorder.  Chris posted in the comments about Silverback, and it looks amazing.  Do that instead!]

3) Feedback Army - Get direct answers to questions about your site. This is a far more low-fidelity solution than the first to options, but it’s simple to setup and super cheap. $10 buys you one paragraph of feedback from 10 people. This type of testing won’t help much with usability (ie workflow issues), but FeedbackArmy is helpful in determining how your overall value proposition to users is received and how users self-report their user experience (ie how much they enjoy the site).

4) PickFu - How many times have you gotten into an argument with a developer or product manager over the size/color/iconography of a simple button? End those arguments quickly with evidenced-based experimentation. With PickFu, you can immediately A/B test stuff like images, buttons, wording, etc with 50 peoples opinions for $5. This isn’t for full-blown user testing, it’s more for making quick, one-off, informed decisions. PickFu doesn’t have the benefit of A/B testing directly in the context of your site, but it’s the ability to run A/B tests without the messy overhead of site integration is a reasonable trade-off.

Side Note: Both PickFu and Feedback Army are arbitrages on Amazon Mechanical Turk, so if you’re an MTurk ninja you could do this type of testing even cheaper by scripting MTurk directly. But, for 98% of tasks/purposes you should just pay one of these services to do the work of creating the MTurk HITs for you instead of re-inventing the wheel.

5) Starbucks - Seriously. Go down to your local Starbucks with two experimenters during coffee rush hour. Go to the person at the end of the line and tell them that if they’re willing to sit with you in front of a laptop and do a user testing session for 7 minutes, the other experiment will wait in line and buy you the drink of your choice. The person at the end of the line has nothing better to do than to wait in line, so they’re very likely to say “yes.” And, a $5 latte for 7 minutes of genuine in-person user testing is a bargain.

Another advantage of the three virtual solutions mentioned above (FeedbackArmy, UserTesting.com, and PickFu) is that they save you time too. Live user testing in-person takes time in preparation, executing the tests, and then creating actionable conclusions and analysis. All these virtual solutions significantly cuts down or eliminates these time requirements. Of course, you sacrifice impactfulness by not doing a lot of this work yourself, but oftentimes this trade-off is appropriate.

I’m sure I missed other useful software and web services, so please add your favorite ways to bootstrap user testing in the comments. So, stop reading blogs and go user test your site.

The Most Helpful Advice I Have Read

“If you review your first site version and don’t feel embarrassment, you spent too much time on it.”

• Reid Hoffman, Founder of LinkedIn

As a pseudo-perfectionist / borderline-OCD’er, I have the hardest time not spending way too much time getting everything to be just right.  This applies to everything:  programming, user interface design and even my commute to work.

Now, whenever I get frustrated that I’m not doing something in the “best possible way”, I just say Reid’s quote to myself and move on.  I’m a much happier / productive person.  I hope his quote will also help you.

How to Get Amazon Books Same Day For Free

Warning:  I’m pretty sure this is wrong — potentially legally, but certainly ethically.

Everyone knows that when you buy a book, you can either pick it up at your local bookstore today or you can save around $10 on Amazon and get it delivered for free in a week or two.  But, there’s a sneaky way to get your book that same day while still paying the Amazon price.

Here’s how you do it:

1. Buy the book at Barnes and Noble / Borders / your local bookstore and save the receipt
2. Order the same book on Amazon and get free delivery (which should be about a week or so)
3. When the Amazon book arrives, return the Amazon book to your local bookstore

Note:  Make sure to confirm that the ISBN numbers are the same on the version you buy from Amazon and your local bookstore.

Yes, it’s wrong and it’s bad.  I don’t think people should do this; local bookstores are struggling as it is.  But, if you are in a bind of some reason, a literary emergency, this may be for you.

One last caution:  I can’t guarantee this will work for you, but I do know that it works.  Also, worst case scenario, you can just return the book to Amazon.

Use Google Latitude To Find Your Missing Phone

Google’s new Latitude service is a way to keep tabs on your friends, but next time you lose your phone, Latitude will become your best friend - a phone tracking device.

Aside from working on your mobile phone as part of their maps application, Google Latitude can also be viewed from a web browser off your iGoogle home page.  So, next time you lose your phone, just log on to Google and see where your phone is currently claiming it is.  It’s LoJack but free!

Everyone should install Google Latitude just for this tracking service. If you aren’t comfortable sharing your location with your friends, don’t.  Just sign up for the service, don’t add any friends, and you’ll be the only person that can see where your phone is. 

Yipit Looking for Experienced Sys. Admin Consultant

Yipit is looking for help from an experienced system architect / server admin / web developer. The work will be on a consultant / freelance basis where we pay an hourly rate. We need you to be in / around New York and to come to our offices (Midtown East or, more affectionately, Turtle Bay).

This is perfect for someone who has some extra time on their hands or is looking to make some extra money and wants to help a young, excited start-up team in New York.

What we can do for you?

• Pay you real money
• Pay you on an hourly basis
• Give you flexible work times
• Be fun, smart and motivated people to work with

What do we need help doing?

• Setting up a site to run on Amazon’s Web Services (EC2, S3, SimpleDB)
• Setting up development, testing and production environments
• Setting up the environments to be able to run Python / Django
• Setting up a dependable back-up system for our code and data
• Helping us think through scalability issues in terms of database design and server architecture

We are looking for a smart individual to help us do this in person and to teach / explain what he/she is doing. We are quick, attentive learners and promise not to be annoying.

Who are we?

I keep saying we because I’m speaking on behalf of the Yipit team but you will be only working with me. My name is Vinicius Vacanti and I’m a co-founder of Yipit.com, a new New York local search engine. Our prototype is live and we have been very encouraged by its performance so far. Consequently, we are currently gearing up to release a city-wide version in 2009.

For more information about Yipit, please visit Yipit or the Yipit Blog.

For more information about me, please visit my blog, my linkedin profile or my twitter account.

Compensation?

Please let us know what you would like your hourly rate to be. We would like you to get paid whatever amount makes you think the job is a worthwhile expenditure of your time.

If you are interested or know someone who might be, we would love to talk to you. Please comment below or email me at vin at yipit dot com

Not that you need to be bribed, but if you suggest the person we end up choosing, we would love to buy you lunch at Shake Shack (or other lunch place of your choice) to express our gratitude. If you are the candidate and contact us directly, we would be happy to buy you lunch.

Small Business Community Organizer - New York Tech Meetup

The new organizer of the New York Tech Meetup, Nate Westheimer, had the idea of launching a “Community Committee” which will be “dedicated to connecting information and resources within the NY tech community”.  Immediately, I thought of a niche community that I am personally very involved with, New York’s small and medium-sized businesses.  The SMB community is in dire need for technology help as it struggles to compete with sophisticated nationwide businesses and online-only retailers.  At the same time, the SMB community is in a position to help the New York community as potential clients and providers of meeting spaces.   The SMB community should definitely be represented in the “Community Committee” and I would like to volunteer my time to make that happen.

The following is a vision of how the New York tech and small business communities could be working together.  The relationship, as laid out below, allows for both communities to efficiently benefit from a stronger relationship.

How do we reach out to the SMB community?

The New York small business community is large (over 50K in New York) and hard to reach.  I have setup a new meetup called the New York Small Business Tech Meetup.  The meetup will bring together members of the New York tech community to demonstrate popular and emerging technologies to small businesses.  In order to populate the meetup with small businesses, we can

• Contact chambers of commerce, business improvement districts and trade organizations in New York

• Encourage NY Tech Meetup members to mention this new meetup to their favorite small businesses

• Through Yipit, I am in constant contact with thousands of small businesses in New York, and will encourage them to join

I am excited to see what a united New York tech and small business community could accomplish together.  Please let me know if you any additional thoughts as a comment below or email me at vin at yipit dot com.

Step-up On Behalf of Your Niche Community

If you think you are in a position to represent a “niche” New York community, I would urge you to volunteer.  You can do so on by leaving a message on the NY Tech Meetup message board.  Nate is looking for volunteers to step up on behalf of all communities that interact with technology including politics, design, video, university, women, VC, government and all other niche communities.

Twitter Should Allow Location Info For Each Tweet

At tonight’s New York Tech Meetup, I will be talking about performing data analysis based on Twitter’s revolutionary data set.  As part of my preparation work, I noticed that a number of interesting location-based services could arise from Twitter’s data set if twitter allowed each tweet to be associated with a specific latitude and longitude.

You are probably thinking that Twitter already lets you specify a location.  You are right, but that location is just a default location assigned to each user on registration.  What I am suggesting is allowing twitter users to submit updates that have a specific latitude and longitude associated with each update.  A third-party client on a iPhone can easily do this by querying the iPhone’s GPS system.

Why would this be helpful?

Here’s a quick example:

During the Atlanta gas crisis, users on twitter started using the #atlgas tag to identify gas stations that weren’t empty.  The logical next step would have been to create a map of these tweets.  But, since the locations were being written in the tweet, it was a serious challenge to accurately parse the messages and auto-create a map.  If each of the tweets accepted lat/longs, it would have become a trivial exercise to produce an extremely helpful map.

Obviously Twitter has a lot on its plate but I continue to believe that it needs to do a better job of making its existing data set more useful to non-Twitter users.  Adding more meta-information to each tweet would certainly help those third-party developers build more interesting applications.

Searches For "mcdonalds jobs" Alarmingly Accelerate

As if there weren’t enough signs the economy is hurting, Google Insights shows us that number of searches for “mcdonalds jobs” is alarmingly accelerating.

In fact, November 9th, the last data point recorded represents the highest number of searches google has seen for “mcdonalds jobs” in the last 5 years (which is as far as the data goes).

Which states are doing this search the most (Ohio, Illinois, Wisconsin, Michigan and Florida)?  I guess the auto-industry’s bailout isn’t coming quickly enough.

Facebook Users and Sarah Palin Are No Longer Listed in a Relationship

Facebook members used to think pretty positively about Republican Vice Presidential Nominee, Sarah Palin.  However, over the last few weeks, they have started thinking 50% more negatively.  How do I know?  Facebook started previewing the next generation of it’s obscure Lexicon service.

Essentially, the service combs through wall posts by Facebook members and performs aggregate analysis on the data similar to Google Trends.  One of the more interesting analysis Facebook performs is “sentiment” tracking.  Whenever a certain term is used in a wall post, it identifies whether the term was positive or negative.

The preview version only allows you to review a small set of topics though, fortunately, Sarah Palin is one of them.  On August 29th, wall posts mentioning Sarah Palin were 80% positive or 20% negative.  Now, wall posts are 30% negative, a 50% increase in negativity.

Breadlines Forming in Manhattan

Overheard in NYC

Foreigner (walking by):  “Wow, I didn’t realize how bad the American economy was…”

Photo of Magnolia Bakery line by gsanjose