Wednesday, August 12, 2009

Beta Testers: Getting Them. Keeping Them.

I've been running Spiderweb Software for about fifteen years now. I owe my ability to make a living writing my cute, little Indie RPGs to a lot of things. Tenacity. A moderate amount of designing skill. Dumb luck.

But, as much as anything else, I owe my ability to stay in business to a group of skilled, dedicated, hard-working volunteer beta testers.

There is a huge chasm between the rough, quivering lump of code you just wrote and a solid product you can distribute to actual people. To make your game/mod/level/adventure publicly available without melting someone's computer, you will need testers. And, if you are a lone individual or running a little company like mine, you won't be able to afford to pay QA people. Thus, you will need to go out and find testers.

I've been doing this for a long time. Here is some advice for how to find the people you need to test your game (or other sort of application) and how to deal with them when you have them.

1. How To Get Testing Volunteers

I am established and have a fan base, so this part is easy for me. I write that I need testers on my web site, take the beta testing application page live, and winnow through a huge pile of applications. If you're just getting started, you will have a harder time.

Make a web page and put up some info and screenshots, so that people can drop by and see what you are making. Then go to online forums where gamers hang out. Write a short, respectful post saying "Hey, I'm making this and I need testers. Drop by my web site and take a look. Send such and such information here." While forums justifiably hate advertisements, offering an opportunity like this is generally tolerated. (And even sometimes appreciated.)

Ask potential testers to send you information about themselves. What sort of computer do you have? What sort of hardware? What sort of games do you like? What have you tested before?

Hopefully, you will get applicants. However, I'll warn you now. You will get very few decent testers. People who can play a game, analyze it, spot bugs, report them in a coherent manner, and come back for more are very rare. They are precious jewels. You'll have to sort through a lot of, well, people who aren't jewels, to find them.

Also, don't be afraid to use your friends, but don't get your hopes up. Very few people have the mental skills and dedication it takes to be a good tester. The odds of any given friend having those characteristics are low. But give it a shot.

2. How To Screen Your Testers

When you get a pile of testers, you'll need to pick out the ones who are potentially useful. One advantage of asking them to write about themselves is that you will be able to scratch out a bunch of incoherent, confused, and crazy folks right away. If they can't answer a few simple questions, they will not be able to report bugs well. We all know how dangerous the incoherent can be in positions of importance.

Once I've picked out a few applications, I make them provide a signed Non-Disclosure Agreement (I let them fax the form, mail it, or e-mail me a .pdf signature). Google "non-disclosure agreement template" to find a few boilerplate versions.

Now, to be honest, if someone leaks your beta, there isn't much you can do about it. But make people sign an NDA anyway. It's legal protection. It stresses to them how important secrecy is. And making them send an NDA in is great for weeding out flakes. If they aren't together enough to send a signature, they probably won't be able to test well.

OK. Hopefully you have a few testers. Some of them might even be decent. If not, you'll be back to begging on the forums again.

3. How To Instruct Your Testers

You will need to give your testers clear instructions. How should they report bugs? Where should they report bugs to? Any particular format you want bug reports in? The exact details will vary depending on the sort of product you are trying to test. However, you will need to tell the testers something.

This is also a good chance to explain to them important basics. Things like how to take a screenshot or how to compress large files before sending them. And make sure to stress that, in any given report, they need to say the exact version of the beta they found the problem in. (So you don't go chasing down bugs you have already fixed.)

4. How To Interact With Your Testers

First off, always be nice. Always take constructive criticism well. These people are volunteers. They are spending their priceless and irreplaceable time helping you for free. Be polite. Be professional. Never lose your temper. If you find yourself writing a snide e-mail, get up. Take a walk. Count to ten. Then delete it. I have lost my temper with a tester only two or three times, and I always felt terrible about it afterwards.

This is a good chance to work on developing a thick skin. Nothing the testers subject you to will compare to the abuse you'll take online after your project has been released.

Don't hesitate to ask for more information about a report when you need it. Ask for exact steps to reproduce the problem. Screenshots. Saved games. Stress to testers that they need to provide lots of details. You never know what tiny scrap of information will help you to find a bug. Vague reports are useless reports. And, if a tester never answers your follow-up questions, that tester should probably be fired.

Yes, you will sometimes need to fire your volunteers. If they don't use the product. If they don't send helpful reports. If they don't treat you in a polite and professional manner. (The way you are treating them. Right?) Just because they aren't being paid doesn't mean that they should always be kept around. They are taking up your time and they are eating up a precious beta tester slot. Also, being able to beta test is valuable, even if no cash is paid. It is professional experience (more on that later) and a chance to actually take part in the creative process. These are valuable things.

5. How To Reward Your Testers

You must treat your testers with the respect and gratitude they deserve. Thank them. List them in the credits. Give them a free copy of your game.

Some testers will help you in order to get experience, in the hopes of getting a real QA job. If a helpful tester asks you for a reference, give one, and make it good. I have had people ask me for references before, and my success rate for getting people their desired positions is 100%.

And Good Luck.

Don't be ashamed to ask for help. For a certain, rare, marvelous breed of person, the ability to take part in the creative process is a great opportunity. Find those people and let them help you. Be nice. Be professional. Figure out what works for your particular product and go with it. After all, without testers, all software would suck.

20 comments:

  1. Thanks so much for this post, Jeff. I know that this advice will come in handy when I'm at this point with my project. Especially the part about giving the testers a few hoops to jump through to check their dedication. I'm always happy to see what you have to say :)

    ReplyDelete
  2. I was a beta tester for you way back in the day (on Exile 2 and 3). At the same time I was working in the QA Dept. at LucasArts. So I must have been a real masochist to want to come home and do more beta testing after work! But that's how psyched I was about your games. :) Also, I was interested to see how testing RPGs compared to testing other types of games.

    Unfortunately, I don't have the time or patience to look at betas (anything) anymore. But I did want to thank you for the opportunity. And I think I still list it at the bottom of my resume.

    ReplyDelete
  3. Whoa. I didn't know indie game companies require beta testers.

    I am currently working as a tester. I like games. How and where can I do some freelance beta testing for indie companies? :D

    ReplyDelete
  4. Like adampasz, I was honored to be a member of Spiderweb's beta tester stable for a good number of years (though I never worked in the field professionally). And it's on my resume as well.

    "Re-playability" gets thrown around a lot with RPGs, but while the QA work could get tedious, I never got bored working through the various Avernum or Geneforge game builds over and over. They are re-playable games.

    I always felt like my contributions were acknowledged and appreciated at Spiderweb, even when I insisted on testing out paths through the game that no sane person would take (like not summoning creatures ever in Geneforge!)

    Helping to solve a pernicious bug is a great feeling, and I'm still pleased that in one of the Geneforge games, there's a NPC with my last name on it. I always made a point of killing him for some reason . . .

    ReplyDelete
  5. I once beta tested for you (Geneforge 4 for Windows) and it was a lot of fun. My husband and I worked together on it, since he has an uncanny ability to "break" games. I had never tested before, but I always felt like I was doing something important and useful. Even though we really only ran into one minor bug that I think had already been found when we saw it, we still felt like we were contributing in some way. I think that's the real key for me, since it kept us going. It made it more fun. You had constant contact with us, and that contact made us feel needed. I'd test again for you in a heartbeat.

    ReplyDelete
  6. Like a surprising number of your former beta testers, I beta tested Avernum II (ten years ago) while I working as a QA tester for ADP. I know you read my bug reports, both because things I found got fixed and at least once you had to ask me to write a better report (my fault!).

    I'm always pleased to recommend your games to people, in part because I know how much care you put into them. Also, because as a college classmate of HWRNMNBSOL, I appreciate seeing him as a cranky old wizard.

    ReplyDelete
  7. I recently came across your blog and have been reading along. I thought I would leave my first comment. I don't know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.


    Susan

    http://onlinemariogames.net

    ReplyDelete
  8. Great post. Fun for me since I've worked in real QA for generally dull government software for a decade, and am generally well respected by my peers as a testing 'expert', and even reasonably paid (!), yet if I moved into games testing, which I would probably be much better at, I certainly wouldn't be valued as a "precious jewel"; I'd take a huge pay hit and barely be valued at all.

    ReplyDelete
  9. hey Jeff =) i'm interested in how to become a beta tester for the Spidweb game series, i would have loved to be a tester for the Avernum and especially Geneforge series!

    ReplyDelete
  10. Nice work. I've bought (if I'm counting right) 10 Spiderweb games. I'm going to pirating Avernum 6 to see the end of it, and then I'm done with your company because of the Sarah Palin shot.

    ReplyDelete
  11. Dear Jeff,

    I love your games, especially the Geneforge series, they kept me company during all my life (I'm 20). I'd love to be a tester of one of your next games.
    How can I become part of it?

    Antonio

    ReplyDelete
  12. Tiffany is about the world's most renowned designers of first-rate offerings.Dissimilar to other vogue companies,tiffany necklaces deal firmly in products for instance jewelry,designer watches,glass wares,lamps,bags,plus more.
    tiffany jewellery
    tiffany silvers
    tiffany uk
    tiffany jewellery uk
    tiffany jewellery sale
    tiffany rings
    tiffany co rings
    tiffany engagement rings
    tiffany bracelets
    tiffany co bracelets
    tiffany necklaces
    tiffany charms
    replica tiffany jewellery
    replica tiffany
    tiffany co
    tiffany and co

    ReplyDelete
  13. The past is gone and static. Nothing we can do will change it.scarlet blade gold, the future is before us and dynamic. Everything we do will affect it rs gold, You laugh at mescarlet blade gold for being different , but I laugh at you for being the same.

    ReplyDelete
  14. You laugh at me for being different, but I laugh at you for being the same LOTRO Gold, the consequences of today are determined by the actions of the past Lord Of The Rings Gold, to change your future,LOTRO Gold alter your decisions today.

    ReplyDelete
  15. No matter the ending is perfect or not, you can't disappear from my world eq2 platinum, In you life, there will at least one time that you forget yourself for someone, asking for no result eq2 plat, no company, no ownership nor love.Just ask for meeting you in my most beautiful years eq2 gold.

    I know someone in the world is waiting for me, although I've no idea of who he is. But I feel happy every day for this 4story Gold, If a woman is not sexy, she needs emotion; if she is not emotional, she needs reason 4story Luna, in this world, only those men who really feel happy can give women happiness 4story Gold.

    ReplyDelete