Thursday, June 16, 2011

The Ultimate Fix Your Broken Game Checklist.


(This is the third of three articles about providing tech support as a small business. The previous chapters are here and here.)

I get two sorts of requests for tech support for my games. First, there's the familiar problems, the things that are my fault or that I know how to fix. I love these. I tell the victims how to get out from under their burden, they get on with their lives (dazed that an actual human read and processed their request for help), and everyone is happy.

And then there are the gremlins. The weird problems. The sorts of issues that are reported by exactly one person out of many thousands. Problems whose cause and fix are a total mystery. These are, alas, most of the problems reported to me. Most of these problems will be caused by incompatible software, hard drive corruptions, wonky graphics cards, or free-floating evil spirit manifestations. Sadly, as the hardware industry seeks out ways to cut corners and make computers ever cheaper, these sorts of afflictions only grow more common.

(Of course, some people say that every problem with a game, ever, is the result of the programmer's sloppiness and error. As if video cards never break and Windows is a flawless piece of software. These people want you to spend limitless time chasing bugs that don't exist. For a sample of this line of thinking, read the comments for my last post. While I acknowledge that some problems end up my fault, wasting energy trying to fix problems you didn't cause and can't fix is a Bad Thing.)

When I get one of these weird problems, I send out a standard list of troubleshooting steps. I have spent years assembling and perfecting it, and it honestly fixes the vast majority of problems. If you are a small developer, I invite you to steal and adapt it. If you are suffering from misbehaving software, I suggest trying these steps. At the risk of sounding slightly arrogant, if everyone would just drop everything, listen to what I have to say, and follow it without question, the world would be a much better place.

Spiderweb Software Tech Support Checklist

1. Restart your machine and try the game again. Run the game when no other applications are running. A lot of glitches and crashes are one-time things and don't happen again.

When someone runs one of my games, it is far from the only moving part in their system. There is the OS. The drivers. The many background processes. The other programs that are running. Any one of these programs might have a bug that messes up sections of memory. (Not to mention the fact that RAM can be corrupted on its own.) The longer a computer runs without a reboot, the more likely that things might get messed up. Then the system goes down and comes back up, and everything is nice and clean again.

Most problems only happen once, due to weird and unreproducible effects, and never happen again. If you get a crash, don't immediately freak out. Take a deep breath, reboot your machine, and try again.

2. Sometimes, game files can become corrupted. Try uninstalling, redownloading and reinstalling the game. This fixes a surprising number of odd problems.

The key step. Any program can become corrupted while being downloaded, installed, or just existing on the hard drive. This step is a surefire way to fix any such flaws. I am constantly amazed by how often I can fix a catastrophic, recurring problem by simply telling the user to uninstall and reinstall. No other mucking about with settings, saved games, or whatever.

This step also has the bonus of making sure the user has the newest version of the software.

One warning note. This step has three parts. Uninstall. Download clean copy. Reinstall. Be sure the user doesn't skip a step, or the beneficial effects are lost. For example, if you install without uninstalling first, the installer might leave the existing (corrupted) files alone instead of copying over them.

3. (Windows Users) Your video card driver files might be out of date or damaged. Get the latest versions of video card drivers and reinstall them. Even if you are currently running the newest version, the files or settings might be corrupted or damaged. Reinstalling might fix the problem.

I always hate telling users to install new drivers, for two reasons. First, they often think that I'm just doing it to blow them off. After all, this is what ALL developers tell them to do. Second, unlike the previous steps, identifying your video card, finding the manufacturer's web site, locating the newest driver, downloading it, and installing it requires a reasonable amount of technical skill. Odds are, your grandmother won't be able to figure out how to reinstall drivers.

And yet, I tell people to do it for one reason. It works. Weird, glitchy graphics? This is probably the answer. If new drivers don't do it, a flaw in the actual video card is probably the answer.

Lately, many of my users running 64-bit Windows 7 have had tons of problems that were solved completely by getting the new drivers. Several of them swore up and down that they were using the newest drivers. They weren't.

I strongly believe that the ugly situation with video cards and drivers is one of the key reasons for the huge shift in gaming from PCs to consoles. It's really not good.

4. (Windows users.) If random crashes happen during gameplay, try turning the sound off. If this helps, reinstall your sound card drivers too.

We're getting to the bottom of the barrel now, and this step isn't near as necessary as it used to be. As sound engines have improved, most of my weird crashes from sound card/engine issues faded away.

However, it does eliminate one more set of drivers to work with, and not using sound reduces the amount of memory the game needs. Every once in a while, this fixes a problem.

5. (Windows users.) Sometimes, reinstalling DirectX solves unexplained crashes.

And it does. Very, very rarely. It happens, but seldom enough that I'm considering removing this step from the list. Also, all of our newest games use OpenGL.

And That's It

If it's a problem or crash nobody has reported before and the user goes through these steps (or claims to have done so) and the issue isn't fixed, I give a refund. I'm out of answers.

I do hang onto the e-mails, though, in case other people pop up with the same problem later on. Sometimes a working program gets bushwhacked by another, poorly-written program. For example, on the Mac side, there was once a version of Quicken Scheduler that caused games to stop being able to see the keyboard. When I got the second report of keyboard failures, I sprung into action. Having the earlier report around helped a lot in figuring out the problem.

Steps I Don't Suggest

There are two steps for fixing mysterious problems that I almost never suggest. I feel that they are very extreme, involving too much work (and perhaps expense) to be worth it just to play my little game.

1. Reinstall the Operating System

This is the nuclear bomb, the way to purge your system of all manner of corrupted files. It's also a huge pain in the neck. I generally only suggest this if the user mentions he or she is seeing a lot of nasty problems while using a number of different applications.

2. Get a New Video Card

Video cards break. Seriously. It's happened to me. Sometimes they fail and you need to get a new one. However, I only recommend looking into this if the user mentions having similar problems over a wide range of games and swears up and down that fresh, up-to-date drivers are installed.

In Summary

Tech support is necessary, time-consuming, and aggravating. It tests my patience more than anything else I do. And yet, doing it well will, in the long run, make your customers love you and earn you money.

Be as patient and kind as you can possibly manage. Only fight battles when you have a chance of winning. Consider, with humility, that a problem might be caused by a bug. Also remember that, very often, it won't be your fault. Respect how shoddy and cheap and poorly maintained computers frequently are. And, always, try to treat your customers they way you would want to be treated.

You will always be amazed at how messed up things can get. Good luck.

37 comments:

  1. "1. Restart your machine" - Well luckily you just release games for Windows and Mac (I think at least) because there are more then a few Linux users that would rather spend months rewriting their kernel then restart the computer. (Note this is a exaggerated claim that I think is technically wrong because any kernel change requires a restart)

    "2. Sometimes, game files can become corrupted. Try uninstalling, redownloading and reinstalling the game. This fixes a surprising number of odd problems."
    Have you ever thought of including hashes with all your game files and having your installer check itself before starting and the files after copying?
    Also no reason the game itself could not do things like save game file corruption checking.

    "if you install without uninstalling first, the installer might leave the existing (corrupted) files alone instead of copying over them."
    But you created/at least have the ability to change, this installer your self though, so why not just have it auto uninstall if the program already exists?

    ReplyDelete
  2. "read the comments for my last post." Your link it wrong (http://www.blogger.com/posts.g?blogID=4338724676892513065).

    ReplyDelete
  3. @Jonathon:

    1. It's not up to me whether people take my advice or not. All I can do is give it. Still valid advice.

    2. It is something to consider. I do a lot of error checking in my code. But it's a Maginot line thing. There's so many files that can be messed up in so many ways, after all.

    3. Windows installers are msi, and, even if I the feature to force autouninstall exists, I think it's really not a good idea. The expected behavior is offering to repair an existing install, which is a nice option but not one I would trust entirely.

    Mac installers are opening a DMG and dragging the folder out. That's the expected behavior. And, even if it wasn't so, forcing an uninstall is still frowned upon. Steve Jobs wouldn't like it. :)

    - Jeff Vogel

    ReplyDelete
  4. "There's so many files that can be messed up in so many ways, after all."

    Well that is the great thing about hash comparisons, it is simply checking if two files are exactly the same without having to have two copies of the file.

    ReplyDelete
  5. @Jonathon: I can't do a hash comparison on their drivers. Or their operating system.

    Also, I can do it for the applications I distribute, but there are serious limits to what I can do for the versions of my application that are repackaged and redistributed by 3rd parties (Direct2Drive, etc). Remember, we indies have to do support for those too.

    In other words, there are many things a good programmer can to do error check (and I do a lot, though not as much as I can, I'm sure), but your trickery is never going to stop every problem, and you'll need to answer a tech support request. Thus, list above.

    Unless you are saying, "Well, if you're awesome, nothing ever goes wrong." And I'm sure that's not what you're saying.

    - Jeff Vogel

    PS By the way. If the hash comparison finds a problem, the next step is uninstall, redownload, reinstall. You seem to think I'm making a mistake, but I'm not sure what it is.

    ReplyDelete
  6. I'd bet that a good chunk of the ones that randomly happen, but seem to go away after a reboot, come from DLLs that get loaded into your process that aren't your own. On Windows, at least, though I'd be shocked if Macs don't have a similar facility.

    Anything that installs a global hook gets loaded into every process, and suddenly has the ability to trash your code/data. And if you look at a windows box that's been around for awhile there may be dozens of the things installed. I had one box that, as soon as I added a second monitor, random processes would fault, and that one turned out to be a global hook DLL installed by the craplet that came with NVIDIA cards. Updating video drivers would have fixed that one, even though it wasn't directly a driver problem.

    Of course, pointer errors in your own code can do the same thing, but those tend to be at least somewhat reproducible.

    ReplyDelete
  7. It's a shame there isn't a reasonably-performant software implementation of OpenGL available as a fallback, like the DX11 WARP renderer. I presume it'd make problems with video cards or drivers really stand out.

    ReplyDelete
  8. As for Mac - we tried dmg but there were some problems: corruption upon download, mounting problems, etc. We got lots of angry mails.
    Also, it's quite hard to build dmg from Linux shell (we use ant for automated build).

    Now we use just zipped application bundle and all problems are gone :-)


    As for Linux users - there is a problem which package type you create as not every distro supports all. Maintaining few package types is time consuming and Linux guys are 10-15% of sales.

    We decided on deb (Ubuntu seems to be quite popular) and gave instructions how to convert it with alien tool, but there are users with ArchLinux etc.

    I am not sure how we can be more user friendly to Linux world without maintaining serveral package types. Anyone has experience with delivering package to Linux users?

    Les

    ReplyDelete
  9. Just some related questions/speculation...

    Do all games, nowadays, need installers? On Windows, for example, couldn't you just distribute an executable that contains all your game assets inside the binary? Sure, if your game data exceeds a gigabyte you will run into OS limitations, but most shareware games are a lot less than that (Avadon, for example, is 100mb, right?).

    Personally, if I am playing a game, I much rather just unzip a directory, or have a single executable like Minecraft, than to have to run an installer and risk it messing up my machine. Since installers essentially need root access on a Windows machine, I am always very nervous about running installers. In fact, the reason I haven't tried any Spider Web Software titles is because I am super picky about the software I install.

    Also, if a game is a single executable, or a directory, I can put it on a thumb drive and take it with me, so it is way more convenient.

    Yet, nearly all games are distributed with installers... so I assume I am missing something, right?

    ReplyDelete
  10. Minecraft does install, it's just automatic and hidden I think.

    ReplyDelete
  11. Jonathan:

    On Windows, Minecraft downloads and stores some data in the AppData directory. It doesn't require root access to the system in order to create directories in the Programs folder, it doesn't change any system files, it doesn't change any directory, it doesn't place itself in the All Programs tab at the bottom left. You download the .exe, click on the .exe, and you are playing Minecraft.

    ReplyDelete
  12. @spammer Jon: That must be the worst and most awkward customer support job to have.

    ReplyDelete
  13. Just a question: How many people here are programmers? I didn't expect these sort of replies considering Jeff's blog is usually more about games than development/tech support.

    ReplyDelete
  14. Raises hand.

    But it is more then often about game development and other more technical stuff then simple reviews, something that would appeal to any game developer.

    And I think everyone should agree that his blog is just so very sophisticated and not written for the average Halo-er or Gears aficionado.

    ReplyDelete
  15. Rex Rhino said: "Yet, nearly all games are distributed with installers... so I assume I am missing something, right?"

    Unless the game is a roguelike (in which case most people do, like you, expect a zip) Windows users generally expect an installer. They don't want to go to the trouble of extracting folders, creating shortcuts and dragging them to the desktop. Some may not even know or care how to do so.

    So the answer is basically, software publishers *could* easily use zips, but use installers instead because that's what most Windows customers expect.

    ReplyDelete
  16. I'm plenty capable as a software engineer, but I hate executables without installers. It means I have to decide where to put it, keep track of where I put it, figure out how to make it available to different accounts on the same machine, figure out when and how to update it, etc., etc. Why shouldn't the distributor do that once, for everyone?

    ReplyDelete
  17. @David desJardins: because it can go wrong and because it is limiting.
    If you just give me an exe that runs the program then I will not have to re-install your program every time I re install my OS or I can even use it from my thumb drive; And I have used programs that simply could not deal with the fact that they were installed to "Program Files (x86)" instead of simply "Program Files"

    ReplyDelete
  18. Obviously, one reason not to use an installer is that if you can't write an installer that works reliably then it can be worse than nothing. But I can't remember having an installer problem in years (except for those that really are doing complicated things, like downloading dynamic content online, which distributing an EXE wouldn't solve anyway). With modern tools it's simply not hard to build and distribute a standard installer. Still, obviously it's a matter of taste.

    If your program is really just a single executable file nothing stops you from installing it and then copying it onto your thumb drive. But most games these days are more complicated than that.

    ReplyDelete
  19. David desJardins:

    Once of my biggest problems, is that an installer requires admin access. It gets access to system files (and can mess things up), it gets access to the registry (and can mess things up). Not only that, if I don't know I will actually like a game, it is a huge pain to run the installer, and then the uninstaller, if I am going to play for 5 minutes and decide I don't like it.

    Why not distribute both an installer, and a zip version?

    ReplyDelete
  20. Windows installers don't have to require admin access. You can have the option to install files in per-user space.

    It also seems way easier to me to run the uninstaller (a couple of clicks) than to figure out where all of the files for an app are.

    I personally can't see supporting two distributions as a developer, way too much hassle. This whole thread is about customer support, you make that harder the more things you try to support.

    That said, everyone is of course entitled to their own preference. I'm just personally much less likely to download and install an application if it doesn't come with an installer.

    ReplyDelete
  21. "I'm just personally much less likely to download and install an application if it doesn't come with an installer."

    Well i can say that is the complete opposite form everyone I know. Personally, me included, A lot of people do not like installing something without a reason and will not install something unless the know it is worth it.
    I know one of my friends does not install anything on his main computer unless it is in his list of approved and tested programs.

    And that is not even taking into account all the people who cannot install programs. My old job is one example of a place that you were not allowed to and I believe that is quite common (and a smart practice).

    But with all this talk, does anyone know how Spiderweb distributes its programs?

    ReplyDelete
  22. If your employer doesn't want you installing software on your computer, then I'm awfully sure they don't want you downloading random EXE files from the internet, that don't even have anything to do with your work!

    ReplyDelete
  23. Who says it does not have anything to do with my work, and no I absolutely was allowed to run anything that did not install itself.

    ReplyDelete
  24. David desJardins:

    Employers usually control what is installed on a computer because of software licencing issues... (i.e., if someone installs a pirated version of Photoshop on a company PC, the company is liable... if someone brings in their own thumbdrive with photoshop, they are not). In every place I worked, we where allowed to use software off of a thumbdrive.

    Also, if a program isn't installed, you don't have to look around for where its files are stored, because there are only 2 places that it can write files (AppData, and its own directory).

    ReplyDelete
  25. Jonathon Wisnoski said: "Who says it does not have anything to do with my work, and no I absolutely was allowed to run anything that did not install itself."

    Then you could run any program that comes with an istaller! Installers install other programs, not themselves ;-D

    ReplyDelete
  26. The original chi flat iron was released as a professional salon straightener. After gaining popularity on the market the cheap chi flat iron was later released for personal consumer use. If you are a professional stylist or someone who loves straightening their chi hair straightener before leaving for work the Chi original ceramic flat iron is one styling tool you cannot live without. Unlike other wholesale chi flat iron before its day the Chi model was developed with moist ceramic heat technology that does not burn or damage the wholesale chi hair straighteners.

    ReplyDelete
  27. I don't know.

    Maybe your code is so much better quality than mine, but I've had completely different experiences - most (like 99,99%) of bugs I've found in my code - even the non deterministic ones, that happened once - turned out to be my fault.

    I don't count the bugs I've never fixed, because how can I know it was hardware fault?

    If user restarted computer or reinstalled program, and the bug never happened again - it's not a proof it was hardware or OS, or anything - more probably it's some dangling pointer that causes crashes depending on specific memory layout that happens in rare occurences. These bugs can even be reproductable in one computer and never happen on anothers, and disapear when user changes character name or anything in his character.

    Restarting computer or clearing cache files (or deleting saves) will help with such problems, and these problems probably won't happen to the same player anymore, and maybe even to anybody else, but the code is still wrong, and the fault is still mine.

    I can understand not every bug is worth fixing, but be honest about it, don't blame hardware, when you don't really have a clue it was it.

    ReplyDelete
  28. It`s very interesting point of view. Too much information can really confuse people. But On the other hand there are always be people who care about the others, and people who don`t. Everybody is making his or her own choice.
    Purchase femara online from online drug store for discount rates.

    ReplyDelete
  29. Tiffany is about the world's most renowned designers of first-rate offerings.Dissimilar to other vogue companies,tiffany necklaces deal firmly in products for instance jewelry,designer watches,glass wares,lamps,bags,plus more.
    tiffany jewellery
    tiffany silvers
    tiffany uk
    tiffany jewellery uk
    tiffany jewellery sale
    tiffany rings
    tiffany co rings
    tiffany engagement rings
    tiffany bracelets
    tiffany co bracelets
    tiffany necklaces
    tiffany charms
    replica tiffany jewellery
    replica tiffany
    tiffany co
    tiffany and co

    ReplyDelete
  30. Arnold Schwarzenegger leather jacket taken from the movie “Terminator-3” worn by Arnold Schwarzenegger available here,buy
    this superb quality Leather Jacket with worldwide shipping


    Terminator Jacket

    ReplyDelete
  31. womens leather jackets

    celebrities fashion jackets with great quality and designed are avilable here, you can buy these jackets on discounted prices
    with world wide shipping so what is better than that!

    ReplyDelete
  32. Hotonesie has the biggest selection of Kigurumi with over 100 Animal Onesies to choose from! We provide affordable onesies and Free Worldwide Shipping.
    cheap animal onesies
    Wholesale onesies
    wholesale adults onesie

    ReplyDelete