On Playtest

This is going to be a series of articles written about playtesting JAGS. I think there are three major elements of this train of thought:

• Philosophy: What, why, and how do we playtest? What do we hope to gain by it?
• Specific Games: What was a game we ran like? What'd we learn? How'd we do it? What do we think this says about JAGS in general vs. that game in specific?
• How Do We Structure Games: We believe there are a set of best-practices for constructing games and playing JAGS (and, likely, other games of a similar sort). In these posts we're talking about the way we structure games and approach them (and play).

Stuff We're Doing Right Now

Big List of Attacks. Did Quantum-Beams and now working on some Resisted Attacks combined with standard damage beams. I'll note that this is something that I've wanted in many, many other games and have not felt they were especially well handled (and, to be honest, JAGS's approach is pretty complicated--but with luck it can be balanced which is good).

On Playtesting 1

My attention is called to This post on Vincent Baker's blog in which guest blogger Ben Lehman tells game designers: Stop Playtesting. Now certainly he's writing (as we all are) for a specific audience ("indie game designers"--possibly people he knows) but I know or at least suspect his observations are meant fairly globally. They might even apply to JAGS. Here's his table of contents and my paraphrases:

1. Textual Errors: You won't find them from playtesters--no one reads RPG texts! The question will be how well your game's rules are similar to other games they've played!!
2. Rules Problems: You won't find them because playtest isn't all that exhaustive. Do careful textual analysis of your own rules and thought experiments.
3. Mathematic Probabilities: Uh, no--do the math. Playtesting won't tell you anything about probabilities.
4. Marketing: As far as I can tell he assumes you either hit your target market or not and you can't "test that" with a play-test (sample size is too small).
5. Development: If you change rules in playtest because of playtest you are doing the wrong thing. Playtesters are too invested in the moment to have a clear view of how the rules should work and you, even if you aren't playing will be equally influenced by the state of the specific game rather than things like the vision of the rules or goals.
6. Finishing Your Game: If you playtest more than a year it's bullshit. Think of it like being proposed to be married--leaving your time table open-ended and over several years ... means you lack commitment.

What About These And JAGS?

Although over the years I've found a lot of RPG-Theory dialog to be stuff I disagreed with (either vigorously or just somewhat) I'm on-board with the essay*. Let's take a look.

Textual Errors: Over the amount of time that JAGS has been written and re-written I've gotten far enough away from it to have to re-read it as a person who doesn't know the rules (let's be fair: I don't 'just' know the JAGS Rules--I know about seven versions of the JAGS rules. That's for starters, and it's not including the multitude of things I've forgotten). So, yes: playtesting does eventually find things. I've re-read sections of the rules and gone "ooh--that's not clear. What the hell was I thinking there."

That happens--but mainly? The way to catch textual errors is two-fold:

(a) Editing -- and you need a 'rules editor' to do this. Someone who understands the game and/or RPGs in general and can give a really literal read in an editing context --and--

(b) Careful writing. It's hard to do a first draft that's even reasonably clear and accurate. So having to re-write things over and over--forcing yourself to do best-practices of telling people what you are going to tell them before you say it and, especially, breaking down rules into component steps--listing them as steps--and then giving concrete examples? That's a way to catch textual errors.

Look at the combat tracker in JAGS Revised for a good example of this. If we'd done a third example with the characters grappling we'd have caught textual errors there too.

Rules Problems: We've caught rules problems a-plenty during playtesting however we don't fix them then. Sometimes using a rule--especially enforcing a careful read of it--can and will find stuff that doesn't work right. What I agree with Ben on is that trying to fix a rule during playtest isn't even nearly the best time. On the other hand, rules that looked good or made sense when you wrote them often don't work so well on review during play (and, especially, play can bring up edge-conditions that even a well-realized set of rules didn't address).

Mathematical Probabilities: Our extensive work with the simulator means I'm right on-board here. I suspect that, despite what he says in the essay, I'd need to pay someone serious bucks to do the statistical analysis I'm doing with Monte Carlo techniques. I don't think playtesting is at all viable for this sort of thing.

Marketing: We don't market and have only basic ideas of who our target market is so this doesn't apply to my thinking.

Development: I don't think playtesting "to develop" works. We try to reverse engineer our rules when possible by determining how we think they should work and then working backwards to make the rule. Sometimes things do come up during play, however, that change our mind about this. Playtesting is good for shaking your vision up. It isn't good for creating it and 'refining it' needs to happen after the fact.

Finishing Your Game: With about five years and counting on JAGS Revised Archetypes I have to say that I don't think his one-year limit is going to work for us. On the other hand, we aren't spending this time "playtesting" JAGS Revised Archetypes so much as simulating it. We're still learning things--valuable things--about the rules-set that we didn't know a year ago. Changing how we test things is giving us a new perspective on things. So long as that keeps happening I think it's awesome.

-Marco

* His tone is provocative, I think--and I'm not supporting that--but being able to rant is a major motivation for  blogging to begin with.