Wednesday, February 23, 2011

On Playtest

This is going to be a series of articles written about playtesting JAGS. I think there are three major elements of this train of thought:

  • Philosophy: What, why, and how do we playtest? What do we hope to gain by it?
  • Specific Games: What was a game we ran like? What'd we learn? How'd we do it? What do we think this says about JAGS in general vs. that game in specific?
  • How Do We Structure Games: We believe there are a set of best-practices for constructing games and playing JAGS (and, likely, other games of a similar sort). In these posts we're talking about the way we structure games and approach them (and play).
Stuff We're Doing Right Now
Big List of Attacks. Did Quantum-Beams and now working on some Resisted Attacks combined with standard damage beams. I'll note that this is something that I've wanted in many, many other games and have not felt they were especially well handled (and, to be honest, JAGS's approach is pretty complicated--but with luck it can be balanced which is good).

On Playtesting 1
My attention is called to This post on Vincent Baker's blog in which guest blogger Ben Lehman tells game designers: Stop Playtesting. Now certainly he's writing (as we all are) for a specific audience ("indie game designers"--possibly people he knows) but I know or at least suspect his observations are meant fairly globally. They might even apply to JAGS. Here's his table of contents and my paraphrases:
  1. Textual Errors: You won't find them from playtesters--no one reads RPG texts! The question will be how well your game's rules are similar to other games they've played!!
  2. Rules Problems: You won't find them because playtest isn't all that exhaustive. Do careful textual analysis of your own rules and thought experiments.
  3. Mathematic Probabilities: Uh, no--do the math. Playtesting won't tell you anything about probabilities.
  4. Marketing: As far as I can tell he assumes you either hit your target market or not and you can't "test that" with a play-test (sample size is too small).
  5. Development: If you change rules in playtest because of playtest you are doing the wrong thing. Playtesters are too invested in the moment to have a clear view of how the rules should work and you, even if you aren't playing will be equally influenced by the state of the specific game rather than things like the vision of the rules or goals.
  6. Finishing Your Game: If you playtest more than a year it's bullshit. Think of it like being proposed to be married--leaving your time table open-ended and over several years ... means you lack commitment.
What About These And JAGS?
Although over the years I've found a lot of RPG-Theory dialog to be stuff I disagreed with (either vigorously or just somewhat) I'm on-board with the essay*. Let's take a look.

Textual Errors: Over the amount of time that JAGS has been written and re-written I've gotten far enough away from it to have to re-read it as a person who doesn't know the rules (let's be fair: I don't 'just' know the JAGS Rules--I know about seven versions of the JAGS rules. That's for starters, and it's not including the multitude of things I've forgotten). So, yes: playtesting does eventually find things. I've re-read sections of the rules and gone "ooh--that's not clear. What the hell was I thinking there."

That happens--but mainly? The way to catch textual errors is two-fold:
(a) Editing -- and you need a 'rules editor' to do this. Someone who understands the game and/or RPGs in general and can give a really literal read in an editing context --and--
(b) Careful writing. It's hard to do a first draft that's even reasonably clear and accurate. So having to re-write things over and over--forcing yourself to do best-practices of telling people what you are going to tell them before you say it and, especially, breaking down rules into component steps--listing them as steps--and then giving concrete examples? That's a way to catch textual errors.

Look at the combat tracker in JAGS Revised for a good example of this. If we'd done a third example with the characters grappling we'd have caught textual errors there too.

Rules Problems: We've caught rules problems a-plenty during playtesting however we don't fix them then. Sometimes using a rule--especially enforcing a careful read of it--can and will find stuff that doesn't work right. What I agree with Ben on is that trying to fix a rule during playtest isn't even nearly the best time. On the other hand, rules that looked good or made sense when you wrote them often don't work so well on review during play (and, especially, play can bring up edge-conditions that even a well-realized set of rules didn't address).

Mathematical Probabilities: Our extensive work with the simulator means I'm right on-board here. I suspect that, despite what he says in the essay, I'd need to pay someone serious bucks to do the statistical analysis I'm doing with Monte Carlo techniques. I don't think playtesting is at all viable for this sort of thing.

Marketing: We don't market and have only basic ideas of who our target market is so this doesn't apply to my thinking.

Development: I don't think playtesting "to develop" works. We try to reverse engineer our rules when possible by determining how we think they should work and then working backwards to make the rule. Sometimes things do come up during play, however, that change our mind about this. Playtesting is good for shaking your vision up. It isn't good for creating it and 'refining it' needs to happen after the fact.

Finishing Your Game: With about five years and counting on JAGS Revised Archetypes I have to say that I don't think his one-year limit is going to work for us. On the other hand, we aren't spending this time "playtesting" JAGS Revised Archetypes so much as simulating it. We're still learning things--valuable things--about the rules-set that we didn't know a year ago. Changing how we test things is giving us a new perspective on things. So long as that keeps happening I think it's awesome.

* His tone is provocative, I think--and I'm not supporting that--but being able to rant is a major motivation for  blogging to begin with.


  1. While there's some validity to his other parts I think, I have to say that I bluntly think Vince is full of crap about his second and fifth points.

    You might not catch all rules problems with playtest, but that doesn't mean you won't catch any. And frankly, the designer is often too close to the problem to be able to see if the rules are actually serving the ends he's trying to serve; in addition, his own groups and design perspective is going to be as biased as anything playtesters bring to the table, but at least with several groups of playtesters there will be a degree of regression-to-the-mean on this (though the degree to which that's true depends on how playtesters are selected).

    That doesn't mean all playtesting is equally useful, but from what I've seen over the years, not playtesting an RPG is a good damn way to ensure it won't come out right.

  2. Firstly, it's Ben Lehman guest-blogging and not Vincent--but yeah, I think that he veers into the "full of crap" zone with his tone a few times (he makes a point that doesn't care about his tone--but I think it's the strident point making which takes his piece from some cautionary advice into the realm where it is no longer asking for a charitable read and therefore can be analyzed as actual, hysterical advice ... and, of course, found wanting.

    But more than anything else, be aware that he's speaking to a select group of people (the indie community and some specific behaviors he doesn't like (eternal playtesting to keep a game from being criticized)--while I think /he/ might believe he has something to tell Wizards of the Coast or White Wolf about playtesting I'm not sure they'd agree and ultimately they're better judges of what they ought to be doing than, well, anyone else is likely to be.


  3. My apologies directed at Vincent if he reads this; that's what I get for looking too quick.

    As to the rest--while that might be a problem in the indie community (the eternal playtest and the rest) his commentary is still nonsensical, and if followed as he states it would go from games that are never done to games that are done--badly. While I sometimes think you and the other JAGS people are perhaps a little excessively focused on Getting It Right, for example, you seem to have found a reasonable compromise on that--you make material available, then go back and hack at it some more.

    Basically, I think if you're serious about a game, you hack at it until its playable (including playtesting) and send the game out; then you refine it more, and if you think you've done enough, you do a new edition.

    (This does relate to one thing I've thought about saying; you guys really should think about making some version of the Archetypes document more generally available, even if (as is obvious) you want to make another run at it. Currently someone who is trying to use JAGS and didn't happen to be around when all the older ad-hoc Archetype material was accessible is left without much support for paranormal abilities at all, and that's not doing them a favor. I realize you aren't as happy with some elements of the Archetypes 1.7 version as you could be (and its obviously missing some things like how to use it with spells and the like) but it really is better than nothing, and that's kind of what someone trying to get into JAGS right now is stuck with. Even something like my possible Morrow campaign really needed some support for psionics; if I didn't have the 1.7 document or the old psionics rules, I'd be having to try and make them up myself, and that's always an intimidating process with a game system you're just getting into).

  4. Kind of an addendum to the last; at some point I think it might be interesting and useful to do a post about what you'd suggest to someone newly getting into JAGS that they modify now, in the light of some of the things you've learned since JAGS Revised came out. I know you've mentioned in email conversations and the like a few things about the cost of higher roll skills (and I'd presume the attributes that could default to them). That might be useful in a "what should I houserule now" kind of way if you feel up to it.