Judge Academy > Transcript – October First Week: Managing Criticism

Transcript – October First Week: Managing Criticism

12:48:46 pm – Jonah Kellman:
Good morning, good afternoon and good evening friends from around the world. Today we are joined for first week with Chris Wilson managing director, from Grinding Gear Games, the organization that produces path of exile, an action RPG, Good morning afternoon Chris.

12:49:02 pm – Chris Wilson:
Yeah, thanks very much for having me. It’s great to be here.

12:49:04 pm – Jonah Kellman:
So to start off, why don’t you introduce yourself? Who are you and how are you connected to magic? You don’t most of the guests. We have have been tournament organizers or other judges. You do not directly. They’re not directly involved in creating magic tournaments, but you have a history with the game. Is that correct?

12:49:26 pm – Chris Wilson:
That so professionally. I run grinding gear games, we make path of excel as you mentioned. I’ve been doing this for 16 years now next month and from a personal point of view, magic is my primary hobby. I started playing as soon as revised got to New Zealand back in, you know, 1994 and I’ve been playing a collecting ever since Um, you know, played it pretty much every pre-release all the way through vintage old school tournaments. You know, haven’t placed that highly that I’ve really enjoyed the game and I’ve also as of the last 10 or 12 years been focusing, a lot more on my actual collection of the game. So I do find that actually helps my work a lot because games, like Path of Exile are very much about acquiring digital assets. And so having a extensive card collection and understanding the motivations of that actually, you know, plays in pretty well.

12:50:11 pm – Jonah Kellman:
That’s fascinating. What’s your favorite format that you play these days?

12:50:15 pm – Chris Wilson:
Um, I prefer old school. I was a big vintage fan for the very long for a very long time, but I think the pace at which they’re printing, you cards, makes it very difficult to keep up with whatever the new tech is in vintage, which is kind of ironic for a format. That’s just mostly terminal, killing

12:50:28 pm – Jonah Kellman:
Yeah, that makes a lot of sense.

12:50:29 pm – Chris Wilson:
loaded and

12:50:31 pm – Jonah Kellman:
So let’s dive into the questions. We have this outlined here. Normally, after about a question or two, I deviate from the outline, just spin off your answers, or jump ahead because things connect that way. But most of our guests, as I

12:50:44 pm – Chris Wilson:

12:50:45 pm – Jonah Kellman:
mentioned, in the past, have been judges and torment organizers. But you have developed some skills that are, I think, deeply connected to great leaders. You’ve been running grinding your games from initially, it was just three folks in a garage to. Now, hundreds of people,

12:51:02 pm – Chris Wilson:
Yeah. Yeah, 165 or so currently people in house?

12:51:05 pm – Jonah Kellman:
Yeah, so I’m gonna just run down a couple of common judge tasks and I’d like to hear if those sound like a thing. You do in your day today. Giving broad direction and goals to team leaders and then providing feedback on their execution.

12:51:19 pm – Chris Wilson:
Yep. Absolutely, we have many departments that have a lot to achieve and they need their feedback.

12:51:24 pm – Jonah Kellman:
Making announcements to the public.

12:51:26 pm – Chris Wilson:
Yep, that’s certainly a big part of the job and understanding how people will react in making sure that the communication is clear is very important.

12:51:33 pm – Jonah Kellman:
Huh? Getting involved, when things go really wrong, to either provide, help or support.

12:51:37 pm – Chris Wilson:
Yep, with live operation games, things can go wrong very quickly at any time of the day and then it’s very important to make sure that everyone’s calm and expectations are managed.

12:51:45 pm – Jonah Kellman:
And then explaining philosophy and why certain decisions were made judging it be of ruling and path of exile. Maybe a patch update.

12:51:54 pm – Chris Wilson:
Yeah, that’s that’s also a good one. We we find that especially in game development. There are decisions which some people won’t like like a game is fun in some part because it’s frustrating and finding the right frustrations and being able to explain why those ones lead to fun. It’s certainly a big part of it and I can imagine that also applies with judging because the game is only fun of the rules are applied fairly and therefore, it’s important to explain to someone who’s relatively heated that for fairness. We’re actually going to follow the rules here rather than let you win the game.

12:52:20 pm – Jonah Kellman:
Yeah. Huh. Something that I am aware of with both judging. And with game design is no matter Yeah. Huh. both judging and with game design is both judging. And with game design is no matter Yeah. Something that I am aware of with both judging and with game design is both judging. Yeah, huh. of with both judging and with game design, is no matter. What decision you make, somebody will be unhappy. I remember a World of Warcraft designer saying If we just sent everybody free resources, some people would be excited. And other people would be so mad. And there’s no way to make everybody happy. So many judges start judging as an extension of a hobby, many people like you, like magic, and they went to different direction with it. They went to their local stores and became judges. I started playing in, I want to say 2,000 roughly, so I haven’t been playing nearly as long as you. And magic has been passion, but it’s turned into a job. This is what I do day today. How do you maintain your passion for a project that’s been consuming your life for at this point, a decade and a half. How do you manage keeping from burning out with path of exile?

12:53:19 pm – Chris Wilson:
While it also helps deaf metric is a hobby because that’s something I can turn to where I don’t have to think about work. If you do anything right, I can focus

12:53:25 pm – Jonah Kellman:

12:53:25 pm – Chris Wilson:
on my cards and it’s honestly managing a big collection and has a lot of them and danity to it. And therefore, there’s a lot of just like sleeping cards and sorting them that I can do while chilling out, but with regard to passion, it really helps in my case because I get to decide what part of Excel is and created it. You know, from whole cloth with my friends to make sure to be we got to make something that we wanted to work on and therefore we’re still working on it decades later because it was the correct thing for us and I suspect that someone would only get into magic judging if they love the game to the same degree like if they know that magic is the thing for them, that will lead them to put the effort in and, you know, the sacrifice is required to become a good judge.

12:54:02 pm – Jonah Kellman:
Yeah, I think that’s very true and I think that a lot of folks while they can’t make the game what they want to be because they don’t work for wizards, the coast, they can make their community what they want it to be, they can either help

12:54:21 pm – Chris Wilson:
Yeah. And I think, from a point of view of

12:54:23 pm – Jonah Kellman:
They can either help develop a very

12:54:23 pm – Chris Wilson:
managing burnout, and that kind of stuff is just important to take time

12:54:26 pm – Jonah Kellman:
competitive one or something that’s

12:54:26 pm – Chris Wilson:
for the things outside of one’s primary focus.

12:54:28 pm – Jonah Kellman:
more casual

12:54:29 pm – Chris Wilson:
Right. You know time to exercise time to eat

12:54:31 pm – Jonah Kellman:
Casual focused around Casual focused around Commander,

12:54:31 pm – Chris Wilson:
eat healthy and some games, right? And that may mean someone playing a

12:54:35 pm – Jonah Kellman:
maybe help bring up a vintage or old

12:54:35 pm – Chris Wilson:
And that may mean someone playing a And that may mean someone playing a different card game, potentially or something like that so they get some something like that so they get some which they’re treating as work where you know, they have a

12:54:40 pm – Jonah Kellman:
school community in their area.

12:54:42 pm – Chris Wilson:
you know, they have a you know, they have a thousand rules you know they have a thousand rules on their head and overly critical of the casual game that they’re playing and so on and so on.

12:54:46 pm – Jonah Kellman:
Yeah, it’s burnout something that you see in members of your team. Like, do many people who work for ggg come from the fan base and Path was their hobby and now it’s their job. Do you see them burning out on the game or on work? And if so, how do you help them out? Do you give them advice?

12:55:06 pm – Chris Wilson:
Where we see a few aspects of this. The first one is that once someone has worked on the game in an intimate detail, it affects their relationship with how they play it, which isn’t work burnout, but it can affect how they play In some cases. It makes them play with even more rigor, they can be a better player because they know exactly how the game works. So basically as a crash course, enforcing them to study because like Magic Path of Excel has a lot of interactions. So it would be like if you wrote learned every metric card, right? Like it’s helpful to some extent to understand more possibilities. The opponent has all, you know, what you could name with, meddling major or whatever the analogy is.

12:55:37 pm – Jonah Kellman:

12:55:38 pm – Chris Wilson:
It’s certainly does affect people’s relationship with playing likewise. Some people just stop playing when they start working on it because it’s their day job and they need to play something else in their spare time with regard to actual like, Work Burnout into Someone Having performance issues at work, Because of the fact that you’re working too hard, we try very hard to make sure that we can recognize the signs of this and to get people time off as necessary. And we have a pretty cyclic release cycle, and this may be true for metric as well. I’m not sure if judging is cyclic, let’s set releases or GPS or various other, you know. Types of events like that, but we have quite a lot of push around, our releases, not necessarily crunch to finish it. Ideally the schedule means the things are finished with our too much overtime, but there’s certainly a lot more work fixing bugs and stuff around the release and then it gets a little bit quieter in the, the main part of development which is where we are in the current leak cycle. For example, today is a moderately quiet day because they’re in a fires, we get to actually do some work and those days are very therapeutic for team members who kind of get to chill work at their own pace. They don’t have a, you know, deadline breathing down their neck and I suspect that the same is probably true for judging where if you’ve just judged a huge four-day event and not got very much sleep and you’ve been traveling and so on maybe it’s good to take some time to not do that for a little while.

12:56:48 pm – Jonah Kellman:
That is absolutely very true. I know during some of my more intense periods. I ran Several events back to back, to back to back and taking two weeks home and just relaxing from that has been very valuable. Now, branching off from that now that you’re at the top of a large Org chart with Dozens of people that report directly to you. Well, you don’t have day-to-day oversight over every single person in the organization because that would be a lot or at least I assume you don’t, What do you do to help build a sense of camaraderie? What That’s the first question here, team building and feedback. How do you work around that?

12:57:28 pm – Chris Wilson:
In our case, we find it useful to involve as many people as possible with the design to make sure that they’re in design meetings, they get, you know, updates about how things are going, get a chance to give their feedback play test, early versions of the content we’re working on. That really helps people feel that they can have an impact in working on it. Now that’s obviously something that works well in a creative endeavor because you can give everyone a sense of buy in there. I don’t understand how that would apply to judging necessarily, but we find that like just general of like philosophical chat with devs is worth having, you know, just talking about, you know, see what see what they want to discuss with the game and give opinions about that and take their feedback into account, That’s really help build kind of a closer than the team. And for the team members who have been less willing to have chats like that. It’s more professional arms and relationship. You know, How is your work going? It’s going good. Thank you. Kind of conversation as the first to actually getting into the nitty gritty of why things are how they are.

12:58:20 pm – Jonah Kellman:
That makes sense. I think the translation to magic and judging is talking about how we approach things from a logistical sense. When you’re running a tournament, how you’re running the end of round and how you’re collecting that last bit of information. There are a bunch Different ways you can go about that. And so talking with your team to figure out what strategies they’d like to implement to execute it, depending on the size of the event, the technology that you have available, that sort of thing, lets you engage more directly with them. So I think you have that same opportunity and similarly you can talk with judges about policy philosophy. Well, the rules are set in stone policy, which is how we handle issues that come up in games has a little bit more flexibility to it. And so if somebody Looks at an extra card during a game. How exactly you handle that where you draw lines? Because we’re dealing with human interaction. There aren’t hard lines and policy everywhere. And there are a lot of judgment calls

12:59:17 pm – Chris Wilson:

12:59:18 pm – Jonah Kellman:
and so discussing judgment calls Can I think also develop that sense of camaraderie that you’re talking about.

12:59:23 pm – Chris Wilson:
I understand I’ve noticed that myself at magic events where when there’s any kind of borderline situation that isn’t a hard rules thing, especially when it’s looking, you know, like, you know, unsportsome like conduct or other kind of things where you have to kind of make a judgment. I’ve noticed that certain judges work really well together. They’ve got the camaraderie. They can easily communicate, they understand what other people are going to feel about it versus the ones who are like brought him for the event and haven’t developed that group feeling. To some extent. I mean, obviously there’s a hierarchy, so they can’t argue with each other too visibly. But there’s a bit more tension between the judges and that makes it seem like a less well run event. If there isn’t unity between the different judges.

12:59:59 pm – Jonah Kellman:
That’s a fascinating insight. That’s not something that I necessarily realize. Everybody had picked up on, but that’s great to hear now. talking a little bit more about feedback and working with individuals have any particular techniques for

1:00:13 pm – Chris Wilson:

1:00:16 pm – Jonah Kellman:
don’t feel delivering critical feedback, particularly more negative feedback would be critical positive feedback like This is very good. You stop doing this. Can you do it? Some more Again We like to seeing that.

1:00:28 pm – Chris Wilson:
Right, I find that early honesty is best, right? Like sugarcoating, things, everyone in my company and also in the judging community is an intelligent person, right? We’re dealing with smart. People who’ve been definitely read between the lines. And so, I found it important to basically, as early as possible after the thing happened once, you know, once there’s no heat in the moment anymore, but not delaying any further to sit down and go. Hey, so that wasn’t good. Let’s talk about what why, and how. And so on, or Hey, you may not be aware. That something happened that’s actually being negative for someone else or for the product, and it’s no big deal, but let’s try to stop it becoming a big deal and I make sure that like, usually when someone is making a mistake or something, bad has happened, you they know that something was up and so having management or head judge or whatever address it and say, Look, obviously not great, but here’s, you know, it’s not going to be a big problem for anyone, but let’s make sure this doesn’t happen again. We’ll take the tension out of that person because they feel the issues being addressed. They’re not wondering whether it’s gonna be brought up. It’s rare for That kind of stuff to really have to escalate any further than that, like a polite conversation about it from, you know, someone in in a position of control generally will cause someone to be very aware of the issue and take it seriously.

1:01:38 pm – Jonah Kellman:
Right, I presume. That’s not something that you’re generally involved in in a day-to-day. It’s generally your team leads to a reporting to you or

1:01:46 pm – Chris Wilson:

1:01:46 pm – Jonah Kellman:
There’s someone else.

1:01:47 pm – Chris Wilson:
I encourage team leads to sort out their own team problems because it’s really scary for someone to drag in a, you know, the managing director into the conversation.

1:01:54 pm – Jonah Kellman:

1:01:54 pm – Chris Wilson:
But proactively saw the problems of myself if I can do because I try to be involved in everything. Like I am essentially the team lead for the things that I am sorting the problems out for so. I mean, I guess I don’t know if there is like a person completely in charge of judging in, you know, United States. Whoever on fears, for example, or whatever. You know, getting them involved will be a major major thing. But, If that person had a close relationship with many of the judges already, then them saying Hey I noticed a thing. Let’s make sure that doesn’t happen, would be more appropriate.

1:02:22 pm – Jonah Kellman:
so this isn’t a question that I had on our outline, but In this conversation and it’s come up, but I think in the path of exile community, you have a reputation of approachability. I think that a lot of path of exile players, feel like they can directly speak to you about issues and I’m guessing that that extends to members of your team. Do you feel that that’s true? And if so, how do you cultivate a position of approachability while also being in a potentially intimidating position as directing manager for an organization? Like Ggg

1:02:57 pm – Chris Wilson:
Well, I mean, I just answer people’s questions when they come to me in their naturally more people, do. I mean, there’s, there’s a bit of a cue right player. So email me would know that sometimes I can’t answer emails, you know, if there’s just too many that day and they don’t need an answer, then I probably won’t. But when I do, there’s a bit of a delay to it and team members will say, Hey, I need to chat you about something. I’ll definitely make time for them, but it may be a look. I’ve got four people in the queue ahead of you right now. Let’s deal with this later, kind of thing. So there it’s mostly a matter of never saying, no to anyone, keeping a list of people you need to get back to you know just making sure that everyone who wanted a bit of your attention actually gets. It will help cultivate that and it is positive. Of course it doesn’t mean having to deal with things that could go to other people. Like I have this analogy of I don’t know, you gotta some shopping place, like Walmart, say in the States and there’s a mess, like someone spilled something on the floor, so you don’t get on the phone to the CEO of Walmart and go, Hey, I know you got like a thousand stores but in this particular, one, there’s some stuff on the floor. Can you make sure someone deals with that because it’s messy. But yet people will send emails to me about smaller counter shoes. I mean, I forward them to the customer support team but it’s interesting what people expect in terms of viability to help. And so, the philosophy I apply, which may be relevant here. Is if people are coming to me with individual things that need investigation that I can’t easily delegate is, I have the approach of trying to fix issues that affect a lot of players rather than affect just one player. You know, I would rather spend my time if I spend my day solving problems that affect a million players, each that is the best impact for everyone. Whereas if I spend my day and solve for individual players problems, it doesn’t really generate as any money or further the product, or the company or anything like that. It’s a nice thing to do and I’ll try to find other people to help those. But for my own personal time, I want to make sure to solve big problems now having Said that individual staff issues to do with morale in various other career, development stuff are big problems, right? Like I will spend, as much time as necessary to help my staff members feel as comfortable and as fulfilled as they can be. Because having staff members who are unhappy or staff members, who are achieving their potential has a massive follow-on effect for the next years of the game. So I have a bit of a different approach to internal stuff versus the

1:05:00 pm – Jonah Kellman:

1:05:01 pm – Chris Wilson:
facing stuff.

1:05:02 pm – Jonah Kellman:
That makes a lot of sense. One of the things that I think a lot of new team leads in judging struggle with is figuring out what Problems or crises, they need to deal with. If something is going wrong, that’s under their authority. Should they delegated to somebody else? Or should they handle themselves? And I think a lot of new team leads. They see a problem. They know I can solve this and I can solve it faster than explain the problem to somebody else. Tell them how to fix it, and have them solve it. So I’m just gonna solve the problem, but then you have the CEO going down to Walmart with a bucket and a mom.

1:05:37 pm – Chris Wilson:

1:05:37 pm – Jonah Kellman:
and I think being able to delegate that to the appropriate individuals as a skill, that is very important and helps with that approachability because people know that they’re going to be Treated with an given an expert to work on the problem.

1:05:50 pm – Chris Wilson:
I am a completely guilty of solving small problems myself because I know that I can, right, if someone contacts me and says, Hey can you sort this thing? I can just sort that like if there’s a typo in a news post, right? Like thing up on path of exile.com

1:06:01 pm – Jonah Kellman:

1:06:02 pm – Chris Wilson:
and there’s a spelling mistake and that’s right. So it takes less time for me to click edit fix this. Filling mistake and hit save, then it does the message someone which will sound passive aggressive to message someone on the team going. Hey, there’s a type. Is a is a weird thing to message your teammate. But it’s better to do that because then they understand that a it’ll get noticed speed, they need to be more careful. See it may give some familiarity with fixing it and so on and so on.

1:06:27 pm – Jonah Kellman:
yeah, I could definitely like, I have

1:06:28 pm – Chris Wilson:

1:06:29 pm – Jonah Kellman:
a great relationship with my boss, but I could still be a little just like, why did you email me that? I had a typo in the article. You could have just fixed it, but There’s value in both directions. Now. One of the things that judges do, we enforce rules and policy and A rule against the player, whether it be from a judgment call or an objective call based on the comprehensive rules can spark frustration and criticism. You are the public face of grinding your games and path of exile. How do you manage receiving emotionally fuel feedback and particularly criticism?

1:07:08 pm – Chris Wilson:
Well, every time that someone takes the time to write in something very angry is because they are passionate about the product. They, this is their thing. They enjoy doing kids their hobby. It’s the thing. They drive. Joy from that means they like it. That means they like me and whatever achieved and made. But right now, they’re not having fun for some reason. Either, a valid reason or an invalid reason, but that person isn’t having fun and they’re expressing it in an inappropriate but still expressive way. And so I have to understand this is someone who has been touched by the game, someone who maybe I can learn what the problem is, but it’s not valuable to take it. Personally, I’ve developed very thick skin over time like Yeah. It’s just I can emotionally not get charged by reading that the interesting emails that come in. Having said that I do take it, I do take a very poorly when people criticize teammates, especially teammates are working very hard. Like, it sucks. When there’s someone who, you know, will literally in some cases, forgo time, you know with it like newborn child in order to get a thing done in time, right? They’ll sacrifice time that they can’t get back in order to make a thing special for the players. And then they get called out by name publicly and then they feel really crap and they can’t turn up to work the next day because they feel bad that kind of thing there is terrible. And so, you know, I have a zero tolerance policy for players criticizing team members, they can throw whatever they want at me and so, Because, you know, some communities online are kind of angry by default. I try very hard, not to engage with reading them and make sure that I have my community team summarize, that kind of stuff just for my mental health of, not seeing things, be criticized team members.

1:08:37 pm – Jonah Kellman:

1:08:38 pm – Chris Wilson:
And so like, for example, I generally don’t load up our subreddit and read every comment every single day. But I do have community members summarize. All right, so the community players are very angry about this issue here. Here’s the core points, blah blah, and and I read the summary. And if necessary go get into the thread, usually regretting it, because of the wording people use, but Yeah. The other the other piece of advice would this is when someone approaches you with criticism or whatever, it’s often the case that they legitimately have a problem, their solution that they have is probably a solution that’s relatively one-sided or hasn’t been fully thought through. It doesn’t know the entire context. And so looking at their thing, they present you the solution will seem ridiculous and it’s easy to say, Well, that’s stupid. I’m not doing that. But the correct thing to do is say, Okay, solution is stupid agree, that’s not happening, but there is a problem here. They people can assess that there is a problem, they feel there is one. And so, coming up with a better solution to the problem is a good way of helping that person, and we found it quite useful to look at it in terms of phrasing things as problems and solutions when communicating with players they’ll say Hey, this game is crap now because I can’t find these items. I used to be able to find and we should make the items drop more often, right? It’s like saying, you know, I I enjoyed Double Masters because I was getting too rare’s per pack and now

1:10:01 pm – Jonah Kellman:

1:10:01 pm – Chris Wilson:
who rarest per pack in dominaria, please as well, because it’s like twice as many packs.

1:10:06 pm – Jonah Kellman:

1:10:07 pm – Chris Wilson:
You could look at the problem there as being, I found this exciting and I am not finding this exciting, and then find another way to make it exciting. That doesn’t evolve reducing the EV of the Rares and the set or whatever, you know, as a magic design kind of thing. And I don’t understand necessarily how it directly relates to judging because to some extent. It’s rare for magic, place to suggest a solution to the judge is more a matter of, you know, than taking their hand up and saying, something went wrong and be useful because then

1:10:32 pm – Jonah Kellman:

1:10:33 pm – Chris Wilson:
you don’t have to deal with the fact that they’re kind of suggesting a rules then you don’t have to deal with the fact that they’re kind of suggesting a rules interpretation, or maybe they do, Because then you don’t have to deal with the fact that they’re kind of suggesting a rules interpretation, or maybe they do, I’m not sure.

1:10:39 pm – Jonah Kellman:
Sometimes players do suggest fixes like, Oh, let’s just move this over here and move that over there. And we’re like, well That’s good for you, but not fair for your opponent who hasn’t made a mistake and is at a disadvantage because of this error.

1:10:54 pm – Chris Wilson:

1:10:54 pm – Jonah Kellman:
And so, And so players will very often suggest a fix, that doesn’t quite work up. Philosophically.

1:11:03 pm – Chris Wilson:
Mmm, or I could say the opponent and the player both agreeing that a certain fix would fix everything that totally happy. But that would be inconsistent with every other game of magic that was judged in the past and we have to be

1:11:13 pm – Jonah Kellman:

1:11:14 pm – Chris Wilson:
the same

1:11:14 pm – Jonah Kellman:
That’s another very common. Well, we can just fix it this way. It’ll be fine. We’re like that. Doesn’t If you play in another event, you’re not gonna get the same solution and in another round, somebody might not get the same solution and it wouldn’t be fair to them and we want to have a consistent game.

1:11:30 pm – Chris Wilson:
Yeah, and we feel fairness is very important with our game. In particular, like one of the things path of Excel offers is a fair playing field where players can’t pay for advantage and so on. But there’s also means fairness with regard to the way that rules are enforced. And making sure that like I don’t know. For example we recently started selling tickets to a convention that we’re holding next year to do a bunch of like reveals for path of Excel too and users will come to us and say, Hey, look, you must have like a spare ticket of this type because I accidentally screwed up and they did blah blah, and it would be really nice here. If you could, just give me that one ticket. I know you’ve got it, and it’ll be certainly very easy to say. Actually, we probably do have one spare, because someone refunded one. Here is your ticket and fix their problem. We are really nice thing to do. But this the other emails of the same

1:12:11 pm – Jonah Kellman:

1:12:13 pm – Chris Wilson:
issue, and for fairness we have to say, Look, guys, there’s lots of you, we’re going to try to work out a fair way, which may be saying to the first person. You were the first person to ask the first person to try to purchase this ticket type and fail, and therefore, we feel that order of time as a fair metric. So yes, you do, get it. But at least there would be choice behind why there is a bias, you know, a choice that everyone could agree with.

1:12:34 pm – Jonah Kellman:

1:12:35 pm – Chris Wilson:
another example, relating to our

1:12:36 pm – Jonah Kellman:

1:12:37 pm – Chris Wilson:
staff as we have this dinner at that convention, that we can bring some stuff to. It’s like 80 staff and 200 plus players. And so we have to choose which staff members are going to go to this, and we have a lot more than 80 staff. And so using like, how long you have been at the company is a probably fairer metric, then whether you are a team lead or how much you are paid or whether I like you or various other factors. And so we’re trying to find something we’re winning with us. What the rule is, we say, Oh, we ordered people by time at the company or maybe people who missed out on last time to dinner first, then a time at the company or just something, where if you listen to it at face value, you can’t really argue with the fairness of it. And we found, it helps to solve this problems by just establishing rules that at least there is a rule there. Even if they disagree with the fact that it biases against them or something, at least we can say, it wasn’t arbitrary. It wasn’t based on human emotion. You know, it wasn’t you made it.

1:13:27 pm – Jonah Kellman:

1:13:28 pm – Chris Wilson:
Plea of look, I have to win this tournament because, you know, my wife’s been telling me, I never win tournaments and like, I just have to win one, you know? You feel sorry for the guy but you can’t run a tournament based on that.

1:13:39 pm – Jonah Kellman:
Right. most relaxed levels says, Even at the most relaxed levels Right? Part of our policy. Even at the most relaxed level says, Don’t take into account a player’s experience or age when you’re making a ruling. You need to be fair and treat everybody equally and What you were talking about with ticket to something that comes up fairly often with events that have limited registration sometimes for a particular set, if it’s a very popular draft format, we’ll have a limited number of seats in a given draft or in a sealed tournament. And so we’re like, Okay, the first 64 people to register, will have 65 people interested in it or a hundred. And we’ll be like, okay first 64, Some people have now dropped from the event. We’re gonna look to see who tried to register first. Very similar to your system. It’s just Executing on fairness and having a rule that can be followed.

1:14:28 pm – Chris Wilson:
Right. Yeah, And if someone amazing like Lsv walks into the room minutes later when there’s already a hundred people on the waitlist and says I want to play you have to say I’m sorry there’s a waitlist and you’re at the end of it even would be to have a pro

1:14:38 pm – Jonah Kellman:

1:14:39 pm – Chris Wilson:
would be to have a pro player playing in that event and stuff. It’s not fair to do so.

1:14:43 pm – Jonah Kellman:
Right. Now, this is gonna be a little bit of a turn. So one of the things that path of exile does is launch new content, every few months. This is Almost actually synced up with magic releases. So it’s about that study of a cadence for major releases every year. These leagues these launches see a spike in player engagement really require all hands-on deck. There are also a lot of opportunities for things to go wrong when you’re starting something new, even if it’s something you’ve done frequently, This reminds me of launching a large event even though I’ve done at this point dozens or hundreds of sealed events and drafts getting the player seated, getting them to open their packs. Getting them into debt construction starting round one is far away the hardest in most chaotic part of the day and so we launches I presume are the most chaotic part of your year. As person in position of leadership and authority. What do you what tips do you have for supporting your team? When things look to be on the verge of disaster? Especially when it’s a situation where stepping into fix things personally is likely detrimental.

1:15:57 pm – Chris Wilson:
Well, that’s kind of all hands on deck around there. So everyone’s doing their partner. If I have to step in and do my part that I absolutely will be like, I don’t think anyone’s ever had a problem with me fixing things myself, it’s tricky because there’s a kind of an element of human fatigue to leave launches as well like, In the preparation for getting everything. Ready things are busy beforehand in leagues that are managed badly and have poor production or you know, unexpected surprises. Then there are some, you know, some amount of overtime that’s going on beforehand and ones that I managed Well, that’s less and people can be better rested, but we do our league launches on a Saturday morning and this is analogous too. So on potentially working all week and then judging an event to the weekend, right? Especially if they have some setup, I don’t know. Understand if it’s set up and they might be something, they’re doing to volunteer, maybe they’re up, you know, playing magic with their friends like before or whatever, but they’re tired. And then this league launches occurring or An event on a Saturday morning. I’m just gonna go through the whole weekend and it’s gonna be long and it’s going to go schedule and it’s Tricky. Because we get a situation where we have to almost manage sleeping. Rosters for staff. I mean this is all voluntary. Of course, people are excited to be there and we have people who can step in but the passionate ones who really want to be there at launch even after working. All week, certainly, we have to acknowledge this person is tired. Now, they’re going to make mistakes, they need to go and take a break, and we try very hard to force people to have a day of weekend and like when I’m, you know, overoster myself on doing production work and making patches. So that the guy who would be doing, it gets a day off after having done so much work and that kind of stuff is important. And also, when things are going wrong, making sure that people understand that a, the financial impact of this to the company is quantifiable and not necessarily as bad as they’re worrying, and be, it’s not their fault, right? Like, you know, I’m happy to take any blame because whatever happened happened on my watch. You know, if someone didn’t do something well appropriate. If so impressed the wrong button due to tiredness. That’s my fault for working them too hard. Someone press the wrong button due to like port training. That’s my fault for not training, them better. If they press the wrong button because as badly labeled, then I should make sure the buttons are better labeled and so on and so on and so on, you know, there’s there’s everything is preventable with enough foresight and so I’m happy to take that responsibility into reassure someone. Yep, some bad stuff is happening and technically you were the one that press the button that made this bad thing but a like, honestly Don’t worry about it. The league launch is going great still, you know, it’s gonna not have a particular, large impact. Things are still above average, It’s good. Also not your fault. I shouldn’t have the button there, you know, that kind of thing and that helps a bit because there’s probably a feeling of I don’t understand this from an employee’s point of view, but my impression is that were someone to be to feel like a cog in a machine and then their cock breaks that they’re letting everyone down or losing more money than their salary is worth or various other negative things. Maybe they worry they’re gonna get fired over it. And a quick chat of like Halo is totally fine, right? Like we’ll look into later making that button safer. But for now let’s just carry on with running this that really helps quite a lot, and I don’t know if there’s necessarily an analogy with judges, but if someone, you know, someone marks up the event is still occurring and maybe two players are angry now. And maybe there’s a bit of a, maybe there’s a bit of some, you know, Twitter drama about the fact that there was a bad judge call, but the event is still running, everyone else is having fun and we can fix that.

1:19:10 pm – Jonah Kellman:
Yeah, I think that’s a very important lesson for judges to learn. And for folks, in game design, sp In general to understand while. Game. The game might be a huge portion of your life. It might be your career. In the end, this is not generally life or death. This is a game I was managing an event and I for the first time I was in charge of a staff of 30% 30 or so people. And I talked to my boss and it’s like, Hey, I’m really nervous that I’m gonna screw this up and he was like, So keep in mind that you are teaching people to play games, the worst case scenario is Somebody has a little bit less fun playing a game. I was like, Oh, that’s very useful perspective knowing that the cost of a stroke while can mean somebody has a bad day or even a bad weekend, This is unlikely to be something that. Impacts somebody’s. whole livelihood just from a simple

1:20:12 pm – Chris Wilson:
Mmm. Yeah.

1:20:13 pm – Jonah Kellman:
like, making a mistake on a ruling being like, Oh, right, that trigger wasn’t actually detrimental. I gave you a warning that I shouldn’t have. That warning is going to go away at the end of the tournament and nobody’s gonna know that that mistake ever occurred. As an extension of that, as an extension of dealing with errors. As previously mentioned, you’re the face of the organization and oftentimes, when the community is upset, they call on you, to fix it. How do you apologize for mistakes or speak to errors or misreads of Either the community or what the game needed or that sort of thing.

1:20:57 pm – Chris Wilson:
The tricky thing is here is that I am generally apologizing for mistakes that I was unaware of at the time that they were made as in We have a large team, lots of people making decisions. And my time is spread thin enough that I don’t get to check every decision or be aware of it or communicate it. And often people are not angry at the factor changes made. They’re angry at the communication about the change not being clear enough, right? Like we could post a thing today that blatantly said We’re going to meet this change. Yes, people would disagree with that, but it’s very rare for them to personally blame over that one. But if there’s a change, there’s even minor that gets in where we didn’t properly, explain the consequences that’s enough forgivable thing because we owe them. Good communication. And so Frequently. The apologies that I have to make are ones where something will get changed. I will not understand the consequences or not understand that it was changed, which is my fault. I should understand everything to do with it. Or maybe I do understand it and then I do a poor job of explaining it to the players and then when they find out the ramifications of it, they then you know, call from my head which is fair enough. And so I approach each, I approach each one of these by explaining, Here’s what happened. I let people down by not understanding this and then I try very hard to put something in place to prevent that happening again. And so I feel that just saying I’m sorry becomes empty after a while, right? You know people won’t actions rather than words you should include. And I’m sorry because it’s path. You know, it’s psychopathic not to sociopathic not to but if you It’s important to explain what caused that problem. So, people have a bit of, you know, understanding of the human element. It’s very important to not in any way. Lay any blame on team members, even if they directly did the thing because it was my responsibility to

1:22:38 pm – Jonah Kellman:

1:22:38 pm – Chris Wilson:
suit that thing and we don’t want the community going, after the team members who will just quit and response to being targeted because it’s not worth it for them to be, you know, the focus of reddit death

1:22:48 pm – Jonah Kellman:

1:22:49 pm – Chris Wilson:
laser. It’s also important to explain the steps that are being done to prevented in the future which for technical things, enables us to where it means that we basically can put a technical thing in place like to give a random example. We have servers, they have gained processes, running on them. If a server was turned off and it had an out-of-date old version of the game running on it and then we turned it back on. Could that thing then connect to our main server cluster and try to have an old version of the game? We’re going to play a connects to they might you know have data saved in an incorrect way or whatever and so that had happened before because of the fact there wasn’t versioning on the service saying Hey I am this exact timestamped version of check some version of the game in the master server saying, No get out. You’re not part of the current cloud basically and so we fixed that by implementing technical countermeasures so that they have versioning and they they sign in properly and that means that were this to occur in the future would be physically impossible. And so that’s something where when the players say what went wrong here and how you trying to fix that? We can explain this should not happen again. If it happens again, no, there’ll be a mistake and they will respond by finding the cause of that mistake and fixing that as well, the same can be done for decisions, it could be that you say, Look as a, you know, a head judge for this event. There was a case where someone checked a DQ with me and I just didn’t Even listen, I said, whatever DQ away, I don’t care in the future, I will not do that anymore. You know, in the future, I’ll take the seriously. I promise that was a big mistake and theoretically as long as you uphold your promise there and learn from it then that’s okay. So those are the elements of our apologies. I don’t know if they help or not, but we do them because they you know, they feel like the right thing to do.

1:24:19 pm – Jonah Kellman:
yeah, I mean it does come up fairly frequently as a head judge of a larger tournament. You’re generally not apologizing for excessive disqualifications but it’s very frequent that a player will come up to me when I’m either a judge or a head judge and be like three rounds ago. This judge gave me this ruling and it sounds weird. Is it correct? I’m like, unfortunately, that ruling was incorrect. I can’t do anything about that now, because you, that was three hours ago, you’ve the game is over, and your next game is over in your next game is over. But here’s what we have in place to prevent that from happening. We have this appeals process, where if you’re unhappy or uncertain about ruling, you have this option. And so I think it’s very similar in here the processes to prevent this from happening. Again, here’s what should have happened. Here’s where the error was so that you as the end user aware. So if there’s something that you can do

1:25:29 pm – Chris Wilson:
That appeals stuff is fascinating because I could imagine people taking it. Personally, If you’re a judge you give a ruling and the person who’s heated says No going to go above you basically. Especially many judges are playing or judging events where they know the players, right? Like it’s a local LG For example, you

1:25:42 pm – Jonah Kellman:

1:25:43 pm – Chris Wilson:
You know this is Bob your almost your friend who’s just you know gone behind your back and said I don’t believe you as a judge. I think you’re incompetent I’m going to go and talk to the head judge because I think you’re wrong and that does come up in the workplaces as

1:25:53 pm – Jonah Kellman:

1:25:55 pm – Chris Wilson:
well. Like for example We have a quality Like for example We have a quality the game, they’ll have feedback, they’ll give it to the various team members. So if a designer does something like changing well. Like for example we have a quality assurance department, they’ll play the game, they’ll have feedback, they’ll give it to the various team members. So if a designer does something like, changing the drop rate of an item or it’s properties or whatever quality assurance will give feedback and sometimes well. assurance department, they’ll play the game, they’ll have feedback, they’ll give it to the various team members. So if a designer does something like, changing the drop rate of an item or it’s properties or whatever. Quality assurance will give feedback and sometimes quality insurance will say, This is wrong. And the designer will say No, it isn’t and then they’ll close the issue and say, No, trust me. We’re just doing this right? I know you’ve said you don’t like it. Subjective, we’re going ahead with this for reasons and sometimes they are the one in the wrong. Sometimes, it’s qa who’s wrong, you know, the design that was right to put their foot down. I’ll say, we’re doing this and sometimes it was qa was right and the design and Stake and so QA feels a bit railroaded by that because what can they do? They can open the issue again which gets closed immediately. And you know it’s a because people are passionate about this kind of stuff.

1:26:39 pm – Jonah Kellman:

1:26:41 pm – Chris Wilson:
And we have tried very hard to recently. Give quality assurance more of an appeals process where I have a meeting with them every week, where Friday is a one o clock. I sit down with QA and I’m like, what’s happened in the last week? You disagree with? Are there? Any changes to the game? You actually think about that. The design team is telling you are good, because the design team of generally, right? You know, you possibly wrong as a QA person about that. And here’s the reasons why I make sure they understand. But likewise there are many times where there’s important nuggets of gold in there with a QA people who often have more experience with the actual game itself realize stuff. And then the designer is incorrect by shooting down that feedback and this creates situations where we have almost created an appeals process. That’s hopefully not having hurt feelings for anyone because the design is knowing, I’m gonna be asking QA, what have you disagreed with And here’s the reasons why I make sure they sure they understand. But likewise there are many times where there’s important nuggets of gold in there with a QA people who often have more experience with the actual game itself realize stuff. And then the designer is incorrect by shooting down that feedback and this creates situations where we have almost created an appeals process. That’s hopefully not having hurt feelings for anyone because the design is knowing, I’m gonna be asking QA, What have you disagreed with this week And here’s the reasons why I make sure they understand. But likewise there are many times where there’s important nuggets of gold in there with a QA people who often have more experience with the actual game itself realize stuff. And then the designer is incorrect by shooting down that feedback and this creates situations where we have almost created an appeals process. That’s hopefully not having hurt feelings for anyone because the design is knowing, I’m gonna be asking QA, what have you disagreed with this week And here’s the reasons why I make sure they understand. But likewise there are many times where there’s important nuggets of gold in there with a QA people who often have more experience with the actual game itself realize stuff. And then the designer is incorrect by shooting down that feedback and this creates situations where we have almost created an appeals process. That’s hopefully not having hurt feelings for anyone because the design is knowing, I’m gonna be asking QA, What have you disagreed with this week and have a chat And here’s the reasons why I make sure they understand. But likewise there are many times where there’s important nuggets of gold in there with a QA people who often have more experience with the actual game itself realize stuff. And then the designer is incorrect by shooting down that feedback and this creates situations where we have almost created an appeals process. That’s hopefully not having hurt feelings for anyone because the design is knowing, I’m gonna be asking you a, What have you disagreed with this week and have a chat about each of those topics? And so we found that helps Human Element. If it doesn’t feel like someone’s going to dub someone into me when they come and like snitch about something bad that happened, this is a case where I’m soliciting the snitching.

1:27:38 pm – Jonah Kellman:
Right. I mean, part of the appeals process that we have we understand that players will do anything in their To win. And I’ve had players say to me Judge, I’m pretty sure you’re right, but if you are, I’m going to lose. So I’m going to appeal and hope that you’re wrong. And at a certain point you have to be like Yes you have the right to get the head judges opinion. I’m gonna go get that head judge and have them answer it for you. But I’ve also seen head judges talk with the players, talk with the judge away from the table, usually stepping away from the players. And being like, Let’s have this separate conversation to discuss the ruling very often. The hedgit will say, I’m going to overturn this ruling and sometimes the floor judge who initially took the call says, I think you’re missing this, or this is why I ruled in this way, which the player didn’t mention and that’s and the head judge sometimes. Will listen to that. And so making sure that the person Who is appealed has a voice in the process? I think is very important, whether it

1:28:39 pm – Chris Wilson:

1:28:39 pm – Jonah Kellman:
be the judge or somebody from QA or that sort of thing.

1:28:43 pm – Chris Wilson:
I understand but as long as there is a final decision maker who can make

1:28:46 pm – Jonah Kellman:

1:28:47 pm – Chris Wilson:
the vision and move things along, as long as they’ve heard everything right, like it sucks. If they made the decision when they don’t realize that there was a lie being said here or something like

1:28:54 pm – Jonah Kellman:

1:28:55 pm – Chris Wilson:

1:28:56 pm – Jonah Kellman:
Now, we’re starting to get towards the end of this show. Now a few months ago on an earlier broadcast, I had a few guests and we talked about imposter syndrome. Is that something you’re familiar with? Is it a challenge that you face? And if so, how do you work with it?

1:29:11 pm – Chris Wilson:
I generally feel that I’m kind of running all day at work. Making decisions one after another and caught up in the moment and there isn’t much time for self-reflection. But occasionally I do feel the feeling of kind of waking up of like Wait, why is this me here? You know, like, It’s not too bad. It’s mostly just it’s rare that someone actually is an imposter, which is why it’s called impostor syntra. You know, it’s the feeling the incorrect feeling that it shouldn’t be you having this position. But you got there through hard work. These judges have achieved a lot in order to get to where they are. I’ve looked into becoming a level one judge before I looked in the materials. And I said, I am not good enough. I am magic, right? It’s a player. I make too many mistakes. This is a lot to study, I don’t have the time. I know the willpower to do this thing, right? It’s a thing. There’s too hard for me and these judges especially once you get through to higher levels, managed to do that and selflessly put in all of this time for their community and so they’re not imposters even if they feel that way. And so yeah, the community is a lot of respect for the judging stuff.

1:30:13 pm – Jonah Kellman:
That’s I think something that a lot of people need to hear that they’re there because of their hard work and I just want to thank you so much for joining us. I think that’s all we have. Unless you have anything else you’d like to add

1:30:27 pm – Chris Wilson:
I had a great time. Thanks so much and hope everyone continues to enjoy magic.

1:30:31 pm – Jonah Kellman:
Fantastic. And we’ll be back later this week with other first week broadcasts. Thank you.