All right, we’re gonna get started. So we are here to talk about
some of our learnings and techniques that we have for reducing
biases in the user research process.>>All right, so we are part
of the Xbox Research Group. We’re both currently
working on Minecraft. My name is Melissa. My academic background’s in
social health psychology. Before I came to Xbox, I did a lot
of research on diversity inclusion across race, gender, and
sexual identity in public health. I joined team Xbox in 2015. Since then I have worked in
the Halo and Minecraft franchises. And I’ve also done some cross game
research on people’s experiences of social and multiplayer gaming. And I’m currently a member of
Team Xbox LGBTQ, Blacks Xbox, and the Women of Team Xbox. And I do a lot of work within
Xbox Gaming for Everyone.>>So my name is Jerome. So my academic background is in
social and cognitive psychology. But at this point, my much longer background is working
on Xbox First Party franchises. Which I think I’ve worked on almost
every one at this point, but most recently that is Halo and
now Minecraft. And then I also lead
Team XBox LGBTQ. So just to give you a brief roadmap
of the talk that we’re gonna be doing today. First we’re gonna be talking about
the positive and negative biases that all humans have and how those
biases affect game development. Then we’re gonna talk about
combining games, users, and research into games
user research and how that can help reduce
game development bias. But those same biases that
affect game developers can affect researchers. And then last, we’re gonna talk
about work we’ve been doing in our groups to reduce those biases. Reducing or completely eliminating bias is
a big challenge for everybody. So this is very much
a work in progress for us. But this is something that we
wanted to share where we are. And hopefully, we’ll be able to
learn from all of you after you hear what we’ve been doing. So first,
we’re gonna talk about bias and how it shows up in game development. Since many of us here
are researchers, I’m sure you’re familiar with many of the biases
that we’re gonna talk about. But this is something that we
wanted to share in addition, for your knowledge, because we’ve
found this framing of biases to be a really helpful thing when
talking to game developers. To help them understand how their
own biases come into play when they think about their players. So going back to basics here. So this sentence here or partial
sentence is actually a sentence that we wrote in our group over ten years
ago as a description of what we do. All about collecting unbiased data,
but I really wanted to dig
in here in this talk, about what it means to be unbiased,
and to collect unbiased data. So just to define bias here. So bias is a preconceived
positive or negative opinion, and something that doesn’t
match reality. So thankfully in the games industry, most people recognize
bias is a bad thing. They want to help reduce
their biases, but biases can show up in less
obvious ways as well. A lot of people have
good intentions, but they unintentionally exclude players
or disadvantage them in some way. And especially players who
have different play styles or experiences from their own. So this is something all humans
have this unintended biases and they skew our judgement. And they’re really all automatic for all of us just because of
the way that our brains work.>>So as you know, these biases
happen because there’s so much for our brains to consider at all times. That in order to deal with it all,
our brains use heuristics, or these short, simple, efficient rules that help us
make decisions unconsciously. So we’re not brain scientists, so
shout out to all of you who are and apologies in advance for
oversimplifying the brain a bit. But we know that there are a lot of
different neurological systems that work together and
contribute to bias. We’re just gonna use one as a simple example that you can
use to explain how that works. So here you have
the fusiform face area. It’s an area of the brain that
helps us recognize faces and distinguish them from one another. You learn to quickly recognize
the faces of your in group who are people that you’re consistently
exposed to as you grow up in your own environment. However, you don’t get really good
at distinguishing faces of out groups, so people that you’re not
exposed to on a consistent basis. Neuroscience research
involving brain scanning and analysis has shown us that
we can’t recognize faces of our groups members as easily. And we can’t distinguish
between them as quickly. Since humans are often automatically
less comfortable with things that are unfamiliar to us,
this can trigger some unpleasant, but unconscious reactions
to members of outgroups. Reactions that you don’t
necessarily wanna have or even know that you’re having since
it happens on an unconscious level.>>All right, so what we’re really
talking about here in unconscious or implicit bias. So I was a research assistant in
school in a lab where an unconscious bias measurement tool was developed
called the implicit association test that’s shown here. So how many people here have tried
the implicit association test for themselves? All right, great. So for anybody who hasn’t, this is
a test that measures what your brain does automatically,
based on reaction times. So it’s all about whether
it’s more automatic for you to associate groups or concepts
with positive or negative ideas. So this goes back to those
automatic mental shortcuts. It’s easier for our brains to jump
to some conclusions than others. And it’s true, even if we’d
like not to have these biases. The researchers behind this test,
Benagy and Greenwald, they referred to this as
the hidden biases of good people. And I think that’s a pretty
good way to sum it up. So this extends definitely
into game development and the biases that game
developers have about players. I’m sure if you think about game
developers that you know, some of them probably have preconceptions
about categories of players. Such as, highly competitive
multiplayer players, people who stopped playing our game, or people who comment on
our YouTube videos [LAUGH]. They and maybe you, have the picture in your head
of what those people look like.>>So when you’re dealing with
automatic mental processes and hidden biases, it’s hard to counteract even if
people have the best of intentions. We can make sure that
we’re not excluding or disadvantaging players based on
any number of factors including identities, backgrounds,
skill sets, and preferences. Luckily, it’s not
actually impossible or harder than world peace. We all in this room know how much
games user research can help with incorrect assumptions and
beliefs about players. But biases can get baked into
the development process in ways that are not solved solely by introducing
some user research into that. So we wanted to point you to
resources to combat that. And some of it has to do with
the approach that the whole development team takes.>>All right, so one thing when
putting together a development team, it’s really important to think
about the unconscious biases there are that affect team composition and
how the team communicates. So Microsoft actually has created
some unconscious bias training that’s 100% free and
available for anybody to take. And we’ll have a link to it at
the end, but if you haven’t seen it, it’s a really good start for you and
others you work with to start thinking about how to
address unconscious bias. And it covers some topics including
how great a team diversity can lead to more creativity and productivity. And also ways to avoid team member
exclusion due to communication style, background or team culture
as well as many other things. So another thing to think about
when your team is getting started, is the approach to customers. So for anybody who was here for Tom
and Lauren’s talk just before lunch, this is the inclusive
design approach. If you weren’t, what the inclusive
design approach is really all about is focusing from the start on
how to better include players. These may be players
who are unintentionally excluded in the past. And that diverse team member
perspective that I was talking about in composing your team is something
that really enhances that inclusive design approach in terms of bringing
more perspectives to the process. This also,
there is a toolkit for it. And we have a link at the end. And so now,
you’ve got your team together. They’re trying to take
an inclusive approach. How else can bias show up in
the game development process?>>So here’s where we get back
to the unbiased data that we started with. Game designers and other developers
very much wanna make a great game experience for their players. They have their own ideas about who
those players are, how they play, and how to understand them. As games user researchers, I’m sure you’re all very familiar
with the biases that development team members can bring
to player understanding. But as we talked about, this is
a framework that we found is really useful for explaining psychological
principles behind bias. And establishing some common
language with development team members who may not be as
familiar with research. So first is the false
consensus bias, which leads us all to believe that our
perspectives are more typical and widely shared than they are. The false consensus bias makes
us assume that other people think just like us or at least more
like us than they actually do. In game development, it can often
make us think that other people play and enjoy games exactly
the same ways that we do. Of course, the truth is that game
developers are not just like everyone else. Months or
years of working on a game or just game development in general,
means that developers are already coming from a very different
perspective than most players. And even if they realize that
their perspective is different, the curse of knowledge bias
gets in the way which makes it impossible to break
outside of that perspective. A lot of research has shown that
people are unable to ignore information that they hold, even when it’s your
best interest to do so. Being an expert makes it really
difficult to place yourself in the mindset of someone who doesn’t
have the private additional knowledge that you already have. You’re cursed with that knowledge. And being an expert in game
development actively gets in the way of player understanding. The false consensus and curse of
knowledge biases combine to lead developers to form misconceptions
about real players. And they can exclude or disadvantage those who don’t
match those misconceptions.>>All right, so if you’re a game
developer you can’t rely entirely on your own perspective. And you need to get feedback and try to understand other
people’s prospectives. So here we get into another bias,
which is that humans assume that feedback that reaches us most
easily is the most popular and important feedback. And we assume that other feedback
is less popular and important. Which is the availability heuristic. So in game development,
it’s very, very easy for people to pay the most attention
to the loudest voices out there or the voices that are most
available to them. That can be anywhere
that loud voices show up. It can be online on forums,
social media, Reddit, anywhere that has a small
proportion of loud voices. And those voices really should be
listened to in proportion with the rest of the audience. So, that vocal minority of players
whose opinions are most available to developers they often differ from
quieter players in systematic ways. And the game gets customized them
in ways that leave those quieter players out. So I’m gonna talk about one example
here of how skewed those kind of opinions can be from some research
we’ve done in Xbox research. So during Halo 5 development,
the team heard a lot of feedback. They had put out a beta,
it was something, they had included some new features,
brought back some old ones. And they heard lots of community
feedback about that beta. And a lot of the feedback
centered around whether players should be able to
sprint in the game or not. And based on online conversations,
the team came to us and said hey, this really seems like
it’s a 50/50 split. About 50% of players we hear
from really love sprint, want it in the game. About 50% say no way, this
shouldn’t be in the game at all. So we ended up doing some
research with a larger and broader set of players than
who they were hearing from. And we found out that it
was only 13% of engaged players who were opposed to sprint,
and 87% of their engaged players
were totally fine with it. And this is something that we had
done some earlier research to understand more about
how these groups defer. And we’d ask people about how
much do you participate online in discussions about Halo? And kinda unsurprisingly
based on these results, the anti-sprint people were
much more vocal online, so that’s why the developers were
hearing from them much more often.>>And of course, another danger in
trying to gather feedback in any kind of group or public setting
is social influence bias. Hearing other people’s opinions can
actually lead people to change what they say themselves. Or to simply not say anything at all
if they disagree with the original commenters’ thoughts. This research actually
started in in-person groups. But newer research has focused
specifically on online environments and it’s replicated those results. There have been some randomized
experiments done actually, an article that was
published in Science. That involved evaluating and
rating things on social media, that showed that if you plant some false
positive reviews of a product, you inflate subsequent ratings of other
people online who have read those. And doing the opposite,
planting false negative reviews, does the same thing. Some people do love to argue,
but a lot of people don’t. And dissenting opinions often get
silenced in a group experiments. So you may have heard about some
research into how marginalized people’s voices aren’t heard as much
in the classroom, and workplace, and other social settings. And the same thing applies to
other social environments as well. And there’s one last bias that
get in the way when people try to gather feedback, confirmation bias. We pay more attention to
information which lines up with and confirms our previous beliefs. And we pay less attention to
information that contradicts those beliefs.>>All right, so we just talked
about a whole lot of biases. So thankfully,
game development teams like the ones that many of you
work for, or hope to work for, have games user researchers to
help them reduce these biases. And through what we do as our job, we can help get them that unbiased
data we were talking about and a less-biased view of
the player experience. Unfortunately, exactly the same
biases that we were talking about can influence the user research
process, but in different ways. So as researchers, we can apply
research methods to try to get representative feedback from
across the gaming audience. And this helps combat
the availability heuristic by making more viewpoints available
to the development team. So as researchers we can
apply research methods to get that representative
feedback across audiences. And that combats the availability
heuristic by making more viewpoints more available to
the development team. But what’s depicted here is often
not the reality of doing research, we’re not paying equal attention
to everyone in our audience. Sometimes we have very good reasons
for focusing on specific parts of the gaming audience, but we should
be very conscious of our choices and who they include and exclude. So for example in availability, let’s first talk about
reaching participants. So who here has done research that
relies on people signing up in some way to get feedback, such as being
a fan of a franchise, early access, a beta? What about research
with people who play a game within a certain duration of
a time, like a specific week or day? Research in which people my have to
self-identify as a gamer in order to find out about
the research in general, like going on a gaming news site or
gamer social media? There may well be players in that
audience that we’re not reaching or who would never have the opportunity
to take part in research. If they’re only
recruited in these ways, because they don’t show
up in those places. And we may be reaching some kinds of
players who want to take part but they’re unable to or reluctant
to for a variety of reasons. Maybe there’s research that requires
people to take multiple hours out of their day at a time where people
with jobs or who are in school, who have childcare needs, actually
can’t make it to the research. Research that relies on people
getting to a specific location, or research that people
just don’t think is for them, because they’re not enough
of a gamer to participate. Volunteering, specific time,
and also during research. Even for players and
audience members that we do reach, often as researchers we’re filtering
them out of participation. So some of the limits that
we have are in place for good reasons like legal restriction. Sometimes they’re there for
less good reasons, like too narrow definitions
of our gaming audience. So for example, who here has
had an age limit for a study? A minimum or maximum age for
participants? There might be some players in your
audience that are actually outside of those age limits
that have been set. Also, what about research that
focuses solely on current players of a game or franchise,
or beta participants? What are you potentially missing
about those who maybe have stopped playing the game, or
who are in the audience, but not playing the game for whatever
reason, can dig into some of those? Participants can also get excluded
by improper use of proxy variables, to define if someone is in
the audience for a game. So for some reason, rather than
asking people directly whether or not they play certain games, or are on a specific gaming platform,
some researchers have used proxy variables that exclude
parts of the audience. And all real proxy variables that
we’ve seen in gaming research are how long their
gaming sessions are. How many hours someone
plays per week. Whether or not someone
defines themself as a gamer. Their employment status, or income. And the amount of money that they
spend on games in a specific duration of time. And by the way, this book,
Weapons of Math Destruction, by Cathy O’Neill actually has
a couple chapters with great discussions of proxy variables and how they can sort of harm
data collection analysis.>>All right, so going through that
list of availability biases that can show up in research, it may not
have seemed completely fair. For an individual study, there are
often very good reasons why you want to restrict participation because
of the focus of the study. But unfortunately, doing that
thoughtlessly can mean that your collection of research can have
a cumulative effect, where you’re hearing much more from certain
kinds of players than other ones. And the access that you have to
players’ voices can look a lot like that availability biased influenced
access that we talked about for game developers. That there are voices that are
reaching you much more often than others, and that’s exactly what as
researchers we’re trying to combat. So we’re gonna move on now into
talking about our toolkit for addressing this. So the reality of research is for
anybody no matter the size and resources that you have, is that
there are always limited budgets, there are always limited time,
there’s always limited resources. And some people are literally
just more expensive to reach with research. But we can be much more deliberate
in our research choices to hear from more people in
the games audience and hear from types of players
we’re not hearing from already. And often we can help reduce bias by
refocusing the resources we already have without even spending
additional resources. So some of the approaches
we’re taking. First is understanding
who you’re missing. So the first thing you can do is
measure player behavior patterns, often through telemetry and compare
that to your research participants. You can look at who’s
well represented, who’s poorly represented in
your research participants. And some kind of obvious
differences that we’ve seen that are ones that should be
obvious to all of us. But it’s something that can really
affect things when you look over all of your research, is that players who play less are
less likely to respond to research. And players who play
less are also more likely to have negative
opinions of your product. So you’re probably hearing fewer
of those negative opinions than you would if you were reaching
across the entire audience. So second, you can look at what
types of players you’re missing and do audience research or consult
other audience research to better understand the whole world
of players out there. So it may be industry
research out there on what the gaming audience
looks like as a whole. It could be players on the platform
that you’re aiming for for the title or players for similar games and
just really look at who’s missing. So once you have that understanding, you can move on to ways to expand
your research participation. So first, you can expand where you
look for and recruit participants. Redo some specific work on looking
at where we have those gaps and targeting both physical places and online places where people tend
to gather where that have a high concentration of people we’re
not hearing from otherwise. Also, timelines it’s unavoidable sometimes we need to do research
that requires a big time commitment from people to be able to deeply
understand their experiences. But that doesn’t mean that
needs to be all your research. It’s something that you can really
fill in the gaps with shorter research, and research that fills
all different time availabilities. So that you are hearing
from more people across their own availability. Third, focus on being
where the players are. So this can be in a virtual sense,
looking, doing research in game,
doing research online, doing research remotely where you’re
contacting people in their homes. But you can also target research for
unique play environments. So for example,
we both work on Minecraft. Minecraft is something
that’s used in schools. And there’s certain kinds of
learnings you can only get from seeing how it’s actually used
in the classroom environment. Another great example, actually,
is one of our coworkers, Melissa Dewolf, recently did some
research on children’s hospitals. The games are used a lot in
children’s hospitals for recovery, but they’re used in different
ways over different time periods with different restrictions due to
the treatments that are going on, That there are really ways that we
can help make those games better in those environments in ways that
you may not be thinking about. So first you’ve tried to understand, then you’ve tried to
expand participation. Third, you really wanna focus on
spending your research time where the gaps are the biggest
problems between there’s well represented players and
under-represented players. Focusing on their opinions,
their behaviors, their experiences. And then spend your resources
accordingly to help understand those differences and get those
unique learnings from those people you may not be hearing
from otherwise.>>So as researchers, we can cut
down on social influence bias through methods that use
individuals, not groups. In social science,
we’ve known since 1939 that group discussions change
the opinions that people report. And as we already mentioned,
experimental research done online shows that that’s
true about digital or Internet based situations as well. There’s a lot of research
to indicate that people in groups converge towards one opinion. And that opinion tends to be more
extreme than one that they would hold if they were answering
questions on their own. So for example I hate Sprint. It’ll ruin the game, versus Sprint
may be okay, but it’s not for me. Thankfully, individual based
research does tend to be the rule in most game user research. However, there are a few
things to consider. First, use group settings like focus
groups for what they’re best for which is idea generation where you
get the value of people bouncing ideas off of each other. But don’t use it for opinion
measurement where it’s impossible to separate out the effect
of the loudest voices and the opinion change. Another thing that we can do is
adjust analysis where social influence is unavoidable. For example, if we’re doing research
on multiplayer gaming sessions, the other players will
impact the experience of each individual in the session. Thus, the individuals become
non-independent units of analysis. And some of the assumptions of our classic analysis techniques
are actually violated there. In these cases, we really should
treat the multi-player group as the unit of analysis,
as a cluster, not the individual. However, we know that that takes
a lot of analytical power and a lot of people. So an alternative is to manage
multi-player match making in our gaming session to try to
introduce as much independence into the experience as possible. You can place or randomize players
into different multiplayer groups throughout the session to try
introduce that independence. For research topics that are heavily
discussed within a gaming community, it might take a little bit of
angling to break through and go past the group consensus. For example,
again from the Halo Sprint research, we didn’t just ask Sprint,
thumbs up, thumbs down. We dug into how players felt about
the Sprint in the game as a whole, what they thought about Sprint for
other people. What they about it for themselves. And what effect they perceived
that it would have on the game as a whole. We also asked some questions that on
face didn’t necessarily seem related to Sprint, but were important for
understanding the issue. For example, we asked players
how important Sprint and mobility was to them. And this helped us make the Sprint
experience better for everyone.>>So those are all general
techniques we’ve used to reduce research bias. To get us closer to being able to
see everyone in the audience and reach them with research. But we found, even after using some of those
general techniques, there are still certain kinds of opinions that
were disproportionally excluded. There were definitely certain
kinds of players that were disproportionally excluded. And we really needed
more targeted techniques. So the reason for this is that
things like play styles or preferences or experiences are not evenly
distributed across the population. I’m gonna talk a little bit about
some research that was done by a researcher named Dr. Margaret Burnett at Oregon State
University called gender mag. But it was looking at individual
differences in software problem solving and
how that clusters by gender. So looking at here is how much
people tend to prefer to tinker and experiment to figure things out with
software is something that the blue bars are males,
the light pink bars are females. And I think the important thing here
is that those bars don’t match up. There’s a different distribution. And if you have a game that’s
relying a lot on people tinkering, experimenting to figure things out, you’re probably disproportionately
excluding females. Couple other examples, we’ve talked
a couple of times about the Sprint research that was something that there was a big difference between
those were vocal online and not. And there are lots of
other examples out there. For example is gonna be talking
later today, presented last year about competitiveness and how that
interacts with age and gender. And how that can disproportionally
exclude people who aren’t this competitive.>>So those uneven distributions
sometimes do cluster by demographic categories such as gender,
race, ethnicity. And to understand
the disproportionate effects, you need to measure how the players
in your audience identity. So here are some things to
consider when you’re asking players about their identities. First make sure that your
response options make people feel represented, and not excluded. Always have an open-ended
response option. And periodically analyze those
responses to the open-ended question to determine whether you need
to add additional categories, or change these as
the ones that you have. So for example, on some current
research that we’ve done, we’ve analyzed some open-ended
response categories. And notice that we needed to
add an additional category for Middle Eastern or
North African identified folks to capture those people who
identified that way there. It’s good to get feedback
on those questions for other people on your team, and
that’s why it’s always good to have a diverse team as well,
to give that kind of feedback there. Also, always allow
people to opt out. Have a prefer not to
answer category so that people can choose not to answer
the question if they don’t want to. Consider also that how
people identify is going to vary by region and location. So if you’re conducting some
international research, you might need an international
version of this question. And additionally, some of the questions that you ask
could pose some legal complications. So for example, some questions
about health conditions or disability status may be interpreted
as collecting medical information, even if you’re just asking people
whether or not they have one. And also in some regions asking people about things like their
sexual orientation and gender expression can actually expose them
to legal risk in their countries. So make sure you talk to your legal
department when you’re putting these questions together. The researchers need to be careful
about another effect of asking players about their identity,
which is stereotype threat. Stereotype threat is a predicament
in which people feel at risk of confirming negative stereotypes
about some group that they belong to. People often get so
stressed out trying to disprove stereotypes That the stress
unintentionally and negatively drives down
their performance. For example, if you’re a woman and
you’ve heard of a stereotype that women are worse at competitive
multiplayer games, that can actually affect how you perform at
competitive multiplayer games. It doesn’t actually
matter whether or not you think
the stereotype is true, your performance is still affected
just if you know that it exists. One thing that can make the effects
of stereotype threat worse, is making someone think about
their group membership. So recently researchers have
been investigating this directly in gaming. And it still holds even
within a gaming environment. Women who felt more threatened at
the beginning of a gaming session, completed fewer levels and performed
more poorly than women who did not. And the presence of male
opponents exacerbated that poor performance for
the threatened women, but not for
the women in the control position. So from a research perspective, there’s an easy way
to avoid this trap. Don’t ask people identity or
demographic-related questions before they play, or
at the beginning of a session. Ask about them at the end
of a play session. Another potential factor is
the makeup of gaming sessions. Research in many domains has
shown that threat decreases as representation and
the stereotype domain increases. So this is implications for any groups settings such as
play test and focus groups. We need to pay attention to the make
up of our group gaming sessions and try to make them more diverse
in an effort to reduce threat. So when you wanna talk to players
about gaming experiences related to their identities, there are several
other things to consider. One, start with less
sensitive topics and build some rapport by
establishing common ground. This gives your players
the comfort and the confidence to want to talk
about those sensitive topics. And this is another good reason to
have some diverse team as well. Anticipate your players’ needs and
adjust appropriately. So one of the things that we found out when we’ve done research
is that, some but not all, deaf and hard of hearing players
communicate through sign language. Some do lip reading, and some are bothered by the presence
of an interpreter at all. So make sure that you ask
up front what kind of accommodations people need,
and don’t assume. Another example of that is
Microsoft’s autism hiring program, which we’ll link to you at the end. There are several adjustments that
we’ve made to our hiring process to make it more predictable and
to reduce unnecessary pressure. So we can apply some of those same
principles to our gaming sessions, trying to keep
the process predictable. And reducing pressure can help those
with similar challenges adapt. Also be aware of past
research on identities and how it affects how people might
respond within a session. So for example, women and people of color tend to have lower
self efficacy about their skills and abilities surrounding
technology usage. So different ways of
advertising a task or a session might influence people’s
decisions to participate. For example, if you put an ad
up that says, we’re looking for skilled, avid, or
experienced gamers. That might exclude some people
who don’t necessarily see themselves that way and might
disproportionately exclude women and people of color. They’re also a little
bit more likely to be self deprecating about their
skills and abilities in a session. So some participants may engage in,
what we call, discounting behavior with statements
like, I’m not a real gamer. Or maybe I don’t know this cuz I don’t really play
this game that much. Their statements might not actually
be reflective of their actual experiences, they’re just
trying to discount so they don’t feel as
bad in the session. So we need to take this into
account in our analysis and discussion of findings, especially
when presenting them to developers. One thing that we can do to combat
this is to remind participants that we’re testing the product and
not them. And also communicating that people
from diverse groups have found success in the task. And that might help participants
focus more on the task and not specifically on their identities
and the stereotypes thereof. So research gets even more
challenging when you wanna try to understand the negative gaming
experiences that players might have. Especially if they’ve been targeted
based on their identities, like their gender, race,
ethnicity, sexual orientation, or gender expression. So we’ve done some research on
online harassment and games. And we’ve found that a lot of
people who are targeted for their identities
actually don’t report or talk about their bad experiences. Even when we directly
ask them about them. This very likely means that
we’re actually underestimating the severity and
the frequency of those events. We’ve discovered some reasons for this underreporting that can help
other researchers when they’re trying to understand
bad experiences. So players feel that harassment is
inevitable and so wide spread that it’s actually become baked
into the player culture. So in order to combat this, a lot of our players have told us
that they feel like they need to develop this thick skin to
protect them from those effects. And pretend like they’re unbothered
or act like they don’t have any negative emotional
reaction to harassment. In their eyes, being defensive
about harassment only increases the opportunity for
them to get targeted. So what this means is that, our
players set a really high bar for the kinds of gaming
incidents that they report. Both to enforcement services,
but also to us as researchers. Players often dismissed racial slurs
and casual sexism as not a big deal. But just an unfortunate part
of gaming that it was their responsibility to counteract
with their thick skin. So that’s important to remember when
we’re trying to collect information about these experiences. Interestingly, online harassment
also affects people’s categorization of games as well. Players are more likely to exclude
games from categories if their early negative experiences
with games in that genre conflict with later positive
experiences with other games. For example, a lot of people
in our research had negative early experience with shooter games. They didn’t consider
themselves shooter players. But then they later had positive
experiences with a few games like The Division and Overwatch,
which they mentioned specifically. And they didn’t consider those
games to be real shooters. And they had all kinds
of explanations and reasons why they thought that
those games weren’t real shooters. And they were less likely to think
of themselves as shooter players, even if they spent hours and
hours playing these games. So if we ask players like this about
their time spent playing specific genres, we might get answers that
are not really reflective of their experiences. And that can actually have a
disproportionate effect from players of minority groups, including women,
people of color, and people from LGBTQ identities who are actually
more likely to get harassed online. So these are tough challenges to
fully overcome in our research, but we have a couple of recommendations. First, ask people about experiences
that they have seen negatively affecting others. It’s often easier for people to talk about events that
weren’t as personally painful. And a lot of our participants
actually talked about wanting to protect others. Maybe others who haven’t
developed that thick skin yet. It was a motivation for
them to report. And so they were much more willing
to talk about incidents that they had seen happen to others. Another is to be careful about
the language that you use when you are asking these questions. So for example, we found that
using words like harassment or targeted were actually too loaded. Often players quickly
mentioned that no, I don’t feel that way or they even
felt confused about why we were asking the question
in the first place. So using slightly more neutral
phrasing like negative experiences or even just asking players if
they had ever reported someone for bad behavior. Elicited more responses, and was
actually more likely to capture the behavior that our players consider
to be lower level incidents. Also, asking from multiple
angles is a good recommendation. So simply asking players to walk
us through what a typical gaming session looked like for them. Or asking them what were
some good and not so good things about online multiplayer
gaming actually got a lot of responses about harassment toxicity. Even though we weren’t directly
asking about that in those questions. And finally we should be specific
about what we want to know and what we’re asking players. For example, if we’re asking players
about experiences in certain genres we might actually
wanna use example games when we’re talking about those genres. Either instead of or in addition to actually naming
the genre in the question. And that can ensure that
we’re capturing the kind of information and
the people that we actually want. However, there is a danger of
going too far in research and focusing solely on
player identities. You can end up over categorizing or stereotyping players which washes
out individual differences and intersections of experiences
that exist in all groups. So make sure that you’re treating
player identities and proportion with all of your other research
inputs when you’re doing analysis.>>All right, so after doing all
that kind of work to help reduce bias in the user research process. It’s really important that that work actually is integrated into
the machinery of making a game. If the user research
insights are not integrated, you’re not actually reducing
the game development bias. And otherwise,
if it’s not really deeply integrated into the decision making
process of a team, the biases we talked about
can really still run rampant. The false consensus bias, curse
of knowledge, confirmation bias, all of those can affect the way
the team thinks about players and makes their decisions. However, in addition to deeply
integrating into the research process, there’s one other important
thing for helping user research provide an unbiased perspective,
and that is independence. So it’s really important that
researchers stand apart from influence. So in our organization, there
are several ways that we do this. First, is that research is 100%
own everything about research. They always have final say
on research on results, on how things are messaged. And data interpretation is
never left to the developers. So it’s also really important
that researchers are not simply presenting data. We not only learn what’s going on, we get a deep understanding
about what will help. So it’s important that researchers
are always advocating for directions for improvement. Not becoming designers in terms
of having a specific way the game should be designed. But always pushing, what would help
players have a better experience? So next, it’s really important
that you’re predefining and defining success ahead of time,
so teams are accountable. So in our group,
we have ways of defining good and bad results in advance. One thing is that we have a lot
of different comparison data. So we’re able to compare
both to games in the genre, games over a longer history and
Teams are unable to discount certain kinds of results because we can say
that is actively worse than other games that they look
at as competitors. So one last one is
in our organization, researchers are never
managed by a game team. They always report up
outside of game teams. This is really important, so that there’s not the pressure
of pleasing your boss. I think when you’re
reporting to a game team, I think everybody can
try to be independent, but at the end of the day, you’re
trying to make your boss happy. If you’re outside of that
team that doesn’t have to be a consideration anymore.>>So related to this, here are some
two last biases that should keep us all on our toes and
working hard to improve things. One is the bias blind spot. As humans, we’re much better at
identifying other people’s biases than we are at identifying our own. So make sure that you always have
multiple researchers with different perspectives involved
to check one another.>>All right and here’s one very
last especially insidious one which is the moral credential bias. So let’s say,
you’ve done work to reduce bias and you’ve been recognized for
reducing that bias, you’ve been rewarded in some way, recognized
by the organization for it. Which is great, but sadly research
has shown that getting that recognition, getting that reward,
actually it makes it more likely you’re gonna make future
biased decisions. [LAUGH] And this goes back to those
automatic matter of shortcuts that we were talking about
at the beginning. Once you’ve been recognized and
somebody said, good job for reducing bias. There’s some part of our brain
that says, good that work’s done. I don’t need to do that anymore. [LAUGH] So that makes it really
critical kind of as with all the things we’ve
talked about today. That you’re really deeply building bias reduction techniques
into everything you do. If you’re just relying on people
to remember it or do it ad hoc, that’s something where those biases
can easily sneak their way back in. You have exactly the same problems
you’re trying to combat in the first place.>>All right, so
to recap what we discussed today. First we took a journey
through the positive and negative biases that we
all have as humans, and how those biases affect our thinking
and the judgments that we make. Second, we explored how even though
our job descriptions are to provide an unbiased perspective to
our game development teams, those same biases can creep into
the game’s user research process. And third, we talked about some
work that we’re doing to reduce those biases, and
maximize benefits for players. And we discuss the lessons that
are important for inducing bias for any where it can be found. So I hope you can take all
of this and do you use it. Thank you. And here’s a list of some
resources that you can refer to, and we’ll leave it up while
we answer some questions. [APPLAUSE]>>Any questions?>>Hi, so early on, you talked about
recruitment and how it could be biased recruiting from your
company website or social media. I was wondering if you had how or where you recruit your participants
for like this user tests.>>So, one of the things we’ve
tried to do is go to locations that are a little more diverse or have
representation from other folks. So, for example, we’ve targeted
conventions that are specifically to like or to Blerds, like Black Nerds,
that are specific to specific groups that we’re trying to reach out to
and increase our representation. And a lot of that
requires liasioning and looking out into the community and
seeing what’s going on. So keeping a finger on the pulse of
what’s going on and communities that we might not necessarily
have our hooks into as much. So that’s one of the techniques
that we’ve used there.>>So another one that we’ve used
is by building into, well so actually, on the XBox platform. As a whole people can opt in to
being contacted by Xbox, and that’s something that allows us to
reach out to specific people who may have never reached out to
us in the first place.>>Great talk, have you guys noticed from your
own kind of observations on this, that there’s like a hierarchy
of biases you start tackling? Or is it just kind of like
a multi-faceted universe, and any particular developer or other stakeholder you’re talking to,
may have one or many of these they are suffering
from, to a great degree?>>Yeah, I think the most often what
I see, and part of why we talked about so many examples of it
is the availability heuristic. I think lots of people think
I care a lot about customers. I’m getting feedback from customers,
and don’t really understand or think about all that feedback that they’re
not hearing about because they’re only looking in certain places or
hearing from certain customers. I think that’s often one that, especially if people are already
working with researchers, one that can most cue how will
they think about their players.>>I can also say social influence
is probably another one that we see a lot. A lot of our teams like to look
at the forums to get feedback from people and
they also like to bring people in, in groups to talk about opinions and
things like that. So that’s probably one of the other
biggest ones that we see. But a lot of the teams do have also
the multifaceted group as well. So there’s a couple different
angles that we address them from.>>I’ll go. So I know you talked about the under
representation of people who don’t play games so much, or
people who drop off. Those people, of course, are
historically really hard to access. You know we’ve tried some things and
we’ve had some success but it’s still a really hard challenge. Do you have any tips for
reaching those people and getting them to talk?>>One of the things that I do
is when I’m at conventions, or talking to people about recruiting,
I get a lot people who approach me and saying,
I don’t think I’m much of a gamer. And I ask them,
do you play mobile games? I usually start there do
you play games on PC, lots of games that people don’t
really think of as games. I’m like, you’re a gamer [LAUGH]. You should definitely
consider participating. I also talked to a lot people and reassured them we test people
that don’t play games, because we’re interested about bringing
new people into the audience. So a lot of it is our ad hoc work
at showing up at conventions, in places where people might
have come with a friend. We actually meet a lot of people at
gaming conventions who came with a friend and they say,
I’m not really a gamer. And they sort of spread that
throughout their networks that like, Xbox does test people
who don’t play games or they test people who maybe don’t
play the games that we traditionally think of as real games so to speak. And so part of that is really about
helping redefine what we think of as a gamer and spread the word that we’re actually
testing a wide variety of people.>>Yeah, I think another part of it
to is we often do broader research that isn’t specifically about,
hey did you play this game, or did you recently play this game? But we, a secondary goal of
that is including some things that may help us hit some of those
people who are former players. Just like, hey we’re just one last
player game in general, and you did? You did stop playing that game? We’d love to talk to
you more about that. I think often with a direct contact
asking about a specific game, people are like,
I don’t play that anymore. I don’t care. They’re not gonna respond to that,
but more general research about gaming you may
be able to better reach them. All right.>>Good, thank you.>>[APPLAUSE]