Carmen Won't Talk to Me
Applied Research Project (M.Des Thesis)
2024
This project looks into the intersection between large language models and game-making, particularly how we may approach natural language interfaces and AI NPCs using language models in order to serve the expressive goals of the underrepresented game-maker.
The final result is the game-sketch Carmen Won’t Talk to Me, a game about connection, relationships, and queer, Filipino girlhood.
The making of this thesis evolved out of a two-fold response: encompassing my love for games with their ability to increase our capacity for care, and my anxieties and excitement around AI language models. In this project, I’ve created a game with language models, documented the process, and synthesized insights into best practices for underrepresented game-makers.
Quick Navigation ↓
Rationale
Underrepresented makers in particular are one of the most vulnerable groups who are impacted by generative AI. Firstly, because their labour is appropriated by how these models are trained, and secondly, the languages and cultures of people in the margins may go unrepresented in these models or how they are applied.
Artificial intelligence continues to take a significant hold on our lives, and I see this work as an intervention. This conversation is going to go with or without us, so as a maker I wanted to know how AI works and what we can do with it to better prepare ourselves in a world where AI tools are the norm, and share that knowledge with others.
What I seek out of this investigation are equitable outcomes, and to advocate for the democratization of not just access to AI tools, but also the knowledge of how to use them to serve the expressive goals of game-makers.
Process
In undertaking this work, I am employing a Research through Design methodology that uses an iterative process of connecting the conceptual ideas of the amazing scholars I unpack in my literature review, and connecting them with specific design examples, through prototyping and collaborating with the large language model.
I am also documenting my process in a journal, which I use to synthesize guidelines for future game-makers. Finally, I test the validity of this approach through conceptual playtesting, and answering the question: were my expressive goals indeed met?
The Story
In Carmen Won’t Talk to Me, you play as a girl from Manila, Philippines, reminiscing about her high school past in a conservative all-girls school. You were best friends with Carmen but now, years after graduation, your relationship has deteriorated. As you attempt to re-construct the past, remembering your conversation with Carmen and that argument five years ago between your friends Trish and Leah, you try to find where it all went wrong. This game is about losing a close friendship and being given the opportunity to repair that relationship.
The expressive goal of Carmen Won’t Talk to Me is to invite players to deepen their understanding of queer, Filipino girlhood, through a fictional story inspired by my own experiences.
First Prototype
How might we integrate a large language model (LLM) into a narrative-driven game?
For the first prototype, I was inspired by the work of previous scholars who sought to bridge a natural language interface with games, in particular Mateas and Stern’s work on Facade, and bridging their approach with the conceptual ideas of Tanenbaum and Tanenbaum’s commitment to meaning, and their paper on narrative-driven games and the importance of NPC (non-player character) design in delivering a pleasurable game experience.
Important considerations for a game with LLMs
For a player to take meaning out of their actions in a game, we must make them accountable for the things they do. Action equals consequence. If I press a button, my character swings a sword, and I defeat the big bad monster.
Similarly, in a game that allows players to use their own words to interface with it, there needs to be consequences for what players say.
On top of that, it’s important to design conversational agents, the NPCs, with character, give them a sense of lifelike-ness and your players will more easily believe them.
The language model must perform in four key areas. First, it should be able to understand the context and narrative setting to provide appropriate responses. Second, it should have a unique personality and background. Third, it should possess its own goals and motivations. Finally, it’s essential to have the ability to perform intent analysis so that it can react meaningfully to player input. In other words, the model must comprehend what the player is saying to produce accurate and relevant responses.
To achieve this, it’s necessary to communicate with the language model and emphasize the importance of these four areas.
Authorial intent
As I develop my game, I guide the behaviour of the LLM using prompts. With this technique, I can instruct the model to simulate the roles of two distinct NPCs, each with their own unique personalities and motivations. Furthermore, I place them in the same ‘physical’ space, which, in this case, is a high school classroom located in Manila.
I gave the model a script, with sample dialogue for each character, for each scenario. This is important because, as makers, we don’t want the model to run away from us and start talking about things unrelated to the situation. So, by giving the model a script, I can instruct it to deploy verbatim responses where appropriate. So in this way, we can enact authorial control over the narrative, while giving space for the model to improvise within the parameters you set it.
The three story beats
In practice, the prototype starts with the first story beat, which is a few rounds of dialogue between the player and the two characters, where the player gets a grasp on the situation by asking the AI NPCs (Trish and Leah) questions about their argument.
Then, the language model is prompted to enter the second story beat, which is to ask the player which side they’re on in an argument regardless of how the conversation is going.
The player responds, and that’s when the model conducts intent analysis, trying to determine if the player sides with Trish or Leah. The result of that analysis is sent as a message to Unity, which the game engine then ascribes one out of two outcomes. Either Carmen is hopeful, or she looks away, seemingly uncomfortable by how the conversation is going.
First Prototype - Evaluation
Through the first prototype, the language model was able to facilitate a three-way conversation, while retaining the personalities and motivations of each character, deploying dialogue when needed, and improvising as required. But most importantly, it is able to do intent analysis, communicating this analysis with the game engine, which allows me, as the designer, to ascribe certain outcomes as the result of this analysis. The game can now meaningfully respond to what players say to these characters.
Second Prototype
How can a game that uses LLMs meet my expressive goals?
Stating briefly, the expressive goal of my game is to deepen my players’ understanding of queer, Filipino girlhood, inspired by my own experiences.
The argument is a pivotal point in my game. I chose this situation because I was curious about how we might approach language models to handle a nuanced situation like this. There is no right or wrong answer. Or rather, what I think is right is different from what another person might think is right. So I really wanted to engage with this conflict of values, so that my player can experience what it might be like to be subjected to the complex workings of a system that seeks to oppress queer identity. To put my player into the life of a teenage Filipino girl who had to go through those things, facing forces larger than herself while navigating the complex relationships with her friends.
A natural language interface was crucial in exploring these themes because, as we know, language is subjective, it is charged with the held values, beliefs, and worldviews of its user.
I found that a game like this, that seeks to explore these themes using language, was made possible because of AI language models.
Environment, narrative, and the social effect
My second prototype addresses my expressive goals in three ways.
The first is through environment, this means the graphics that are used, how it emulates places in the Philippines, and how the characters look like Filipino girls.
The second is through narrative, and how this story attempts to be uniquely Filipino, by exposing how the schools’ religious ideologies have effects on the lived experience of queer Filipino girls.
And third, through the social effect of language. This means, what is the social impact of what we say to these characters, how it might agree or conflict with the held values of the characters, and how they might react as a result of this agreement or conflict.
Second Prototype - Evaluation
I embarked on developing this second major prototype to explore whether a game using LLMs could align with my expressive goals as a game-maker, and I was happy to discover that it could. My experience in building this prototype and reflecting on each step is relevant in addressing the central focus of this discourse, which is to examine how a game with LLMs could serve the expressive goals of underrepresented game-makers.
The next step was to conduct conceptual playtesting to determine if my expressive goals were achieved. As discussed earlier in this section, my game aims to invite players to deepen their understanding and appreciation of queer Filipino girlhood, drawing from experiences like my own. The game seeks to achieve this by integrating natural language interfaces and leveraging the unique affordances of LLMs, exposing players to the social effect of their utterances and how they may conflict with the values and worldviews of the AI characters, whose circumstances represent a common conflict in the Philippines that so often succeed at suppressing queer female identity.
After wrapping up my second prototype, the game was ready for conceptual playtesting. But first, I’d like to talk about the findings. What did I learn through this process of making this game and writing about it?
Best Practice Guidelines
One of the contributions of this research is the synthesis of this approach into best practice guidelines or suggestions for future game-makers so they may reproduce this approach into their existing game-making practice
The Lens of the Game-Maker + The Lens of the Player
Before delving into creating a game with expressive goals, it’s important to consider where the game-maker is coming from, and where your player is coming from. Just as how the identity of the game-maker influences what kind of games they create and what stories they share, just so that the player’s orientation influences how they receive the game-maker’s message. Basically, no matter what message you are trying to convey, your audience will not receive your message uniformly, and instead, what we can do as designers is craft an experience that fosters plausible and relevant interpretations of the work.
I saw this reflected in the result of the playtesting session, which we will unpack at the end of this section
Natural Language Interface — Collaboration between the LLM, the Game-Maker, and the Player
The next diagram illustrates how a natural language interface was implemented in my game. There are three components, the first is the AI’s ability to conduct intent analysis on player input. The second is to design in-game goals that the player meets and ascribe outcomes when these goals are met. These outcomes help to guide player behaviour, which can encourage them to do certain actions. For example, if the players know that characters react differently to certain things, they might be encouraged to, firstly, converse with the characters, and following that, perhaps take sides or take no sides at all in a conversation.
Facilitating Expressive Goals with the LLM
This next diagram looks at the relationship between using a natural language interface and connecting it with the three main conceptual ideas of this thesis and how it might be used to serve the expressive goals of the game-maker.
The first idea is the importance of keeping players accountable for the words they use in the game. This keeps player utterances meaningful, and rewards players for committing to the actions and desires of the player character.
The second idea is to encourage empathic emotion through the narrative situation and character identification. This is Keen’s theory of narrative empathy. Keen’s theory is used to guide writers and novelists, but in this case, I am connecting Keen’s ideas to the ability of players to be active participants in a narrative-driven game, by building upon existing game-making practices and techniques.
I go deeper into this in my document, but basically what it means is that the first-person perspective in games can be powerful in supporting empathic emotion because not only do we have access to the player character’s internal mind, travelling with the character and getting to know their thoughts, desires, and motives, but through controlling a character’s actions, we go one step further and become enactors of their will. By committing to the character’s will, we identify with the character, and we can speak with the character’s voice. It is not through character traits that we relate to them, but rather through their motives. By committing to their will, we are engaging in this transaction where the character’s desires and motivations also become our own.
And finally, the third idea is about the rules of engagement, and how these rules foster plausible and relevant interpretations in the player. What it basically means is that a character’s motive should be able to align with the actions that the player is allowed to do. This is where a translate the previous diagram on existing game-making practices and apply it to my game which uses language models. In the case of Carmen Won’t Talk to Me, the protagonist’s main motive is to reconnect with Carmen and mend their friendship. Therefore, the primary means of navigating the game involves communication, using your own words to navigate the story, and rewarding players for doing so. It is through the consequences of player action that players derive insight and meaning.
Keeping Players Accountable in a Game with LLMs: Environment, Narrative, Social Effect of Language
The final diagram unpacks the first conceptual idea, which is how we might keep players accountable in a game that uses language models. I touch upon this in my second prototype, but in short, I chose three areas where the game changes to reflect the result of players’ words.
The first is the environment. After the argument scene that, in story, happened five years ago, players can see the words they used imprinted on the journal in front of them in present day. Additionally, the plant that was once withering on the desk, which represented the player’s relationship with Carmen, is green and lively again.
The second consideration is narrative. Depending on the outcome of the argument, it changes Carmen’s attitude toward you and her perception of that event.
Then finally the social effect. Depending on what the player says, the AI characters rise to meet the player’s words, activating each character’s unique motivations and held values. There is a social impact to what the players say.
These diagrams serve as suggestions for future game-makers and demonstrate the approaches I use in my game, so that they might apply them to their own existing practice. The final step was to have others play the game and determine if the expressive goals were met.
Conceptual Playtesting
Carmen Won’t Talk to Me lasts about 15 minutes on average, and after playing the game I spoke to each of my four playtesters and asked them some questions and we had a chat about it. After encoding all four interviews, here are the themes that came up.
Novelty. Several players pointed out the contrast of how, in previous narrative driven games, the game might provide pre-set options for dialogue, so being able to use their own words and have characters respond differently depending on what you say was new.
The Players as Social Beings. The game prompted the players to think about what they’re going to say before they say it. Several players felt that they were a part of the discussion in the argument scene and that the interface encouraged them to conjure up their own thoughts instead of responding mindlessly.
Perception of Characters. The players appreciated how the game incorporated Filipino slang, which added to character believability. Additionally, they found that the first-person perspective made it easy to relate to the protagonist. Playtester D in particular enjoyed being able to adopt the persona of the protagonist and appreciating their malleability, and they felt a sense of control that even though Carmen won’t talk to the protagonist, there is an opportunity to make Carmen come back to them. These findings in particular were very rich as they related to how the narrative situation of the game helps promote character identification with the protagonist and allows for the adoption of the protagonist’s feelings and motives.
The Flexibility of Values. The players acknowledged how values can change in the moment, due to factors such as social pressure. For example, how one might capitulate under social pressure to align with the views of the group, even if maybe you don’t hold the same beliefs.
Common Vulnerabilities and Hopes. What came up conversationally amongst players was how they related it to their own personal experiences. For playtester A, who was the only one of the four to say that the girls should break up, they related their own experience of being in a religious school in their home country, and how the students had to deal with the school meddling in the students’ personal affairs. For playtester B, it was about societal pressure, and how in their experience they felt only academic pressure, and felt sorry that the characters had to deal with other things outside of that. For playtester C, they focused less on the school environment and more about girlhood relationships, and they recalled a memory of a falling out with a friend. And finally for playtester D, they recalled moments in their life where they desired to stand up for others facing discrimination.
What was particularly intriguing about the conceptual playtesting was the correlation between how players related to the game and its story, and their interpretation of its themes. For instance, Playtester B, who experienced primarily academic pressure from school, interpreted the story’s theme as the conflict between young people and systems of authority. In contrast, Playtester C, who recalled a personal falling out with a queer friend, saw the theme as being about girlhood friendships and reconciliation. Additionally, there was a consistent theme among playtesters of feeling involved in the conversation and the importance of speaking up against oppression or being held accountable for the words we say to our friends.
Conceptual Playtesting - Evaluation
These findings align with the conceptual ideas and frameworks explored throughout this thesis, particularly how the player’s ‘lens’ influences the affective capacity of their experience with the game and, consequently, their interpretation of it. Each playtester offered a unique interpretation colored by their memories and experiences, and as game-maker, I found all interpretations aligned with my expressive goals. These interpretations facilitated connections between the players and the characters of my game, revealing common vulnerabilities and hopes.
On a personal level, I am deeply grateful for the time my playtesters dedicated to my game, and their interpretations of the game touched me profoundly. Through this fictional narrative, inspired by my experiences as a queer, Filipino maker, I felt in a sense that I was understood.
In Conclusion
Thinking upon what it meant to me to have my game connect with players in this way, I want to end by emphasizing the driving motivators of this thesis. Which is, the importance of democratizing access to AI tools, and the knowledge on how to use them, so that underrepresented game-makers can continue to create stories that matter to them, and they can reach their fullest potential in this new era of working and living alongside artificial intelligence.
Works referenced:
- Keen, S. (2006). “A Theory of Narrative Empathy.” Narrative, 14(3), 207–236.
- Mateas, M., & Stern, A. (2003). “Façade: An Experiment in Building a Fully-Realized Interactive Drama.”
- Tanenbaum, K., & Tanenbaum, T. J. (2009). “Commitment to Meaning: A Reframing of Agency in Games.” UC Irvine: Digital Arts and Culture 2009.
- Tanenbaum, T.J. (2011). “Being in the Story: Readerly Pleasure, Acting Theory, and Performing a Role.” In M. Si, D. Thue, E. André, J.C. Lester, T.J. Tanenbaum, & V. Zammitto (Eds.), Interactive Storytelling. ICIDS 2011 (Lecture Notes in Computer Science, vol 7069). Springer.