Over my career, my research has been driven by several interests: (1) developing new engaging experiences that can make meaningful contributions to users’ everyday lives and (2) understanding human behaviors as they interact within interactive artifacts that stimulate learning and problem solving, such as games, simulations, and interactive narratives. A central focus of my work is on developing new tools and methods that can push the boundary of research in understanding and augmenting human experiences towards making social impact in areas of education, training, and health. I take a multidisciplinary approach in tackling this area of focus integrating several disciplines and fields, including Games, Interactive Narrative, Psychology, Human Computer Interaction, Artificial Intelligence, Graphics, and Data Science.
Games, especially Serious Games, have, for many years, been used as applications for entertainment, education, training, crowdsourcing, as well as platforms for understanding human behavior, which creates opportunities for making a great societal impact. When I first started my research in this area, games as a field did not exist. I was very lucky to be surrounded by very supportive colleagues, advisors, pioneers, and mentors who have helped me in creating and developing my work in what was, at the time, a very new and slowly growing research area.
Within this research area, my work has targeted two broader themes. The first is focused on developing novel automated tools and techniques for authoring, adapting, and personalizing virtual environments, such as games, interactive narratives, and simulations. Given the potential impact of these virtual environments and their applications beyond entertainment, developing tools that can enhance their design and production processes will subsequently have a high societal impact. The second theme is focused on developing data-driven techniques and tools for player modeling. Data from games and interactive narratives provide a unique opportunity to expand our understanding of human engagement and interaction. Developing new data-driven methods for analysis of players’ behaviors can aid in this understanding. Further, such methods can help us develop adaptive interfaces and personalized applications and experiences, which can immensely improve the design of games and their applications.
My research work won several best paper awards and honorable mentions. I was named a HEVGA fellow in 2017 for my accomplishments to the field. Through this work I published 27 Journal Articles, and 112 peer-reviewed Conference Papers in leading conferences and journals in the areas of Games, AI, ML, and HCI, including WWW, CHI (Computer Human Interaction), FDG (Foundations of Digital Games), IVA (Intelligent Virtual Agents), RecSys (Recommender Systems), and AA (Autonomous Agents). Further, I published the first book on Game Analytics — a new and growing subfield that specifically targets the intersection between Games, Data Science, Artificial Intelligence, Machine Learning, and Visualization. The book was published in 2013 by Springer. I also published the first textbook on Game Data Science published by University of Oxford Press in 2022. I am currently working on another book on Serious Games to be published by MIT Press.
Below I describe some sample projects that I did throughout my 26 year-old career.
1. Semi-Automated Tools for Games and Interactive Narratives
1.1 Virtual Characters and Agent Models
My very first work in this area focused on developing believable characters through modeling and simulating the emotion process within a virtual character. To this end, I developed FLAME (Fuzzy Logic Adaptive Model of Emotion) [13], where fuzzy linguistic variables were used to represent emotions and a fuzzy rule-based system was used to simulate the decision making process for the character using emotions as part of its cognition. I used reinforcement learning as a base for the character to learn about its environment, and derive variables necessary for emotion simulation. This work received much citations and made some impact in the field.
More recently, in collaboration with Matthew Gray (Theatre) and Stacy Marcella (Computer Science and Psychology), we developed a new virtual character model that embeds warmth and competence as variables that can affect trust and rapport [14]. In this work, we integrated the virtual humans architecture — an open source platform to simulate virtual humans developed by USC. We worked with the theatre department to develop a behavioral model of warmth and competence using the Laban Movement Analysis framework — a theoretical framework used within Dance and Theatre. This process is shown in the figure below. This model was developed through expert labeling of video recorded rehearsals. The labels were transformed into rules that dictate gestural movements and parameters that synchronize motion with speech.
Modeling Warmth and Competence in Gestures and Non-verbal Behaviors, see also video
1.2. Lighting AI Systems and Tools
When developing a complete experience, designers usually consider many aspects of the environment, such as level design, lighting, and camera placements and movements. Such design elements have a great impact on the narrative delivered and subsequently on the user experience. While developing characters has received much attention in the field, research work on lighting design was scarce. To remedy that, I investigated the development of lighting design authoring tools.
2. Developing Data-Driven Techniques and Tools for Player Modeling
As I continued to develop AI systems and tools to enrich the players’ experience, I was surprised by how little we know about players’ engagement and experience. While there has been some research work investigating the player experience, the area was still very nascent. Further, the methods used for Human Computer Interaction at the time was not developed for this area of research. And even though we have been gathering much data on the user experience, we did not yet have a good set of methodologies or methods to help us capture or model such rich experiences. This shifted my focus towards the second theme, where I developed novel methods to understand and model the users’ experience within games. I will discuss some example work here.
2.1. Methods for Studying Pervasive Games
One of the challenges with current Game User Research approaches is ecological validity [3]. Recently researchers developed approaches to understand player engagement in ecologically valid or naturalistic settings, e.g., A/B testing is now widely adopted. A/B testing within games is the process where a team develops two different versions of a game and deploys both. The team then collects data on the two versions and uses analysis of this data to make decisions about the design. The use of game logs and metrics, such as hours of play are often used. However, these approaches don’t capture issues with the design in a more granular manner. In our work with the company IgnitePlay, we developed a novel methodology called Data-Driven Retrospective Interviewing (DDRI) [8].
DDRI is a mixed-methods approach where interviews are used to gather participants’ insights about their gameplay retrospectively, similar to video-based retrospective interviews but using visualized game logs instead of videos. Participants use the visualized gameplay summaries to retrospectively explain their experience with the game over the time of use (typically days of use in naturalistic settings). The researcher can annotate the graphs for further analysis and triangulation with interview data. This approach also allows the researcher to use game logs to develop interview questions.
DDRI is a mixed-methods approach where interviews are used to gather participants’ insights about their gameplay retrospectively, similar to video-based retrospective interviews but using visualized game logs instead of videos. Participants use the visualized gameplay summaries to retrospectively explain their experience with the game over the time of use (typically days of use in naturalistic settings). The researcher can annotate the graphs for further analysis and triangulation with interview data. This approach also allows the researcher to use game logs to develop interview questions.
Using DDRI, we were able to gather important information about the design of SpaPlay, a game developed by IgnitePlay to promote healthy eating and exercise. For example, we found that retention was affected by how the game was embedded into participants’ life styles. We also uncovered specific social game mechanics problems that are tied to specific contexts of play and participants’ habits. This led to several design recommendations for personalization of the SpaPlay game mechanics and content towards more social interaction and better integration with different life styles.
2.2. Methods for modeling Problem Solving Behaviors
For several years we have been developing various ways to model and visualize player data to help understand the player experience to support design [2, 7, 9]. Our most recent system developed in this area was Glyph [15]. Glyph was developed to understand the players’ decision making process and strategies as they interact in a virtual dynamic environment. We used this visualization system to uncover issues with progression in games, such as Wuzzit Trouble — a game developed to teach algebraic math to middle school students. A paper on Glyph won best paper award at FDG 2015.
Leveraging on our visualization work, we developed a new approach to model player behavior integrating visualization and machine learning allowing us to develop a human-in-the-loop approach to behavior modeling [1] (see the Figure below). This methodology has been used in multiple projects within our lab to understand and model players’ problem solving behaviors. These projects include a game developed to teach programming, called May’s Journey [4], developed in our lab, where the methodology was used to understand debugging behaviors [5]. We are currently using the learned patterns to develop an adaptive help system for the game. We also used the methodology to decipher wining strategies for Dota2 [1].
Human-In-The-Loop Approach to Modeling Player Behaviors
3. Serious Games
I helped design and develop several serious games.
The first game in this area is a game I worked as an advisor for in collaboration with an indie company called IgnitePlay. The game was called SpaPlay. The game was developed with the aim to help encourage players to adopt physical activity and health eating towards losing weight and behavior change.
I advised the development of an Alternate Reality Game called Daedulus developed by Extra Ludic Inc. as part of a DARPA project investigating ways to use data driven techniques to measure team performance and the effect of personality and individual differences on team performance.
I advised the development of an Alternate Reality Game called LUX, which was developed as a way to measure and understand emotion regulation, coping and resilience of first year students at UCSC campus.
3. Future Work and Plans
I am currently working on several projects connected to the contributions listed above. One project is focused on expanding the current methodology discussed above and shown in the figure above. In particular, we are working with Charles River Analytics to expand this work, where we are exploring the development of a probabilistic programming language for HAP — a behavior language developed by Bryan Loyall [6]. Using the probabilistic version of HAP and a human coded HAP behaviors for a game, we can then infer goals and behaviors with some certainty. We believe this approach to plan recognition will be more tractable than other approaches proposed in the community.
This work can expand the methods we use to model players’ behaviors, goals and intent. It also has a far reaching impact and contribution within the area of Machine Learning, specifically in developing human interpretable models as well as models that encode human knowledge.
References
[1] S. Ahmad, A. Bryant, E. Kleinman, Z. Teng, T.-H. D. Nguyen, and M. Seif El-Nasr, “Modeling individual and team behavior through spatio-temporal analysis,” in Proceedings of the Annual Symposium on Computer-Human Interaction in Play, ACM, 2019, pp. 601{612.
[2] A. Canossa, T.-H. D. Nguyen, and M. S. El-Nasr, “G-player: Exploratory visual analytics for accessible knowledge discovery.,” in DiGRA/FDG, 2016.
[3] H. Desurvire and M. S. El-Nasr,”Methods for game user research: Studying player behavior to enhance game design,” IEEE computer graphics and applications, vol. 33, no. 4, pp. 82{87, 2013.
[4] C. Jemmali, E. Kleinman, S. Bunian, M. V. Almeda, E. Rowe, and M. Seif El-Nasr, “Using game design mechanics as metaphors to enhance learning of introductory programming concepts,” in Proceedings of the 14th International Conference on the Foundations of Digital Games, ACM, 2019, p. 65.
[5] C. Jemmali, E. Kleinman, S. Bunian, M. Almeda, E. Rowe, and M. Seif El-Nasr, “Maads: Mixed approach for the analysis of debugging sequences of beginner programmers,” in Proceedings of SIG on Computer Science Education, 2020.
[6] A. B. Loyall, W. Reilly, J. Bates, and P. Weyhrauch, “System for authoring highly interactive, personality-rich interactive characters,” in Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation, Eurographics Association, 2004, pp. 59{68.
[7] D. Moura, M. S. el-Nasr, and C. D. Shaw, “Visualizing and understanding players’ behavior in video games: Discovering patterns and supporting aggregation and comparison,” in Proceedings of the 2011 ACM SIGGRAPH symposium on video games, ACM, 2011, pp. 11{15.
[8] M. S. El-Nasr, S. Durga, M. Shiyko, and C. Sceppa, “Data-driven retrospective interviewing (ddri): A proposed methodology for formative evaluation of pervasive games,” Entertainment Computing, vol. 11, pp. 1{19, 2015.
[9] M. S. El-Nasr, A. Gagne, D. Moura, and B. Aghabeigi, “Visual analytics tools{a lens into player’s temporal progression and behavior,” in Game Analytics, Springer, 2013, pp. 435{470.
[10] M. S. El-Nasr and I. Horswill, \Real-time lighting design for interactive narrative,” in International Conference on Virtual Storytelling, Springer, 2003, pp. 12{20.
[11] M. S. El-Nasr and B. K. Smith, \Learning through game modding,” Computers in Entertainment (CIE), vol. 4, no. 1, p. 7, 2006.
[12] M. S. El-Nasr, A. Vasilakos, C. Rao, and J. Zupko, “Dynamic intelligent lighting for directing visual attention in interactive 3-d scenes,” IEEE Transactions on Computational Intelligence and AI in Games, vol. 1, no. 2, pp. 145{153, 2009.
[13] M. S. El-Nasr, J. Yen, and T. R. Ioerger, “Flame|fuzzy logic adaptive model of emotions,” Autonomous Agents and Multi-agent systems, vol. 3, no. 3, pp. 219{257, 2000.
[14] T.-H. D. Nguyen, E. Carstensdottir, N. Ngo, M. S. El-Nasr, M. Gray, D. Isaacowitz, and D. Desteno, “Modeling warmth and competence in virtual characters,” in International Conference on Intelligent Virtual Agents, Springer, 2015, pp. 167{180.
[15] T.-H. D. Nguyen, M. S. El-Nasr, and A. Canossa, “Glyph: Visualization tool for understanding problem solving strategies in puzzle games.,” in FDG, 2015.