The Multimodal Learning Analytics Handbook

This handbook is the first book ever covering the area of Multimodal Learning Analytics (MMLA). The field of MMLA is an emerging domain of Learning Analytics and plays an important role in expanding the Learning Analytics goal of understanding and improving learning in all the different environments where it occurs. The challenge for research and practice in this field is how to develop theories about the analysis of human behaviors during diverse learning processes and to create useful tools that could augment the capabilities of learners and instructors in a way that is ethical and sustainable. Behind this area, the CrossMMLA research community exchanges ideas on how we can analyze evidence from multimodal and multisystem data and how we can extract meaning from this increasingly fluid and complex data coming from different kinds of transformative learning situations and how to best feed back the results of these analyses to achieve positive transformative actions on those learning processes.

This handbook also describes how MMLA uses the advances in machine learning and affordable sensor technologies to act as a virtual observer/analyst of learning activities. The book describes how this "virtual nature" allows MMLA to provide new insights into learning processes that happen across multiple contexts between stakeholders, devices and resources. Using such technologies in combination with machine learning, Learning Analytics researchers can now perform text, speech, handwriting, sketches, gesture, affective, or eye-gaze analysis, improve the accuracy of their predictions and learned models and provide automated feedback to enable learner self-reflection. However, with this increased complexity in data, new challenges also arise. Conducting the data gathering, pre-processing, analysis, annotation and sense-making, in a way that is meaningful for learning scientists and other stakeholders (e.g., students or teachers), still pose challenges in this emergent field. This handbook aims to serve as a unique resource for state of the art methods and processes.

Chapter 11 of this book is available open access under a CC BY 4.0 license at link.springer.com.

Michail (Michalis) Giannakos is a professor of interaction design and learning technologies at the Department of Computer Science of the Norwegian University of Science and Technology (NTNU). He is the head of the Learner-Computer Interaction lab and his research focuses on the design and study of emerging technologies in online and hybrid education settings, and on developing new ways for humans to interact with interactive learning systems.
Giannakos has co-authored more than 200 manuscripts published in prestigious peer-reviewed journals and conferences (including Computers & Education, Computers in Human Behavior, IEEE Pervasive Computing, IEEE TLT, BIT, BJET, ACM TOCE, ACM IDC, ICLS/CSCL, Interact).
Giannakos, together with Mike Horn, is the Editor-in-Chief of the International Journal of Child-Computer Interaction (Elsevier). He is also on the Editorial Board of IEEE Transactions in Learning Technology, IEEE Transactions on Education, Behaviour & Information Technology, and the International Journal of Information Management, and has served as a guest editor on highly recognized journals such as BJET, Computers in Human Behavior, ACM TOCE and IEEE Multimedia. He has served as an evaluator for the European Commission (EC) and the US-NSF, and he recently authored a book on Experimental Studies in Learning Technology and Child-Computer Interaction (Springer) and co-authored the Educational Data Literacy textbook (Springer).
Giannakos has worked at several research projects funded by diverse sources like the EC, Microsoft Research, The Research Council of Norway (RCN), US-NSF, the German agency for international academic cooperation and Cheng Endowment. Giannakos is one of the experts in the Norwegian task force (formed by the ministry of education and research) for introducing learning analytics to Norwegian K-12 schools and universities. He is also a recipient of a Marie Curie/ERCIM fellowship, the Norwegian Young Research Talent award and he is one of the outstanding academic fellows of NTNU (2017-2022).
Giannakos usually attends the ACM Interaction Design and Children (IDC) conference (he was a Chair for IDC 2018, and PC-chair for IDC 2022), the ACM conference on Innovation and Technology in Computer Science Education (ITiCSE) (he was a Chair for ITiCSE 2020), and the Learning Analytics and Knowledge (LAK) conference.
Daniel Spikol is Associate Professor of Computational Thinking in the Computer Science and Science Education departments at the University of Copenhagen (UCPH). He is part of the university's Center for Digital Education, which investigates and helps support digital education transformation. Daniel holds a PhD in Computer Science for Linnæus University, Sweden (2010), an MSc in Visual Studies from Massachusetts Institute of Technology (1992), and a BFA from Rhode Island School of Design (1987).
The central theme of his research investigates how to integrate computational thinking and tinkering into education. His work combines physical computing and robotics with STEM and STEAM education to provide an authentic context to engage people in science and computing. Additionally, his research investigates how people collaborate with multimodal learning analytics (inspired by social signal processing ambient computing). Daniel combines science, design, digital art, and computer science to investigate how people learn and play. He has 70 peer-reviewed scientific publications. He has worked for different companies and research institutes and founded several companies.
Daniel has been active in developing SOLAR's SIG for Multimodal Learning Analytics, participated in the program committees for different conferences such as LAK, EC-TEL, ICALT, AIED, IDC, and ICQE. He has also been a principal investigator for European and national research projects. Previous to UCPH, he was the scientific manager of the Internet of Things and People Research Lab at Malmö University and part of the Computer Science and Media Technology department.
Daniele Di Mitri is a research group leader at the Education Technology group at DIPF Leibniz Institute for Research and Information in Education and a lecturer at the Goethe University of Frankfurt. Daniele received his Ph.D. title at the Open University of The Netherlands (2020) in Learning Analytics. His Ph.D. project, the Multimodal Tutor, investigated the potentials of collecting and analysing multimodal data during physical interactions for automatic feedback and human behaviour analysis. Daniele's current research focuses on designing responsible Artificial Intelligence applications for education and human support. He is a "Johanna Quandt Young Academy" fellow. He was elected "AI Newcomer 2021" at the KI Camp by the German informatics societyand received the "Martin Wolpers Award" in technology-enhanced learning 2018. He is a member of the editorial board of Frontiers in Artificial Intelligence journal, a member of the CrossMMLA, a special interest group of the Society of Learning Analytics Research, and he is the chair of the Learning Analytics Hackathon (LAKathon) series.
Kshitij Sharma is is associate professor of learning technologies at the Department of Computer Science of the Norwegian University of Science and Technology (NTNU). He is member of the Learner-Computer Interaction lab and his research focuses on the using artificial intelligence and machine learning to study students' behaviour and design emerging technologies in online and hybrid education settings for them to learn in more effective and efficient manner. Sharma holds a Ph.D. in computer science from EPFL, Lausanne, Switzerland.
Sharma has co-authored more than 100 manuscripts published in prestigious peer-reviewed journals and conferences (including BJET, JCAL, Computers in Human Behavior, IEEE Pervasive Computing, IEEE TLT, BIT, BJET, IMWUT, ACM CHI, ACM IDC, ICLS/CSCL). His research is focused on studying relations between students' profile (individual or collaborating) and interaction patterns with technology. Sharma's research interests center on making sense of students' experiences and actions to optimize and redesign the systems. The goal is to understand why and how diverse learner categories (based on their motivation, strategies, expertise) use technologies (online and standalone) in the ways that they do and then to adapt to their needs (knowledge management and enterprise-adapted e-learning).

Xavier Ochoa is Assistant Professor of Learning Analytics in the Department of Administration, Leadership, and Technology of the Steinhardt School of Culture, Education, and Human Development. He is also core-faculty of the Learning Analytics Research Network (NYU - LEARN). Xavier holds a Ph.D. in Engineering (Computer Sciences) from the University of Leuven - Belgium (2008), an M.Sc. in Applied Computer Sciences from the Vrije Universiteit Brussels - Belgium (2002), and a B.S. in Computer Science from Escuela Superior Politécnica del Litoral (ESPOL) - Ecuador (2000).
Since the start of his research career, Xavier has been working at the convergence point of Education and Technology. His research work focuses on how to use cutting-edge technology to augment human capabilities in education to innovate pedagogical practices that, in turn, lead to better learning processes. He started his career working on quantitative analysis and modeling of the production, finding, consumption, and reuse of digital learning materials. This work conducted his research into the nascent field of Learning Analytics where he was invited as a keynote of the first Learning Analytics and Knowledge conference (LAK). Currently, Xavier works on Multimodal Learning Analytics, a sub-field that involve the use of advanced quantitative techniques from the field of learning analytics, together with recent advances in machine learning, computer vision, speech and natural language processing, and smart sensors to build and study the educational effect of tools that augment the awareness, self-reflection, sense- and decision-making of students and instructors in both physical and online learning environments. He is the lead investigator of the Augment-Ed research group.
Xavier Ochoa has served as Vice-president of the Society for Learning Analytics Research (SoLAR) and is currently the co-Editor-in-Chief of the Journal of Learning Analytics. He is also a founding member and long-time coordinator of the Latin American Community on Learning Technologies (LACLO). He has also received the Best Researcher Award (2018 and 2012) and Best Professor Award (2013)at ESPOL and the 5-year best Researcher in Computer Science by the Institute of Electrical and Electronics Engineers (IEEE) in Ecuador (2014). He has also received several Best Paper Awards at international conferences, including LAK.
Rawad Hammad is a Programme Leader for MSc Computing, Programme Leader for MSc Digital Education, Technology Enhanced Learning Research Group Leader, and a Senior Lecturer in Computer Science and Digital Technologies at the University of East London. Rawad has an extensive experience on Software Engineering, Technology Enhanced Learning (TEL), Artificial Intelligence in Education, and Smart Health research and practice. Rawad contributed to and led various international projects, published research articles and has been involved in various conference programme committees including EC-TEL, LAK, AIED and BUSTECH. Moreover, Rawad is supervising postgraduate students, currently 5 PhD students in addition to varied number of MSc students. Rawad is an executive committee member of the International Society of the Artificial Intelligence in Education (AIED) and committee member of different research bodies, conferences including: EC-TEL, LAK, AI for Post-COVID Education, Networks in Education. In 2018, he received his PhD in Software Engineering from the University of the West of England (UWE). In 2010, he received his MSc in Cognitive Computing from Goldsmiths University of London. He led numerous international projects such as: TRANSFER and SmartTech which include partnerships with different universities, research centres, governmental institutes from different countries including Japan, Germany, Finland and Middle East. Prior to coming to University of East London, Rawad was a Senior Education Solutions Researcher/Analyst at King's College London; Researcher at Centre for Complex Cooperative Systems at the University of the West of England.