Report Title: Multimodal LLMs as Social Media Analysis Engines

Time: 14:30, Friday, May 16, 2025

Venue: Lecture Hall B501, Science and Education Building, Feicui Lake Campus

Speaker: Professor Jiebo Luo

Affiliation: Department of Computer Science, University of Rochester, USA

Organizer: School of Computer Science and Information Engineering

Report Abstract:

Recent research has demonstrated the outstanding capabilities of Multimodal Large Language Models (MLLMs) in general visual and language tasks. Growing attention has been paid to the performance of MLLMs in specific domains, especially social media content. Essentially multimodal in nature, social media content includes text, images, videos and audio, requiring models to understand the interactions between these different communication modalities and their impacts on information transmission. However, understanding social media content remains a challenge for current machine learning frameworks. To evaluate the capabilities of MLLMs in social media analysis, this study selected five key tasks: sentiment analysis, hate speech detection, fake news identification, demographic inference, and political ideology detection. The research team first conducted preliminary quantitative analysis on each task using existing benchmark datasets, then reviewed the results and selected qualitative samples to demonstrate the potential of GPT-4V in understanding multimodal social media content. GPT-4V exhibited excellent performance in these tasks, demonstrating its advantages in joint understanding of image-text pairs, contextual and cultural awareness, and extensive common sense knowledge. Nevertheless, despite these strengths, GPT-4V still faces challenges, such as the "hallucination" problem (generating inaccurate or fictitious content), difficulties in understanding multilingual social media content and adapting to the latest social media trends. Several strategies are thus proposed to enhance the model's performance in these tasks. The findings of this study offer promising directions for the future development of MLLMs, emphasizing the importance of polymorphic information analysis to deepen the understanding of social media content and its users.

Speaker Profile:

Professor Jiebo Luo is a faculty member in the Department of Computer Science at the University of Rochester, where he joined in 2011 following a distinguished 15-year career at the Kodak Research Laboratories. He has published over 600 technical papers and holds more than 90 US patents. His research interests include computer vision, natural language processing (NLP), machine learning, data mining, computational social science, and digital health. He has been actively involved in organizing numerous technical conferences, serving as Program Chair for ACM Multimedia 2010, IEEE CVPR 2012 and IEEE ICIP 2017, and General Chair for ACM Multimedia 2018 and IEEE ICME 2024. Professor Luo also serves on the editorial boards of several top-tier journals, including IEEE TPAMI, IEEE TMM, TCSVT, IEEE TBD, ACM TST and PR. He is a Fellow of ACM, AAAI, IEEE, AIMBE, IAPR and SPIE, a Member of the Academia Europaea, and a Member of the National Academy of Inventors. He received the ACM SIGMM Technical Achievement Award in 2021 and the IEEE Computer Society Edward J. McCluskey Technical Achievement Award in 2025.