Title: Large Language Models as Computational Engines and Virtual Domain Experts for Visual Data Analysis
Date: Thursday, October 10, 2024
Time: 9 a.m. - 11 a.m. ET (US)
Location: Technology Square Research Building (TSRB) 334
Virtual meeting (hybrid): Click here to join Zoom meeting
Alexander Bendeck
Ph.D. Student in Computer Science
School of Interactive Computing
Georgia Institute of Technology
Committee
Dr. John Stasko (Advisor) - School of Interactive Computing, Georgia Institute of Technology
Dr. Alex Endert - School of Interactive Computing, Georgia Institute of Technology
Dr. Clio Andris - School of City and Regional Planning, Georgia Institute of Technology
Dr. Cindy Xiong Bearfield - School of Interactive Computing, Georgia Institute of Technology
Dr. Ross Maciejewski - School of Computing and Augmented Intelligence, Arizona State University
Abstract
Advances in generative artificial intelligence have led to the development of pre-trained large language models (LLMs) which are widely available and broadly useful. For data visualization researchers, LLMs have the promise to extend existing research threads in exciting directions, potentially super-charging visualization systems with their vast domain knowledge and computational power. However, LLM-powered systems pose new challenges for both visualization researchers and our intended system users. For instance, well-documented hallucination and inconsistency issues with LLMs can inhibit visualization system performance and erode user trust. We also have little formal understanding of LLMs’ ability to help data analysts with specific tasks.
The aim of my thesis work is to study the potential use of LLMs as “virtual domain experts” during visual data analysis. This includes two main goals: First, to evaluate LLMs at applying their knowledge bases to data- and chart-centric tasks; and second, to study user task completion, satisfaction, and trust for LLM-powered visualization systems. I addressed the first goal through an empirical evaluation of the GPT-4V multimodal language model on a suite of visualization literacy tasks, demonstrating the state of the art in LLM performance at reading and understanding visualizations. I propose subsequent work to address both goals by assessing LLMs’ domain knowledge and generative capabilities on two specific tasks: question answering and data integration. For each task, I will conduct formative studies, empirical evaluations, and design probes using prototype visualization systems, exploring both technical and human-centered perspectives on the use of LLMs during visual data analysis.