Large Language Models for GUI-related Bug Repair

Bachelor & Master Thesis

Ensuring the quality and user experience of software applications is paramount, especially as Graphical User Interfaces (GUIs) become increasingly intricate. However, identifying and rectifying GUI-related bugs remains a significant challenge due to the complexity of visual elements and user interactions.

This empirical study aims to investigate the potential of Large Language Models (LLMs) in automating the repair of GUI-related bugs in web applications, with a particular focus on the SWE-bench Multimodal dataset [1]—a benchmark designed to evaluate AI systems on visual, user-centric JavaScript software.

By leveraging LLMs, the project seeks to develop innovative methods for:

Bug Repair: Generating solutions to fix GUI-related issues effectively.

Patch Validation: Evaluating the effectiveness of GUI-related bug fixes by letting LLMs simulate interaction behavior.

Required knowledge:

We are seeking motivated Bachelor's or Master's students with:

• A solid programming background (Python and JavaScript).
• Familiarity with web front-end development.
• An understanding of software development processes and tools.

Reference:

[1] Yang, John, et al. "SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?." arXiv preprint arXiv:2410.03859 (2024).