Exploring the Role of Large Language Models in Automating User Interface Tasks

Bachelor & Master Thesis

n recent years, the integration of Large Language Models (LLMs) has transformed various aspects of technology. One emerging area is the utilization of LLMs to automate User Interface (UI) tasks, ranging from executing complex sequences of actions to manipulating software features across different applications [1, 2]. It can significantly reduce the burden of users, especially the disabled or elderly in using the software. This seminar course aims to delve into the systematic exploration of literature on using LLMs for UI automation.

The primary objective of this seminar project is to conduct a comprehensive and systematic literature review on the application of Large Language Models in automating instructions for User Interface tasks. By reading tens of relevant research papers, students will investigate the capabilities, challenges, advancements, and ethical considerations associated with employing LLMs for executing multifaceted UI tasks.

What you will do will include:

• Develop a method to collect relevant papers.
• Through rigorous analysis, students will synthesize the gathered literature, examining the methodologies, models, algorithms, and applications used in LLM-driven UI automation. They will identify trends, challenges, and potential future directions in this domain.
• Present findings, insights, and conclusions derived from the systematic literature review into a formal research report/paper.
• Create a GitHub repo named Awesome-LLM4UIautomation to display recent relevant research works, tools, etc.

Note that this project can be carried out remotely. Students with great performance may be granted an opportunity to do a paid Hiwi in the coming semester break and even a PhD position in the future.

Reference:

1. Wang B, Li G, Li Y. Enabling conversational interaction with mobile ui using large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems 2023 Apr 19 (pp. 1-17).

2. WenH,LiY,LiuG,ZhaoS,YuT,LiTJ,JiangS,LiuY,ZhangY,LiuY.Empoweringllmtouse smartphone for intelligent task automation. arXiv preprint arXiv:2308.15272. 2023 Aug 29.