GUIEvo: Automated Evolution of Mobile Application GUIs from Mockups

Abstract

With the increasing use of mobile applications in today’s digital world, touch-based graphical user interfaces (GUIs) have become a crucial component of modern software by which end-users carry out computing tasks. As such, the tools involved in creating these GUIs are of fundamental importance. Due to the continuous pressure for frequent releases of mobile apps to keep pace with platform and device updates, the practice of evolving app GUIs is central to mobile app maintenance. Currently, developers manually introduce GUI changes to their apps as they evolve in a time-consuming process that involves creating mock-ups of updated GUIs and then implementing the changes stipulated by the mock-up. To help ease the burden of implementing GUI changes, and to help free mobile app developers to focus on fixing bugs or adding features, this paper introduces an automated approach for GUI evolution, called GUIEvo. This approach aims to assist developers in the process of GUI evolution by detecting changes in GUIs between existing releases and proposed mock-ups using computer vision techniques, and automatically generating updated GUI metadata for the new release. We evaluate our approach’s performance based on accuracy, precision, recall, and F1-score in detecting the GUI changes, and tree edit distance to measure the correctness of generated UI hierarchies. Our evaluation demonstrates that GUIEvo can detect GUI changes with over 85% accuracy, and the generated GUI hierarchies closely match the expected structure with an average tree edit distance of 5.9. This work points toward the promise of automated tool support for assisting in the evolution of GUIs.

S M Hasan Mansur

PhD Candidate

My research interests include Software Engineering, HCI, Machine Learning, Generative AI, Large Language Model (LLM).