FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents

FingerTip contains over 20,000 episodes, covering demonstrations of various daily tasks performed by humans on more than 500 different Android apps, derived from scenarios in users' daily lives, which includes user-related contextual information and users’ long-term mobile phone usage patterns. FingerTip also includes two brand-new tracks, to evaluate the ability of mobile GUI agents to proactively predict user intents and offer suggestions, as well as their ability to personalize task execution in accordance with user preferences.

Data format

The data folder has structure as follows:

├── <user_id_1>
│   ├── <episode_name_1>
│   │   ├── <screenshots>
│	│	│	├── <screenshot_name_1>
│	│	│	├── <screenshot_name_2>
│	│	│	├── <xml_name_1>
│	│	│	├── <xml_name_2>
│	│	│	...
│   │   ├── action.jsonl
│   │   ├── survey_result.json
│   ├── <episode_name_2>
│   │	...
├── <user_id_2>
│   ...

The data contains several folders named with numbers (i.e. user IDs), and each of these folders contains multiple folders named with timestamps (e.g., 20250309_133115), representing all the data episodes submitted by that user. For each data episode, the following information is included:

• screehshots: a list of screenshots for each observation encoded as JPGs, stored in the "screenshots" folder.
• accessibility trees: a list of Android accessibility tree XML files for each observation, also stored in the "screenshots" folder.
• actions: a list of actions represented in the form of JSON dictionaries, stored in "action.jsonl". Each screenshot corresponds to an action.
• intent_description: the user’s true intent in this episode, stored in "survey_result.json".
• user_id: the unique integer identifier of the user to whom this episode belongs, stored in "survey_result.json". This information can be used to retrieve the corresponding user’s user profile.
• time: the timestamp when this episode was collected, stored in "survey_result.json".
• scenario: the category of location where the user was when this episode was collected, stored in "survey_result.json".
• app: the name of the activity running when the episode was collected, stored in "survey_result.json". This information is only used to launch the corresponding app in personalized task execution and is not provided to the LLM agent.

Quick start

Installation

The data can be found here. Download all the data and code to the same-level directory and install the necessary libraries listed in requirements.txt.

Proactive task suggestion track

1. Set the LLM API and name in proactive_suggestion.py.
2. In proactive_suggestion.py, ensure that the paths of all necessary files are correct, including the test task file (we have provided test_suggestion.csv and its subset sampled_test_suggestion.csv), result file, as well as total.csv and user_profile.csv.
3. Set the task difficulty in proactive_suggestion.py, that is, the number of initial screenshots provided to the LLM.
4. The get_prompt function can be modified to customize the LLM Agent.
5. run proactive_suggestion.py.

Personalized task execution track

1. Download and install the Android Debug Bridge (adb) on your PC. This command-line tool allows you to communicate with your Android device from your PC. After installation, add the adb tool to your system's environment variables.

2. It is recommended to use a physical mobile phone for testing, because most Chinese apps can only run on physical phones rather than emulators due to strict app control measures. If you still want to use an emulator, you can download and install Android Studio. If you use a physical phone:

   • Enable USB debugging by going to Settings > Developer Options > USB debugging. If Developer Options is not visible, go to Settings > About phone and tap on Build number seven times to enable it.

   • Connect your Android device to your PC using a USB cable.

3. Set the LLM API and name in personalized_execution.py.

4. In personalized_execution.py, ensure that the paths of all necessary files are correct, including the test task file (we have provided test_execution.csv and its subset sampled_test_execution.csv), result file, as well as total.csv and user_profile.csv.

5. The get_prompt function can be modified to customize the LLM Agent.

6. run personalized_execution.py.