OTS Prompting Guidelines¶
Guidelines for creating effective prompts to train AI agents on web applications.
Core Principles¶
Key Objective
We are trying to teach agents how to use web apps. Think of these prompts as raw instructions. Using assertions, we test if the model has done them correctly. Using learning algorithms, models eventually learn to perform these tasks.
- Think you are teaching agents - Every prompt should be a learning opportunity
- Find the objective point - Each prompt must have a clear, measurable goal
Prompt Requirements¶
Content Guidelines¶
| Requirement | Description |
|---|---|
| Coverage | Create prompts that cover all available features in the gym app |
| Objectivity | Prompts should be 100% objective. No subjectivity allowed for state-changing parts |
| Natural Language | Make prompts as natural as possible - tasks a real human would ask an AI Agent to do |
| Backstories | You can add backstories to make prompts more realistic |
Length Guidelines¶
- Standalone prompts: Around max 6 sentences
- Cross gyms prompts: Can be around 3-10 sentences
Difficulty Distribution¶
The prompts should follow this distribution:
| Difficulty | Percentage |
|---|---|
| Easy | 20% |
| Medium | 20% |
| Hard | 60% |
Response Dependent Tasks¶
Special Rules for RD Tasks
- 10 tasks should be Response Dependent Tasks
- The model can provide subjective answers but still based on concrete data pulled from the site
What to Avoid¶
Don't Be Too Helpful
Prompts should not be too helpful.
Bad Example: "Go to settings, select drop down value x"
This is too helpful because it tells the agent exactly what to do.
- All tasks should be objective (except for the open-ended part of RD tasks)
- Models should still need to find their way to the objective
- Don't give step-by-step instructions that eliminate the learning challenge