In the world of web development, AB testing is an frequently used tool to test assumptions and understand user behaviour. Due to the centralized natural of web apps (eg - all the code runs within your servers) doing AB testing is lightweight and quick.
AB testing in iOS apps is not as easy, because each new test likely requires a new app update. It is still important, but needs to be handled differently.
For Rocket Speller, we wanted to make sure that the game was easy to understand and play, so we used some AB testing for this purpose. Our goal was to make sure the game was fun and easy to understand. The metrics we decided to use to measure this were:
- Completion Rate: What percentage of people complete the game after starting it.
- Games per Session: How many games, on average, were played while the app was open.
Through AB testing we increased the completion rate from 23% to 42%, and increased the games per session from 1.1 to 1.3. In you want to learn more, you can read about how we ran the AB tests or read about the results of the tests.
Best Practices for A/B testing an iOS app:
- Have a clear goal: Any test is useless without a goal. Our goal was simple: Get more people to finish the game once they've started.
- Measure it: If you can't easily measure your goal, then it's not a real goal.
- User users tests to guide you: Watch how people (parents, friends, etc) use your app the first time. This can either help you discover a problem, set your goal, or come up with ideas for your test. Watching someone who doesn't know your app at all provides amazing insight.
- Make your tests count: Since AB tests in mobile apps are more relatively expensive, you have to make them efficient. When we watched people playing Rocket Speller for the first time they would often spell a word or 2 and ask what happens next. So, we tested a 'progress bar' that would indicate how far along people are. By adding 2 different progress bars (with 1/2 of users seeing each) in 1 test we could compare progress bar A vs progress bar B vs no progress bar (from previous release)
- Run 1 test per update: Running multiple AB tests within an app significantly increases the difficultly of testing and the likelihood of a bug. Just run 1 test at a time.