In the world of web development, AB testing is an frequently used tool to test assumptions and understand user behaviour. Due to the centralized natural of web apps (eg - all the code runs within your servers) doing AB testing is lightweight and quick.

AB testing in iOS apps is not as easy, because each new test likely requires a new app update. It is still important, but needs to be handled differently.

For Rocket Speller, we wanted to make sure that the game was easy to understand and play, so we used some AB testing for this purpose. Our goal was to make sure the game was fun and easy to understand. The metrics we decided to use to measure this were:

• Completion Rate: What percentage of people complete the game after starting it.
• Games per Session: How many games, on average, were played while the app was open.

Through AB testing we increased the completion rate from 23% to 42%, and increased the games per session from 1.1 to 1.3. In you want to learn more, you can read about how we ran the AB tests or read about the results of the tests.

Best Practices for A/B testing an iOS app:

1. Have a clear goal: Any test is useless without a goal. Our goal was simple: Get more people to finish the game once they've started.
2. Measure it: If you can't easily measure your goal, then it's not a real goal.
3. User users tests to guide you: Watch how people (parents, friends, etc) use your app the first time. This can either help you discover a problem, set your goal, or come up with ideas for your test. Watching someone who doesn't know your app at all provides amazing insight.
4. Make your tests count: Since AB tests in mobile apps are more relatively expensive, you have to make them efficient. When we watched people playing Rocket Speller for the first time they would often spell a word or 2 and ask what happens next. So, we tested a 'progress bar' that would indicate how far along people are. By adding 2 different progress bars (with 1/2 of users seeing each) in 1 test we could compare progress bar A vs progress bar B vs no progress bar (from previous release)
5. Run 1 test per update: Running multiple AB tests within an app significantly increases the difficultly of testing and the likelihood of a bug. Just run 1 test at a time.

We wanted to run some AB tests in Rocket Speller so that we could measure and improve how kids were playing the game. If there were parts that didn't make sense, or where kids were getting stuck, we wanted to know. After running a series of tests we got pretty good results, and we wanted to share how we actually performed the tests by showing some code.

One of the tests we ran was to compare different types of 'progress views' - an on screen indication of game progress. We had 2 versions of the progress bar. The first only showed the progress for the current section. A section consists of 3 or 4 words that needed to be spelled before getting to a planet where a rocket ship piece would be selected. The second progress bar showed the progress of all of the sections at one time (eg - the full game).

To manage the AB test we used an "Options" class. Here is the header file:

//  Options.h

#define KEY_PROGRESS_VIEW_TYPE @"lbt_rs_progress_view_type"
#define PROGRESS_VIEW_PER_SECTION 1
#define PROGRESS_VIEW_COMPLETE_GAME 2

@interface Options : NSObject
+ (int)abProgressViewType;
@end

As for the implementation, we used NSUserDefaults to store values so that one app would always see the same view. If no value existed in the defaults then we would pick a random value and set it, as you can see below:

// Options.m - class for managing ab test options 
#import "Options.h"

@implementation Options

+ (int)abProgressViewType {
    NSUserDefaults *defaults = [NSUserDefaults standardUserDefaults];
    int progressViewType = [defaults integerForKey:KEY_PROGRESS_VIEW_TYPE]; 
    if (progressViewType == 0) {
        int r = (arc4random() % 100);
        // Split progress views 50/50
        progressViewType = ((r < 50) ? PROGRESS_VIEW_PER_SECTION : PROGRESS_VIEW_COMPLETE_GAME);
        [defaults setInteger:progressViewType forKey:KEY_PROGRESS_VIEW_TYPE];
    }
    return progressViewType;
}
@end

In our main UIViewController we set up a progressView object. The actually type of progressView used was based on the Options class AB test code.

if ([Options abProgressViewType] == PROGRESS_VIEW_PER_SECTION) {
    self.progressView = [[ProgressViewPerSection alloc] initWithFrame:[k PROGRESS_VIEW_FRAME]];
} else {
    self.progressView = [[ProgressViewComplete alloc] initWithFrame:[k PROGRESS_VIEW_FRAME]];
}

Now that the AB test was set up and running, we needed to measure the results. In our case, the goal was to end the game. We used the Flurry analytics tool for this. It allows you to log events within your app. To measure our results, we set up an event when a game was started, and an event when a game was completed. To each event we passed an NSDictionary of values that included the type of progress view we used. The code for reporting/recording the results of the test is below.

- (NSMutableDictionary *)paramsForFlurry {
    NSMutableDictionary *params = [NSMutableDictionary dictionary];
    if ([Options abProgressViewType] == PROGRESS_VIEW_PER_SECTION) {
        [params setValue:@"Per Section" forKey:@"Progress View"];
    } else {
        [params setValue:@"Full Game" forKey:@"Progress View"];
    }
    return params;
}

- (void) startGame {
    // Call this at start of the game / when the goal starts
    NSMutableDictionary *params = [self.gameController paramsForFlurry];
    [FlurryAnalytics logEvent:@"1: Game Started" withParameters:params];
    // ...
}

- (void) endGame {
    // Call this at end of the game / when the goal has been acheived
    NSMutableDictionary *params = [self.gameController paramsForFlurry];
    [FlurryAnalytics logEvent:@"5: Game Finished" withParameters:params];
   // ...
}

Flurry provides nice charts outlining events and frequencies of different values for the parameters of the events, so it's easy to see which of the 2 variants results in more goal completion.

AB Testing in iOS apps is very helpful in improving your app, and, as you can see by the above code, fairly simple to implement.

We believe that apps for kids should be engaging, easy to use, and fun. Rocket Speller taught us a lot of lessons about how achieve those goals and we wanted to share what we learned so that other kids app developers may also apply those lessons.

In the first 3 app updates of Rocket Speller we were able to increase the completion rate by 64% through some simple user observation and A/B testing!

Here is what we learned:

Context and feedback encourage continued play

In the first version of Rocket Speller there was no feedback about how far you were progressing, or context about why you were spelling words (e.g. to collect rocket pieces). We noticed that people would start the game and exit after a few words, unaware what would come next. So, we added a simple star based progress meter along the bottom to show you how close you were to getting a rocket piece.

Kids got 1 star for every word they spelled and once they got 4 stars they would get to a planet where they could start building their rocket. The stars provided immediate feedback that they were doing well and context about how many words they needed to spell. This increased the game completion rate from 23% to 27.5%.

Frequent rewards are important

We tested different numbers of words (learning) required to get to a planet (reward). By changing from 4 words to 3 words we increased the completion rate for the full game from 27.5% to 34%.

What's more, when we used 3 words rather than 4, the number of total games played increased. As a result, those with the 3 word version actually spelled more words in total! Of course, you have to find the right balance. We could have dropped to 1 or 2 words, but then, in our opinion, the learning to reward ratio would be too heavily weighted to reward.

Progress indicators provide incentive to complete goals

In our last test we updated the progress indicator from a simple 3 star indicator into a more complex 'full game' progress indicator.

The new progress indicator provided both short term (next rocket piece) and long term (how long until I can launch the rocket) feedback. This encouraged people to complete the entire game and once again increased the total number of games played per user. The full game progress meter increased the game completion rate from 34% to 38%.

Summary

We firmly believe that when kids are fully engaged they will develop a love for learning and discovery. We will be using these lessons in our upcoming apps and will continue to explore different ways to increase engagement. We hope these lessons help other developers too. Likewise, we'd love to hear what lessons other developers have learned.

One of the most useful tools for iOS app developers is the analytics tools Flurry. It gives you great analytics data on sessions, installs, usage, device info, etc. One thing it is lacking (surprisingly) is data on the versions of iOS that your app is running on. Fortunately that info is easy to get with a few lines of code that make use of Flurry's custom event reporting.

Below we'll outline 2 different ways of tracking iOS version info.  You can place the code in your application didFinishLaunchingWithOptions method in your AppDelegate.

Option 1 : Report the iOS version for each session

This option records the iOS version each time the app is opened. It shows how the iOS versions change over time, but may not be an accurate gauge of what percentage of devices use which versions of iOS.

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
    NSDictionary *params = [NSDictionary dictionaryWithObjectsAndKeys:[[UIDevice currentDevice] systemVersion], @"iOS Version", nil];
    [FlurryAnalytics logEvent:@"Device Info" withParameters:params];
}

Option 2 : Report the iOS version for install

This only reports the first time the app is opened. It is better correlated to individual devices, but will be stale over time since it doesn't catch upgrades.

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions { 
    NSString *key = @"my_special_key_for_tracking_device_reporting";
    NSUserDefaults *defaults = [NSUserDefaults standardUserDefaults];
    int hasReportedDeviceInfo = [defaults integerForKey:key]; 
    if (hasReportedDeviceInfo == 0) {
NSDictionary *params = [NSDictionary dictionaryWithObjectsAndKeys:[[UIDevice currentDevice] systemVersion], @"iOS Version", nil];
        [FlurryAnalytics logEvent:@"Device Info" withParameters:params];
        [defaults setInteger:1 forKey:key];
    }
}

You could also easily implement some code which stores the iOS version in the NSUserDefaults, reports on first open, and reports again any time a change in the iOS version is discovered. We'll leave that as an exercise for the reader.  For us, getting a sense of what iOS versions the app is played most often on is the most relevant info, so we used option #1. Here is what the popularity of different iOS versions looks like in our Rocket Speller in Mar 2012.

Why do this? Well - perhaps you want to use an iOS5 only feature like CAEmitterLayer for fire and smoke. Using such an SDK feature means you can only support iOS 5 devices. If 95% of your users are already on iOS 5 then maybe that's a trade off you are willing to make. If not, you'll have to look into something like cocos2d.

Important: This post deals with tracking user data. If you are an app developer, especially if you develop kids apps, please do the right thing and disclose to your users that you track usage data in your apps. We state in an information panel within our apps that we use Flurry to track anonymous usage data.

Animation is a big part of a lot of iOS kids apps but it's not very clear how to enable interaction with UIViews while they are being animated.  Google searches bring you to various stack overflow threads which simply state that all you need to do is enable the UIViewAnimationOptionAllowUserInteraction flag on the animation like so:

// This is not sufficient to allow interaction with an animating/moving UIView 
// because the position attribute doesn't correspond to the where the view is 
// being animated, but rather where it begins or ends
[UIView animateWithDuration: 1.0
    delay: 0
    options: UIViewAnimationOptionAllowUserInteraction
    animations:^{ myMovingImage.center = CGPointMake(50,50) }
    completion:^(BOOL finished) { }
 ];

If the UIView is moving (e.g. the position is being animated), even though user interaction is enabled, you will not be able to interact with the view.  This is due to how animations work.  The view itself is either at the original position, or at the final position.

If you want to interact with (e.g. move) a UIView that is moving via an animation, you need to use touch detection in the UIViewController and get the 'current' position of the UIView using it's presentation layer.

// If you really want to interact with a moving object - you need to
// 1 - detect the touch in a UIViewController 
// 2 - run a hitest against the presentationLayer of the animating UIView
- (void)touchesBegan:(NSSet *)touches withEvent:(UIEvent *)event {
    UITouch *touch = [touches anyObject];
    CGPoint touchPoint = [touch locationInView:self.view];
    if ([[myMovingImage.layer presentationLayer] hitTest:touchPoint]) {
        // Yay - we've now detected that the animating UIView myMovingImage has been touched.
    }
}