Adventures with Google Play Store Listing Experiments

Posted on Thursday, January 21st, 2016 by Blake Darling
Category: Android

The Creative SDK’s Image Editor is based on the Aviary app for Android and iOS. For the Aviary app, we’ve recently been experimenting with how Google Play screenshots affect download rates.

I’d like to share some things we’ve learned from our experiments with A/B testing.

Background

Our Google Play representative suggested we try updating our app screenshots to improve our app download numbers. His first recommendation was to change the style of our screenshots.

We were using full screen screenshots (on the left in the image below). Instead, he recommended we use in-device screenshots annotated with text (on the right), which is more commonly seen across the Play Store.

Google Play Store A/B testing

He also suggested we experiment with the order in which we tout Aviary’s features. Currently, we call out our most popular feature first (the number of free tools in the editor). He mentioned other apps have had success highlighting the feature that differentiates them from similar apps (for Aviary, this is our content: effects, stickers, etc.). Another suggestion was to try featuring awards and press mentions first.

We’ve been itching to try out the new Google Play Store Listing Experiments, which allows A/B testing for variations of store listings, and this was the perfect opportunity! So I set up a test with four versions against a small subset of users.

The Tests

Below are the images we are currently using in the Google Play Store.

Current Version

This set of images was our control for the tests. As you can see, the individual images are simple screenshots of the app.

We then tried three different tests:

  1. screenshots in the existing order
  2. screenshots with content featured first
  3. screenshots with accolades (awards and/or press mentions) first

Test 1: Screenshots with Existing Order

In the first test, we showed features of the app in the same order that we did before. The difference was that now each screenshot was annotated with an explanation of the feature or benefit to the user.

Test 2: Screenshots with Content First

In the second test, we front-loaded a couple of screenshots that show off our most popular content features: effects, stickers, and frames.

Test 3: Screenshots with Accolades First

For the final test, the first screenshot contained media accolades from Mashable, USA Today, and Lifehacker.

The results

“Text with Existing Order” outperformed “Current Version”, but the others came up short. I was happy to see an improvement, however to be honest I was hoping it would be more significant.

Google Play A/B testing

Then, I remembered that ~70% of our Android userbase is outside the US. This experiment was global, so maybe the screenshots with text annotations didn’t perform as well as they could have because they were not localized.

The Localized Test

Luckily, Google Play allows you to experiment with Store Listings by language. Localizing screenshots is a lot of work, so before doing it for every language, I set up a test in one of our more popular countries (Russia) with two versions.

Current Version

This set of images is the same as our first test above, where the screenshots are annotated with an explanation of the feature or benefit to the user.

Russian Text

For this test, we localized the screenshot annotations from the Current Version test into Russian.

Our hope was that the localized screenshots would significantly improve our download numbers. If not, at least we would save time by only testing on one language.

The Localized Text Results

Success! Combined with the results of the previous experiment, we might be able to expect a ~5% increase in installs just by updating and localizing our screenshots. That is significant.

Google Play A/B testing

This is just a start. Every language might be different, but this confirms it is worth spending the time for further experimentation.

Going forward, I’m eager to continue this experiment with different languages. Also, I’m curious whether using different images or highlighting different use cases for the editor might resonate better or worse on a per country basis.

The Creative SDK Image Editor

You can add the power of our image editor to your Android, iOS, or web app with the Creative SDK Image Editor. With a minimal amount of code, you can be up and running with an image editor that offers effects, stickers, cropping, and more.

Start by integrating the Creative SDK Image Editor into your Android app, then conduct your own experiments with A/B testing on the Google Play Store.

Visit the Creative SDK developer portal to learn more.