Fastlane: a growth engine fueled with ads

How holistic experimentations on ad monetisation amplified with smart UA took Fastlane from 170,000 to 700,000 DAU in 4 months. And growing.

  • Fastlane has reached 16M installs, approaching $30M run rate, and is on an explosive growth trajectory 10 months after launch
  • Fastlane is an evolved arcade shooter game available for free on iOS and Android phones
  • From $5,000/d to $45,000/d from ads in 4 months: what are the lessons learnt from our holistic iterations and partnership with Unity Ads
  • We are setting a new benchmark for ads at $0.13 ad arpdau in the US
  • Our Ad LTV – lifetime value – is now based on true ad performance to gain accuracy
  • We multiplied our User Acquisition budgets by 5x with our lean team of 2. And we are profitable under a month at $0.52 CPI direct

blog_fastlanelogo

16M installs, approaching $30M run rate, 700k DAU, and onto a recent explosive growth trajectory.

Fastlane was developed in 6 months by a team of 8 people. The team’s thesis was that there was a gap in the market between hyper-casual and midcore titles. A gap where casual addictive gameplay can meet $0.25+ arpdau in Tier 1 geos, marrying IAP and ads while maintaining good retention metrics at 12% d28 and attracting more than 100,000 new users daily.

We feel we’ve built a replicable growth engine with Fastlane. Better – we have improved our ad monetisation stack and our understanding of ad LTV – lifetime value – as well as forged long term partnerships that will have a long lasting impact in our future strategies going forward.

blog_realgraph(Fastlane daily active user base and revenue has been growing week on week at an explosive growth rate since November 2017 – and is more profitable than ever)

Fastlane’s stats by mid-March 2018, 10 months after launch:

  • 16M installs, approaching $30M run rate
  • 700,000 DAU (up from 170k in nov 2017)
  • 2.5M+ daily video views
  • $80,000 daily booking (iap + ads), with highs approaching $100,000/d
  • $45,000 daily booking with ads alone (up from $5k/d in nov 2017)

This article sums up our main learnings on our ad monetisation implementation and partnership that increased significantly our LTV. It also explains how global UA with key partners amplified its impact and led us, in 4 months, to profitably:

  • 4x DAU
  • 4.5x revenue
  • 5x marketing spend while more profitable than before

Fastlane: Road to Revenge, an evolved mobile arcade shooter

Fastlane was launched in Mid 2017, a period of low risk growth and calculated bets for the studio – since then, we joined force with Supercell and are committed to make our mark on the gaming ecosystem, define a category hit and make a game that people will be talking about in 10 years time.

Despite, it not being our genre-defining game, Fastlane was, and is, a great learning ground for us in many aspects, including how to automate live-ops in a casual game, how to integrate 3rd party content from Youtubers to a Kasabian soundtrack, ad monetization and user acquisition.

I’m pleased to be able to share some of these lessons in this blog post.

Inspired by classic arcade shooters from the ’80s like Spy Hunter and 1942, Fastlane: Road to Revenge is a one-handed retro arcade shooter with RPG elements, designed to be played in short bursts. Players chase high scores in multiplayer leagues and leaderboards, collect, upgrade and customise exotic cars and unlock devastating vehicle transformations!

The game presents a huge motley crew of characters–many played by some of YouTube’s biggest gaming personalities–as well as powerful vehicle upgrades, outrageous events and fully customisable soundtrack with Apple Music integration.

From $5,000/d to $45,000/day from ads in 4 months: the lessons learnt

An iterative approach

The success we’ve had in the last few months on Fastlane is a result 6 months of iteration and experimentation by the dev and marketing teams working closely co-located.

Fastlane was not our first attempt at in-game ads.  We had included rewarded ad units in both Samurai Siege and Rival Kingdoms but in both cases the features were added post launch and not inserted into the core economy of the game and therefore were not additive.   

In Fastlane we committed from the beginning to design the economy specifically for ad monetisation. This involved being very clear that we would create value for both players and advertisers. This seems like common sense but previously our approach to in game ads was to just focus on the player experience. Of course no one is going to pay to advertise in your game if no players ever engage with the ads and ultimately install your advertisers’ (often a competitor) game. Once you start from the position that you want your players to tap on these ads then you approach ad unit design very differently.  Rather than focussing on how you can make the ad experience cause the minimal disruption to your gameplay, you focus on how you can ensure that once your players leave your game that they come back. This was a very different mindset and the fundamental reason why Fastlane’s ad implementation has been so successful.

It also resulted in the team implementing ads with the LTV components and the player happiness as our top concern. Making sure we are chasing the big picture, not only increasing a parameter (views) and decreasing other ones (retention, IAP) in the process.

That was all fine in theory, but initially it was merely a hypothesis so we tested in Beta. Below is the outcome of a test we ran in beta where we forced interstitial ads after every race. The result was pretty clear. It increased vastly the amount of interstitial views per day as well as ad revenue, but user retention dropped from day 7.  The overall result was negative as expected but it was a good exercise for us to go through as a studio and each subsequent hypothesis was tested in a similar way.

We were not ready to grow our short term revenue while hurting our long term retention. We made no compromise, canned that idea and tested some more.

We a/b tested different approaches to ads with a strict data driven approach and played with caps, frequencies, placements, formats and providers in order to end up with the design that you are seeing today.  6 months after the game’s global release we eventually found winning formula for that stage of the game’s life cycle. This was an optional rewarded ad at the end of almost every race, an interstitial showing up if you don’t make any IAP nor watch rewarded ad.

We also entered into an exclusivity partnership with Unity Ads in Dec-17 using their unified auction to monetise our entire inventory that has proven to be pivotal in our growth journey.

This new setup increased ads arpdau to $0.13 net in the US and $0.18 in CN while we more than tripled our scale with more than 2.5M daily video views globally.

Here is our current ad performance per main geo, in term of weekly views and CPM:

blog_cpms

(US and CN are leading both in video ads actual CPM payouts and weekly impressions)

In addition to significantly increasing ad arpdau, we were able to confidently see in the data that impact on retention and the IAP cannibalisation was more than offset by the increase in ad revenue.  LTV improved by 40% overall.

Fastlane’s arpdau – average revenue per daily active user – in the US:

blog_arpdauUS

(while there was a 15% cannibalisation of IAP, the increase we had gotten from ads led to a net increase of 40% of the overall arpdau)

Our 4 ad design pillars

We had 4 pillars that guided our ad design methodology on Fastlane.

Pillar 1 – Ads must work for the player. Rewards needs to feel desirable and part of the core loop, yet complementary to IAP bundles.

Pillar 2 – Ads must to be displayed at the right moment during the player’s session so that it does not impact negatively retention. In other words, show the ads when the player would be ending their session anyway.

Pillar 3 – Ads must to be part of the game’s world. They need to feel natural for the player. They need to add to the world.

Pillar 4 – The ad implementation must work for advertisers and drive installs. You should look at creating a placement where the players will want to interact with the ads in order to reach the highest CPMs possible, not necessarily the highest amount of views.  Culturally this was the hardest pillar to implement as it is counter intuitive to design to drive people to play competitor’s’ games!

blog_pillars(4 ad pillars that the Fastlane product team lived by during the ad implementation and experimentations)

Fastlane’s key findings

Here are our main findings specific to Fastlane:

» Rewarded ads > Interstitial ads for player retention and CPM

85% of our ad revenue is from rewarded ads and it does not hurt retention as the player has the choice.

» End of a race/session is the best placement for maximising CPM and player’s engagement with ads on Fastlane.

We want people interacting with the ads. In order to encourage that behaviour we found that a rewarded video at the end of a session generated 25% higher CPM than giving the option to watch an ad at the start of a session.

» Giving significant rewards to a player for watching an ad does increase the engagement rate

And it does not cannibalise IAP bundles if the economy is ready for it.  But your rewards must be set to levels that players would not otherwise buy with IAPs.

» Be upfront and unapologetic.

Watching ads is a clear value exchange and is part of the core aesthetic of the game. Not offering the option to pay to remove ads did work better for us. A Fastlane player should WANT ads. TV shows have been designed around ad breaks for years and our game is too as it’s the business model we’ve chosen from the start.

(Our ad implementation makes sure that it feels natural and enhance the brand and the game world)

A different approach than our previous titles

This methodology differed vastly from the approach we took for Samurai Siege and Rival Kingdom where the ad feature was added more than 6 months after the release of each game rather than designed with specific sinks and taps for ad rewarded currencies in mind. This resulted in the rewards being either insignificant or cannibalising IAP in strategy games.  Furthermore our strategy games had an aprdau of $1-2 from IAP, so the bar for in-game ads to be impactful in that economy was very high.

It should also be noted this lesson does not just apply to ads. The same is true for viral and social hooks that need to be designed as part of the core loop to have an impact.

An ad LTV model based on actual ad performance to gain accuracy for user acquisition media buying.

Understanding Ad LTV per campaign is arguably our big learning on ad monetization. It is so easy to make bad decisions by becoming fixated on one or other metric, when ultimately the only metric that matters is LTV.

LTV calculation has been pivotal since the mobile app industry moved to free-to-play.

As a marketer in today’s industry, chasing profitable ROI and growth via targeted paid campaigns is key. LTV is based on complex predictive models when it comes to IAP and developers have become quite sophisticated in predicting what a user will spend over their lifetime just by analysing their behaviour in the first few play sessions. Modelling LTV from IAPs at a user level is not trivial but it is well understood in 2018 and any game developer inherently has the data to do so because they need to associate a payment with a user account in order to deliver the relevant in-game item to the correct person.

However, when it comes to ad revenue, the ad LTV calculation is usually very basic. Historically we would crudely estimate how much revenue each individual user was generating from ads and this was fine because it was a very small part of our business. However today advertising is a $12M+/year business for us – only a small percentage of our overall revenue but significant enough that we would invest in understanding it more and adapt our UA to it.

At launch, our crude ad LTV approach was to simply divide a country’s ad revenue by the number of ad views in that country, and then apportion the revenue per user depending on the number of ads they watched. In the case of Fastlane in the US, ad LTV for the first 2 weeks with this method was between $0.40 and $0.64 per user depending on the source of the user.

blog_adLTVOne

However this is missing the point that most advertisers are bidding on performance, not on view, and that all views in a given country/platform are not necessary equal to another in term of revenue. We since moved from that model and are now attributing revenue based on the true ads performance, data that we’re receiving as part of our partnership with Unity. Which gives us an ad LTV between $0.23 and $0.73 for Fastlane in the US per UA channel. A much bigger spread.

blog_adLTVTwo

This was an eureka moment for the team and would allow us to tailor even more our UA bids to specific media that are bringing higher value users – like we’ve been doing for years with IAP.

We’re now tracking our ad monetization performance not only with the sole amount of views/user/placement in mind, but also taking into account the actions triggered after watching an ad by our users in order to maximise ad arpdau.

Next step, we scaled up UA based on this data to deliver supersonic growth.

In addition to getting better performing ad placements in-game, and improving LTV, we were also getting better at optimising our UA campaigns.  

During this time we managed to increased our oCVR% (installs/imps ratio) thanks to smart ASO and creative iteration from our in-house team and playable ads from partners that allowed us to reach much more scale at a reduced direct CPI of $0.52 from October onwards ($0.34 including organics).

oCVR% on our main video UA channel per month and per geo:

blog_oCVR

(oCVR% increased by 35-75% in the space of a few months depending on geos, which allowed us to scale, and improve ROI)

All these improvements in oCVR% and ad monetization led to the growth we’re seeing today, multiplied by the rocket fuel that is smart UA spend.

blog_ROI(Direct UA ROI has improved. Weekly UA investment was multiplied by 5x with our key ad partners. And it shows no sign of slowing down any time soon – quite the opposite actually)

We’re now spending north of $250k a week on marketing with our lean team of 2, growing week on week, while profitable under a month at a $0.52 CPI. We know more on Ad LTV per campaign, we have higher LTV, lower CPIs, long term partners, better creatives iterations, and there is no coming back.

The lessons we will be taking forward to our future games will be:

  1. Design your game with ads in mind from the beginning if that’s the business model you choose. Use the pillars that we used for Fastlane (or adapt for the new game) and in particular design moments when you can effectively push your players to interact with an ad. Your CPM and ultimately revenue will reflect that.
  2. AB test and track all the impacts of any changes to the in game ads implementation and focus your KPI on LTV improvements, not views.
  3. Attribute ad LTV precisely at a user level so you can target UA campaigns to the kind of people who will generate more revenue from interacting with ads

Devops at Scale: Videos

A big thank you to everyone who came to our offices to see our speakers last Wednesday night at the Devops at Scale event. A big thank you, also, to everyone involved in organising everything for the big night!

We’ve uploaded the videos of our talks, so if you weren’t able to come or are interested in what folk had to say, here they all are!

Steve Lowe: Devops at Scale: A Cultural Change

Sam Pointer: Smashing the Monolith for Fun and Profit: Telemetry-led Infrastructure at Hive

Louis McCormack: Monitoring at Scale

Upcoming Event: Devops at Scale

We’re working with Burns Sheehan, Wavefront and Hive to host an event at Space Ape HQ on Wednesday April 12th on the theme of Devops at Scale.

The evening will explore topics focussing on the adoption of DevOps at scale, hearing from businesses and individuals who have successfully driven these new DevOps approaches.

Richard Haigh and Steve Lowe will be speaking from Betfair, telling the story of their shift to DevOps and how pushing for attitudinal change drives effective DevOps implementation. From Hive, Sam Pointer will talk about how they’ve used a telemetry-first approach to break apart a monolithic application and implement infrastructure transformation at scale. Finally, Louis McCormack of Space Ape Games will take a look at the challenges of monitoring everything, when “everything” keeps changing.

You can learn more at the event page, where you can also sign up to attend.

DevOps-At-Scale-Invitation

Space Ape Live Ops Boot Camp – Part 2 (GDC Edition)

(This is the second in our series of posts on Live Ops.  Part 1 can be found here: https://tech.spaceapegames.com/2016/12/07/space-ape-live-ops-boot-camp/)

Last week at the annual Games Developers Conference (GDC) in San Francisco, Space Ape’s first product guy, Joe Raeburn, took to the stage to share what we’ve learned about Live Ops.

The full video will be available in the GDC Vault soon, but we thought we’d share a summary of the talk and annotated slides for those who can’t get access.

Apart from some quality trolling of Kiwis and puns of goats the point of the presentation was to help frame how studios should think about Live Ops.  Joe’s personal experience was formed at his previous company where he was a Product Manager on the hugely successful Sims Social.  That game rocketed to over 60m users in a matter of months but the studio paid a high price with 75% of the studio’s headcount consumed with running the game.  At Space Ape we nearly fell into the same trap with around the same number focussed on operating our live games in 2015.  

The solution:  we radically transformed how we develop and operate games to comply with Joe’s first commandment:

“LEAN COMMANDMENT 1: Thou Shalt Go Lean … or thine studio shalt be encumbered with unsustainable weight and die.”

Today, we are 100 people, with a little over one third of the studio working on live games that more than pay the bills, freeing up the majority of developers to work on new transformative games.

Joe’s talk shows how we quantify the impact of Lean Live Ops and how we designed systems to ensure that the most desirable content we wanted players to chase, was able to be produced cheaply without reliance on developers.   He also shows how we’ve carried this philosophy through to the design of our new games Super Karts and Fastlane: Road to Revenge.

The full slideshare presentation can be found here: https://www.slideshare.net/SimonHade1/lean-live-ops-free-your-devs-annotated-edition-joe-raeburn

Custom Inspec Resources

When developing Chef cookbooks, a good test suite is an invaluable ally. It confers the power of confidence, confidence to refactor code or add new functionality and be…confident that you haven’t broken anything.

But when deciding how to test cookbooks, there is a certain amount of choice. Test Kitchen is a given, there really is no competition. But then do you run unit tests, integration tests, or both? Do you use Chefspec, Serverspec or Inspec? At Spaceape we have settled on writing unit tests only where they make sense, and concentrating on integration tests: we want to test the final state of servers running our cookbooks rather than necessarily how they get there. Serverspec has traditionally been our framework of choice but, following the lead of the good folks at Chef, we’ve recently started using Inspec.

Inspec is the natural successor to Serverspec. We already use it to test for security compliance against the CIS rulebook, so it makes sense for us to try and converge onto one framework. As such we’ve been writing our own custom Inspec resources and, with it being a relatively new field, wanted to share our progress.

The particular resource we’ll describe here is used to test our in-house Redis cookbook, sag_redis. It is a rather complex cookbook that actually uses information stored in Consul to build out Redis farms that register themselves with a Sentinel cluster. We’ll forego all that complexity here and just concentrate on how we go about testing the end state.

In the following example, we’ll be using Test Kitchen with the kitchen-vagrant plugin.

Directory Structure:

Within our sag_redis cookbook, we’ll create an inspec profile. This is a set of files that describe what should be tested, and how. The directory structure of an inspec profile is hugely important, if you deviate even slightly then the tests will fail to run. The best way to ensure compliance is to use the Inspec CLI, which is bundled with later versions of the Chef DK.

Create a directory test/integration then run:

inspec init profile default

This will create an Inspec profile called ‘default’ consisting in a bunch of files, some of which can be unsentimentally culled (the example control.rb for instance). As a bare minimum, we need a structure that looks like this:

—test
└── integration
│       ├── default
│       │   ├── controls
│       │   ├── inspec.yml
│       │   └── libraries

The default inspec.yml will need to be changed, that should be self-evident. The controls directory will house our test specs, and the libraries directory is a good place to stick the custom resource we are about to write.

The Resource:

First, lets take a look at what an ‘ordinary’ Inspec matcher looks like:

describe user('redis') do
  it { should exist }
  its('uid') { should eq 1234 }
  its('gid') { should eq 1234 }
end

Fairly self-explanatory and readable (which incidentally was one of the original goals of the Inspec project). The purpose of writing a custom resource is to bury a certain amount of complexity in a library, and expose it in the DSL as something akin to the above.

The resource we’ll write will be used to confirm that on-disk Redis configuration is as we expect. It will parse the config file and provide methods to check each of the options contained therein. In DSL it should look something like this:

describe redis_config('my_redis_service') do
  its('port') { should eq(6382) }
  its('az') { should eq('us-east-1b') }
end

So, in the default/libraries directory, we’ll create a file called redis_config.rb with the following contents:

class RedisConfig < Inspec.resource(1)
  name 'redis_config'

  desc '
    Check Redis on-disk configuration.
  '

  example '
    describe redis_config('dummy_service_6') do
      its('port') { should eq('6382') }
      its('slave-priority') { should eq('69') }
    end
  '

  def initialize(service)
    @service = service
    @path = "/etc/redis/#{service}"
    @file = inspec.file(@path)

    begin
      @params = Hash[*@file.content.split("\n")
                           .reject{ |l| l =~ /^#/ or l =~ /^save/ }
                           .collect { |v| [ v.chomp.split ] }
                      .flatten]
        rescue StandardError
          return skip_resource "#{@file}: #{$!}"
      end
    end
  end

  def exists?
    @file.file?
  end

  def method_missing(name)
    @params[name.to_s]
  end

end

There’s a fair bit going on here.

The resource is initialised with a single parameter – the name of the Redis service under test. From this we derive the @path of the its on-disk configuration. We then use this @path to initialise another Inspec resource: @file.

Why do this, why not just use a common-or-garden ::File object and be done with it? There is a good reason, and this is important: the test is run on the host machine, not the guest. If we were to use ::File then Inspec would check the machine running Test Kitchen, not the VM being tested. By using the Inspec file resource, we ensure we are checking the file at the given path on the Vagrant VM.

The remainder of the initialize function is dedicated to parsing the on-disk Redis config into a hash (@params) of attribute:value pairs. The ‘save’ lines that configure bgsync snapshotting are unique in that they have more than one value after the parameter name, so we ignore them. If we wanted to test these options we’d need to write a separate function.

The exists? function acts on our Inspec file resource, returning a boolean. Through some Inspec DSL sleight-of-hand this allows us to use the matcher it { should exist } (or indeed it { should_not exist } ).

The final function delegates all missing methods to the @params hash, so we are able to reference the config options directly as ‘port’ or ‘slave-priority’, for instance.

The Controls:

In Inspec parlance, the controls are where we describe the tests we wish to run.

In the interests of keeping it simple, we’ll write a single test case in default/controls/redis_configure_spec.rb that looks like this:

describe redis_config(“leaderboard_service") do
  it { should exist }
  its('slave-priority') { should eq('50') }
  its('rdbcompression') { should eq('yes') }
  its('dbfilename') { should eq('leaderboard_service.rdb') }
end

The Test:

Now we just need to instruct Test Kitchen to actually run the test.

The .kitchen.yml file in the base of our sag_redis cookbook looks like this:

driver:
  name: vagrant
  require_chef_omnibus: 12.3.0
  provision: true
  vagrantfiles:
    - vagrant.rb

provisioner:
  name: chef_zero

verifier:
  name: inspec

platforms:
  - name: ubuntu-14.04
    driver:
      box: ubuntu64-ami
      customize:
        memory: 1024

suites:
  - name: default
    provisioner:
      client_rb:
        environment: test
    run_list:
      - role[sag_redis_default]

Obviously this is quite subjective, but the important points to note are that we set the verifier to be inspec and we provide the name: default to the particular suite we wish to test (recall that our Inspec profile is called ‘default’).

And thats it! Now we can just run kitchen test and our Inspec custom resource will check that our Redis services are configured as we expect.

Space Ape are hiring for Devops

Here on the Space Ape Devops Team, we’ve been busy building out the tech for our next generation of mobile games and now it’s time to bring some fresh faces onto the team to help continue our journey. If you’re a passionate technologist, Devops engineer or infrastructure wrangler then we’d love to hear from you.

Being a Devop at Space Ape is an important role. On our existing titles, you’ll be responsible for maintaining the quality of our players’ experience, working with the team to roll out new features and upgrades and finding new ways to optimise the stacks. On our new titles, you’ll be working with the development team to build out new stacks, solve new problems and prepare for big scale launches.

Along the way, you’ll learn how we use tools to build and update our stacks and roll them out without impacting our players and developers. You’ll also learn how we write those tools in Ruby, Angular and sometimes Go. Eventually, you’ll learn what it is that our teams need and start bringing fresh new ideas for how we can make things better; perhaps improving our containerisation platform, serverless workloads or the security of our platforms.

If you’re interested, have a poke round some of our other posts and drop your details in on our careers page where you can find out a bit more about the Devops role and the technology we use.