The field experiment reduced Nocturnal’s advertising budget, but the field experiment could have been designed to mainta
Posted: Mon May 30, 2022 6:58 am
The field experiment reduced Nocturnal’s advertising budget, but
the field experiment could have been designed to maintain overall
advertising levels. Design a field experiment to test return on ad
spending without reducing total ad budget. To simplify your task,
design the field experiment to test only two ad channels—Google
Prospecting and Facebook Prospecting.
After consulting with Scott Clark about the learnings that would
be most beneficial to
Nocturnal, and after reviewing their current advertising outlays
across dozens of digital
and non-digital ad channels, Professors Larson and Dotson designed
the field
experiment with a focus on measurement of the following advertising
effects.
(1) Facebook and Google. These two ad platforms together compose
the majority of
online ad expenditures in the United States, both for Nocturnal and
for the market in
general. Because of the structure of ad and purchase tracking on
these platforms,
Facebook and Google increasingly claimed credit for the same
purchases. Learning
which platform was truly causal in producing Nocturnal’s sales
would greatly benefit
Nocturnal.
(2) Prospecting versus Retargeting. Prospecting ads are typically
shown to a broad
swath of internet users, while retargeting ads are shown only to
users who have visited
Nocturnal’s website on some past occasion. Because the purchase
cycle for a new
mattress is long, users might make several visits to the website
before deciding whether
to purchase a Nocturnal mattress. Nocturnal wanted to ensure that
they did not lose
these potential purchases to competitors, and thus advertised
intensively to them.
Nocturnal wanted to know whether this intensive investment in
retargeting ads was
worthwhile.
(3) Advertising Synergies and Saturation. If someone sees multiple
Nocturnal ads on
both Facebook and Google, what is the effect of this multi-source
ad exposure? Do
these ads have less effect because they more quickly result in
over-exposure? Or does
the same message on multiple platforms cause people to take notice,
and thus increase
their purchase interest? On another dimension, since the
retargeting pool is generated
from prospecting ads, does increased prospecting produce better
results from
retargeting?
These three considerations led the professors to design a field
experiment focused on
four channels: Google Prospecting, Google Retargeting, Facebook
Prospecting, and
Facebook Retargeting. By assigning DMAs to differing advertising
intensities on these
four channels, the field experiment could obtain estimates of the
effectiveness of both
Google and Facebook, of both prospecting and retargeting, and of
any synergies
between any two of those channels. The conditions of the experiment
were determined
by a factorial design.
The experiment was implemented on November 18, 2019 and
concluded on January
10, 2020.
Table 1. Field Experiment Design Google Google Facebook Prospecting Retargeting Prospecting 1 0% 0% 0% 2 50% 0% 0% 3 100% 50% 0% 0% 150% 50% 50% 50% 50% 100% 150% 50% 0% 100% 100% 50% 100% 100% 100% 50% 100% 0% 50% 0% 50% 50% 0% 100% 100% 0% 0% 100% 50% 50% 150% 50% 100% 150% 50% 0% 0% 100% 50% 0% 100% 100% 0% 100% 100% 100% 100% 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Facebook Retargeting 0% 50% 100% 150% 150% 100% 50% 0% 100% 150% 0% 50% 50% 0% 150% 100% 150% 0% 100%
Table 2. Realized Advertising Ratios (During/Prior) Google Google Facebook Facebook Prospecting Retargeting Prospecting Retargeting 1 4% 0% 8% 2 9% 46% 86% 5% 0% 51% 51% 0% 81% 115% 61% 117% 9% 45% 52% 56% 113% 116% 53% 91% 84% 10% 88% 114% 58% 43% 90% 132% 8% 80% 49% 117% 96% 11% 46% 0% 101% 43% 49% 0% 8% 74% 91% 8% 53% 14% 85% 58% 55% 44% 118% 59% 8% 87% 125% 53% 111% 12% 4% 122% 95% 44% 5% 118% 116% 66% 6% 123% 8% 86% 94% 114% 97% 69S7WN- 3 4 5 8 9 10 11 12 13 14 15 16 17 18 19
Table 3. Results of regression model of sales ratios as predicted by advertising ratios Estimate Std. Error T value P-value (Intercept) 0.888 0.035 25.62 3.67E-13 Google Prospecting 0.113 0.050 2.27 0.039 Google Retargeting 0.016 0.034 0.46 0.653 Facebook Prospecting 0.073 0.030 2.45 0.028 Facebook Retargeting 0.015 0.038 0.39 0.702 Table 4. Results of regression model on orders placed (ratio of during- to pre- experiment) as predicted by advertising ratios Estimate Std. Error T value P-value (Intercept) 0.992 0.026 38.56 1.29E-15 Google Prospecting 0.153 0.037 4.15 0.001 Google Retargeting -0.008 0.025 -0.31 0.760 Facebook Prospecting 0.091 0.022 4.08 0.001 Facebook Retargeting 0.033 0.028 1.19 0.253 Table 5. Results of regression model on monetary value of orders (ratio of during- to pre-experiment) as predicted by advertising ratios Estimate Std. Error T value P-value (Intercept) 0.897 0.023 39.78 8.40E-16 Google Prospecting -0.023 0.032 -0.72 0.481 Google Retargeting 0.019 0.022 0.85 0.411 Facebook Prospecting -0.007 0.020 -0.37 0.717 Facebook Retargeting -0.013 0.024 -0.53 0.607
Table 6. Analytics reporting from one DMA during the experiment period. This region received the full budget of advertising across all four channels. Region Totals Purchases 680 Revenue $633,867 Analytics Reporting Ad Spend Impressions Clicks CTR Purchases Revenue ROAS Google Prospecting $12,387 1,163,176 2,596 0.0022 11 $10,348 -0.16 Facebook 18,797 0.0069 24 $22,196 -0.39 Prospecting $36,096 2,715,797 Google 2,596 0.0023 54 $50,412 2.37 Retargeting $14,970 1,144,784 Facebook Retargeting $19,566 755,242 5,262 0.0070 88 $82,131 3.20
the field experiment could have been designed to maintain overall
advertising levels. Design a field experiment to test return on ad
spending without reducing total ad budget. To simplify your task,
design the field experiment to test only two ad channels—Google
Prospecting and Facebook Prospecting.
After consulting with Scott Clark about the learnings that would
be most beneficial to
Nocturnal, and after reviewing their current advertising outlays
across dozens of digital
and non-digital ad channels, Professors Larson and Dotson designed
the field
experiment with a focus on measurement of the following advertising
effects.
(1) Facebook and Google. These two ad platforms together compose
the majority of
online ad expenditures in the United States, both for Nocturnal and
for the market in
general. Because of the structure of ad and purchase tracking on
these platforms,
Facebook and Google increasingly claimed credit for the same
purchases. Learning
which platform was truly causal in producing Nocturnal’s sales
would greatly benefit
Nocturnal.
(2) Prospecting versus Retargeting. Prospecting ads are typically
shown to a broad
swath of internet users, while retargeting ads are shown only to
users who have visited
Nocturnal’s website on some past occasion. Because the purchase
cycle for a new
mattress is long, users might make several visits to the website
before deciding whether
to purchase a Nocturnal mattress. Nocturnal wanted to ensure that
they did not lose
these potential purchases to competitors, and thus advertised
intensively to them.
Nocturnal wanted to know whether this intensive investment in
retargeting ads was
worthwhile.
(3) Advertising Synergies and Saturation. If someone sees multiple
Nocturnal ads on
both Facebook and Google, what is the effect of this multi-source
ad exposure? Do
these ads have less effect because they more quickly result in
over-exposure? Or does
the same message on multiple platforms cause people to take notice,
and thus increase
their purchase interest? On another dimension, since the
retargeting pool is generated
from prospecting ads, does increased prospecting produce better
results from
retargeting?
These three considerations led the professors to design a field
experiment focused on
four channels: Google Prospecting, Google Retargeting, Facebook
Prospecting, and
Facebook Retargeting. By assigning DMAs to differing advertising
intensities on these
four channels, the field experiment could obtain estimates of the
effectiveness of both
Google and Facebook, of both prospecting and retargeting, and of
any synergies
between any two of those channels. The conditions of the experiment
were determined
by a factorial design.
The experiment was implemented on November 18, 2019 and
concluded on January
10, 2020.
- The Field Experiment Reduced Nocturnal S Advertising Budget But The Field Experiment Could Have Been Designed To Mainta 1 (24 KiB) Viewed 19 times
- The Field Experiment Reduced Nocturnal S Advertising Budget But The Field Experiment Could Have Been Designed To Mainta 2 (28.89 KiB) Viewed 19 times
- The Field Experiment Reduced Nocturnal S Advertising Budget But The Field Experiment Could Have Been Designed To Mainta 3 (33.04 KiB) Viewed 19 times
Table 2. Realized Advertising Ratios (During/Prior) Google Google Facebook Facebook Prospecting Retargeting Prospecting Retargeting 1 4% 0% 8% 2 9% 46% 86% 5% 0% 51% 51% 0% 81% 115% 61% 117% 9% 45% 52% 56% 113% 116% 53% 91% 84% 10% 88% 114% 58% 43% 90% 132% 8% 80% 49% 117% 96% 11% 46% 0% 101% 43% 49% 0% 8% 74% 91% 8% 53% 14% 85% 58% 55% 44% 118% 59% 8% 87% 125% 53% 111% 12% 4% 122% 95% 44% 5% 118% 116% 66% 6% 123% 8% 86% 94% 114% 97% 69S7WN- 3 4 5 8 9 10 11 12 13 14 15 16 17 18 19
Table 3. Results of regression model of sales ratios as predicted by advertising ratios Estimate Std. Error T value P-value (Intercept) 0.888 0.035 25.62 3.67E-13 Google Prospecting 0.113 0.050 2.27 0.039 Google Retargeting 0.016 0.034 0.46 0.653 Facebook Prospecting 0.073 0.030 2.45 0.028 Facebook Retargeting 0.015 0.038 0.39 0.702 Table 4. Results of regression model on orders placed (ratio of during- to pre- experiment) as predicted by advertising ratios Estimate Std. Error T value P-value (Intercept) 0.992 0.026 38.56 1.29E-15 Google Prospecting 0.153 0.037 4.15 0.001 Google Retargeting -0.008 0.025 -0.31 0.760 Facebook Prospecting 0.091 0.022 4.08 0.001 Facebook Retargeting 0.033 0.028 1.19 0.253 Table 5. Results of regression model on monetary value of orders (ratio of during- to pre-experiment) as predicted by advertising ratios Estimate Std. Error T value P-value (Intercept) 0.897 0.023 39.78 8.40E-16 Google Prospecting -0.023 0.032 -0.72 0.481 Google Retargeting 0.019 0.022 0.85 0.411 Facebook Prospecting -0.007 0.020 -0.37 0.717 Facebook Retargeting -0.013 0.024 -0.53 0.607
Table 6. Analytics reporting from one DMA during the experiment period. This region received the full budget of advertising across all four channels. Region Totals Purchases 680 Revenue $633,867 Analytics Reporting Ad Spend Impressions Clicks CTR Purchases Revenue ROAS Google Prospecting $12,387 1,163,176 2,596 0.0022 11 $10,348 -0.16 Facebook 18,797 0.0069 24 $22,196 -0.39 Prospecting $36,096 2,715,797 Google 2,596 0.0023 54 $50,412 2.37 Retargeting $14,970 1,144,784 Facebook Retargeting $19,566 755,242 5,262 0.0070 88 $82,131 3.20