Making testing purposeful through a learning agenda

A Learning Agenda is a framework that organizes your testing efforts in a way that lets you improve your overall understanding of your customer and/or marketing efforts.  It’s a goal-oriented approach to testing that helps you make decisions about future campaigns, and develop better marketing strategies. It guides the marketer in identifying the right tests in order to achieve near and long-term objectives.

• A test plan or a test brief provides details for each specific test – the who, what, when and how of each test.
• A test calendar is a document that informs when a test is going to be executed.
• A Learning Agenda is the precursor to all of the above; it’s the strategic plan that informs why these tests should be performed.

The idea is to think strategically about what you’d like to learn.  By clarifying the areas where you want to deepen your knowledge, you can design tests that help you do just that.

Here are three key steps to follow when developing a Learning Agenda.

Explore

The first step is to make a list of things you want to learn about your customers.  Identify everything you want to know and pose it in a question or hypothesis format.  Consider the following questions as your jumping-off point:

• What have you always wanted to know about your customers?
• What information is currently missing?
• What assumptions would you like to confirm?

A great place to start is to reference your marketing goals and determine what knowledge will help you improve specific KPIs.  Of course, the list of “Things I Want to Learn” will vary based on company and industry but here are some questions we’ve used to determine testing:

• How can I use email to increase traffic to my company’s blog page?
• How can I get more press coverage for our blog?
• What subject matters and topics will increase engagement with my clients?
• What subject matters and topics will improve traction with our prospective clients?
• Hypothesis (assumption): The use of first name personalization in email subject lines will improve open rates.

Vet

Now that you have a list, the second step is to go through it and vet each question:

• Consider whether the answers will be useful in developing future strategy
• Consider whether the acquired knowledge/data will help execute against current marketing objectives
• De-prioritize or “move to parking lot” the ideas that do not meet above criteria

At this vetting stage, try to keep your final list of questions/assumptions short – 3 – 5 items at most.  This is necessary because there may be more than one way of testing to arrive at a conclusion, or there may be multiple test iterations necessary to get a satisfying answer to each of your learning goals.

Here is a sample of our marketing objectives for 2016:

1. Increase traffic to company site by 15% YoY
2. Improve website conversion by 10% YoY
3. Improve clients’ knowledge and understanding of  product offerings

And here is the list of the questions I’d like to answer. I’ve crossed out the ones that didn’t make the cut, and wrote notes around each one as to why or why not.

• How can I leverage email to increase traffic to my company’s blog page? – Keep. Traffic to blog = traffic to company site. Maps to my goals.
• How can I get more press coverage for our blog? – Does not map to my goals.  Move to “parking lot”.
• What subject matters and topics will increase engagement with my existing clients? – Deprioritize for now, as this doesn’t map directly to the website conversion goal.
• What subject matters and topics will improve traction with our prospective clients? – Keep. This maps to the goal of improving website conversions.
• Hypothesis (assumption): The use of first name personalization in subject lines will improve open rates – Keep because the test is simple.

Construct

Finally, identify the best tests (or surveys) which will answer key questions or confirm major assumptions.

First, reference your marketing calendar to prioritize your tests. Try to find the right content or campaign opportunity that fits the test criteria, not the other way around. This should keep the need for incremental new creative designs to a minimum.

To take the example list above, let’s pick this one and think it through: How can I leverage email to increase traffic to my company’s blog page?

You may want to break this down into more detailed questions that will provide more information:

1. Would a ‘new post’ alert drive more clicks?
2. Would a larger CTA button drive more clicks?
3. Could teaser copy from the blog post drive more incremental clicks?

Then, match the right tests which will best answer the questions:

Tests (in corresponding order):

1. Test a ‘new post’ alert with X% of the audience
2. A/B test for CTA button size
3. A/B test for copy

Once the Learning Agenda is set, it’s time to implement and document by:

• Developing a test calendar
• Creating detailed test briefs for each test, that inform specifics such as segment size, sample size, control group size, and test duration
• After each test, record the results right away in an ongoing test results log.
• Last but not least, review the results once a month or once a quarter to glean the actionable insights, then summarize your learnings and document them so you can inform future Learning Agendas.