Bing Ads Advertiser Science has come up with another webinar that focuses on keyword attribution explained with the ‘Six Degrees of Kevin Bacon’. Now if you are wondering much on how Kevin Bacon got termed here, you must go through this webinar for the explanation. Learn about what the approach is, how you can expand your keywords and how can set up your keyword assist experiment. Hear from the Bing Ads Science Team members Lars Hirch, Frances Donegan-Ryan, Lin Huang with MJ DePalma moderating the webcast.
You can view the webinar here.
Here’s the transcript for your reference:
MJ DePalma: Welcome everyone to the Bing Ads Webinar on the Kevin Bacon Approach to Keyword Attribution. My name is MJ DePalma and I’ll be your host today. This is the second webinar with our Advertiser Science theory. We are excited to share the findings and methods from the Advertiser Science team, so that you could achieve more with your advertising dollars. Before we begin, let’s review some housekeeping items to enhance your experience with us today. Our webinar console that you see is customizable on your site, where you can click the content widgets that you see on your screen and move them around or size them for your screen size.
You can expand your slide area by clicking on the ‘Maximize’ icon on the top right of the slide area or by dragging the bottom right corner of the slide area. If you have any technical difficulty, please click on the ‘Help’ widget. It has a ‘?’ icon and covers common technical issues. We recommend, if you can, to leave the widgets as they are so you have the ability to see all the great resources and Q & A at the same time. For Q & A, please feel free to input your questions as you have them, but for the most part, we will get to the list of questions at the end of the webinar, and try to answer as many as possible.
If we do have the time, we will definitely answer any unanswered questions with a blog post. We do capture all the questions, so don’t worry about that. You’ll see a ‘Resource’ widget in the upper right hand corner that we will refer to during the webinar. Go ahead now and download the ‘t-test’ Excel spreadsheet, so you’re ready to follow along later in the presentation and also for your convenience, the deck is already available for download there as well. Also there are links to stay in touch with us in the deck and there is also a Twitter widget that will enable you to tweet right from the webinar console with the #Bing Ads and #bingadswebinars which will appear in your upper right hand corner under ‘Resources’.
We also will provide all sort of handles of the speakers in the next slide. An on-demand version of the webcast will be available approximately 4 days after the webcast and can be accessed at the link at the bottom of all the slides in our deck, but also it’s the same link that went out for the registration, so you can share that with your colleagues who could not attend today. Last housekeeping items, we would love your participation in our survey at the end of the webinar, so that we can improve in any way possible and also our Advertiser Science team has a special question for you in the survey, so be sure to take a look at that, so that you can have direct influence on how you possibly might build the next future.
So let’s begin by introducing who we have on the webcast today. We have Lars Hirsch, Director of Advertiser Science with Bing Ads and you could see his Twitter handle there below, if you would like to tweet him. We also have guest speaking today, Lin Huang, who is responsible for the sixth keyword initiative, so it’s really cool to have him on the call today, and of course we have Frances Donegan-Ryan, better known as ‘SDR’ around the office, you know, and just really happy to have her participating in the conversation.
Frances: Hi everybody. Thanks for joining us.
MJ: And so, here is the agenda today and here’s what you will learn. We will go over what is the Kevin Bacon approach to keyword attribution. The Advertiser Science team’s keyword is just experiment results and how to find keyword assist data in your account and last, but not least, how to design your own keyword assist experiment. And so Frances, why don’t you start this off? What exactly is the Advertiser Science initiative?
Advertiser Science Overview:
Frances: Thanks MJ. Again, welcome everyone. We’re really excited to chat with you all today. Advertiser Science, so this is an awesome team that we have here at Bing Ads and their whole job essentially is to take a look at the Bing Ads auction and all the data that we can collect there. They do deep-dive into that data and they turn it around, experiment with it, test it, analyze it, brainstorm around it and essentially come up with ways that our advertisers can improve their Bing Ads campaign and then obviously, after doing all of that work, they share those results and those methods with us. Essentially because of, we’re really the brains on the team and a lot of the tools that we have in-house, we are able to take huge amount of data and analyze it in a way that assist individual campaign an account owner’s couldn’t do and so, one of the tests that they had done is look at keyword attribution and that’s what we are going to chat about today.
So they took a look at the data and at which keywords are assisting in conversions, so we are going to talk a lot about conversions, conversion methodology, but what we found or what they found were some really interesting insights into the keywords that are leading up to the conversion keywords. So whether they are keywords that are clicked on in ads as part of the consumer buying decision process or they are just ads and keywords that had impressions leading up to the conversions and they found some really interesting insights that we want to share with you so you can enhance your own campaign. So MJ, talk to us a little bit about why we are calling this the ‘Six degrees of Kevin Bacon” for anyone who doesn’t know what that is, tell us what it is?
MJ: Sure Frances. So why this analogy, ‘Six degrees of Kevin Bacon’? Well, thinking about this concept of keyword assist that the Advertiser Science team tackled, Frances and I were talking one late night and it occurred to us that it’s sort of like the game of ‘Six Degrees of Kevin Bacon” and we were kind of excited about that. And this is a diagram from a Wikipedia page on the concept that the game is based on, which is ‘Six degrees of separation.” So, as you can see, there is a Point A and a point B with six connections between and a plethora of paths from one to the other, but the idea is that you can get to the end result within six degrees or in Kevin’s case, you can link that actor Kevin to a movie, six movies away. So that’s the idea, is that six degrees of separation or you know, ‘The Kevin Bacon Approach to Keyword Attribution’, we thought that was kind of fun to look at it like that.
So how do we know Kevin? You know, just if you could allow us a little creative license with this analogy, how may we personally be connected to Kevin Bacon, so let’s take Frances. Frances knows me and I happen to be friends with a Broadway television movie actor Deirdre Lovejoy. All her friends like to call her ‘DD’ and she knows tons of people who know Kevin, which she also mentioned to me that she met Kevin once. So one, she is good in the grand scheme of things in paid search, if Kevin is a keyword that represents that last click to conversion, we would want to know DD since she leads to Kevin and knowing me wouldn’t hurt either since I know DD, but more compelling of a connection path is this second example. Frances, tell us about your connection to Kevin.
Frances: Okay, so I mean, now we might be reaching a little bit here, so just stick with us. So MJ knows me, I know David [Collyer], he is my best friend from high school and David [Collyer] is an actor and he is in the following which was or is the TV show that Kevin Bacon is currently on. So if you were trying to get, you know, Kevin Bacon is your converting keyword, you are trying to pitch him with [inaudible 08:06] or you want to invite him to a dinner party, knowing me is a really great insight because I know David and David is a direct link to Kevin. So I can get to Kevin in one step and understanding that helps all of you. Now that we are all friends, you know, we’re all probably Twitter friends or atleast webinar friends. Now you have a one-step to Kevin Bacon.
And so again, stick with me. Here is how this all relates back to paid search and keywords. The opportunity that you have here to get in step ahead of your competition or get enhanced and optimize your campaigns even further, is understanding all of these as distinct keywords. So what’s known as your converting keyword because in Bing, we use the attribution that we give is ‘Last Click’, you’re going to know what was the keyword that caused that conversion, whether it was, you know, a sale or signing up for a newsletter or watching a video, whatever your goal is.
You already know what that is and what you don’t know or what you might not be aware of, are all these distinct keywords, the different paths that searchers and customers use to get there and understanding what those are, so that you can take advantage of them, so that you can increase budget, so that you can create all new campaigns, focusing on those assisted keywords – that is the opportunity here, that’s what’s going to get you ahead of your competitors, if you are in a very highly competitive state and it’s what’s going to lead to more conversions down the road.
So that’s how Kevin Bacon applies to paid search. We hope that that was clear. Do you have any other questions, feel free to tweet me and again, MJ and I will probably have to stop doing late night brainstorm if we are going to keep making sense moving forward, but we thought this would be a fun way to bring these test to you. So attribution modeling, this is where the whole concept starts. I’m going to hand over to Lin. Lin, tell us a little bit more about attribution modeling and what your team did when you were looking at this experiment?
Lin: Yeah, absolutely, thank you, Frances. So, for those of you who were in the webinar last time for the brand queries as well, I think this also has a lot of attributions with that. So two of my years in the last six years working with the Bing Ads ad advertiser, so we have, get to the impression that the ad advertisers in general, they have a pretty good sense of what’s happening there, keywords that have been historically successful and they are able to kind of expand their campaigns based on those. But what about for things like, you know, their brand queries, for instance, those are things that they definitely pay attention to, but what about the things that, leading towards those success? So is there anything that they need to additionally kind of pay attention to?
So those are the things that we’re going to be focusing on today and we will kind of often get to more details around the actual method that we are using here, which gives you more ideas around how do you make sense of all those attribution keywords. For those of you who haven’t seen kind of very much involved in this whole conversion attribution thing, so what we mean, like idea, conversion attribution, that the clicks that are leading towards the eventual conversion, so in a lot of the analytics tool with the SEM providers, you can kind of definitely look at the clicks right before the eventual conversions and a lot of kind of platforms like Bing Ads, currently we have the capability to kind of let you understand what are the last clicks before your eventual conversions and we’re going to be moving a little bit kind of beyond that.
At the same time, we would like to know that in order to get all those insights about these keywords, you will first need kind of enroll in our UET, Universal Event Tracking, so that’s definitely kind of be a prerequisite for you to kind of, to get things started in here. So let me talk to you a little bit about how you want to enable the Universal Event Tracking or in short UET. So once UET tag is installed by the advertisers across their website, the tag reports user activity on the advertiser website to Bing Ads. Advertisers can then create conversion goals to step by which sub-set of user actions on the website qualify to be counted as conversions. So it can be, you know, competing in a survey or kind of successfully kind of get to the order submission page, things like that.
And then you start with Bing Ad Tracking System, the UET. UET is a mechanism for advertisers to report the user activities on their website to Bing Ads by installing one site wide pack. UET is a prerequisite for advertisers, you know, to get all those insights about the conversions and also the remarketing. So it’s not just limited to the, a topic that, about the conversion that we are going to be talking about today. It has a lot of applications in the other domains. So once UET is installed by the advertiser across their website, the tag reports user activities on the advertiser’s website to Bing Ads and UET must be installed for us to kind of go further in the latter slides when you guys are talking about those experiments and so just take some time to look at our ‘Resource’ link. There is going to be some more kind of information for you to get this started and set up.
MJ: So just to recap quickly Lin, UET, Universal Event Tracking is the Bing Ads conversion tracking tool, so unlike in Adwords, it’s just one piece of code and then you said you can set up multiple goals after you’ve enabled or put that code on your website. So you don’t have to go multiple pieces of code for different goals. It’s one; it carries across the full website and then you can set up all the different goals you care about, so this can be, as you were saying, you know, the different steps that you want the customer or a visitor to take on your site. But in order for us to or in order for you to be able to do this experiment today on keyword attribution, you must have the Universal Event Tracking already set up in your site. So that’s something you don’t have yet; definitely set it up as soon as you can. If you have access to your site, you can do it very quickly or else we will put you a webmaster to do that and once you have let it collect data for a couple of weeks, then you’ll be able to fully run this experiment.
And like Lin said, you’ll be able to do lots of other things like remarketing, and other, you know, awesome activities! It also gives you more information about how visitors are behaving on your site so it can help the site optimization, landing page optimization etc. So we have a question and a tip. When you install UET tag, do not forget to pause old campaign analytics of Bing. So that’s a great tip and thanks for [inaudible 16:17] posted that in the Q & A. That’s true, although most is, we sensed that, the old thing, campaign analytics, a few months ago, so they probably aren’t collecting any data any more. But yes, definitely make sure those things are paused or turned off once you get UET, which is our new system working.
MJ: And also we had a question from Michael: Can this code be installed through Google Tag Manager? And the answer is ‘Yes’.
Frances: Yeah, it thinks and speaks back and forth with Google Tag Manager. Okay, so back to you, Lin.
Lin: Yeah. Thanks, Frances. So like you just mentioned, this is just a screenshot of the Bing Ads UI and you can see how you can get things all set up and specify your goals, and enable your UET and for the detailed kind of information, again, I would like to refer you to the ‘Resource’ link in the webinar and you can get all those information in depth.
Frances: Yeah, it’s just, actually, if you just go: BingAds.com/UET, you’ll be able to get a ton of resource on that page and there is a fun video too that helps you do your set-up, step bty step.
Lin: So let me just dive in a little bit to the model that we’re going to be offering in here. So traditionally, I guess a lot of you guys have already had access to a lot of the attribution related insights. So here, on the market, there are a couple of things that are very prevalent. So things like you just pay attention to predominantly the last ads, that will click before your eventual conversions, that’s so called ‘The Last Ads Approach’, so that’s actually the same approach that which is currently as default being offered on Bing Ads and a lot of the other kind of SEM tools. And also you have probably seen some other kind of a rules ads like paying attention to that first ad in the session and then probably you can also choose to, if you want, and you have some kind of a knowledge about how things work out and you can choose to kind of a [inaudible 18:39] even ways across all the keywords that are kind of present along the path of the search funnel.
So there could be some even more kind of explicit ways of doing that. Like I said, if you are having historical success with that particular positions of the ad clicks, that are going to be very much associated with your eventual conversions, you can be then kind of paying a lot of attention to those, like you can do the positions, related ways; you can kind of design and define your role accordingly and you can also kind of believe that time is a kind of irrelevant dimension in there. You can design all those customized models in there.
Lars: I think it’s worth pointing out the more complicated model is not necessarily better, right? The goal here, the goal isn’t to redistribute value to upstream clicks or keywords. The goal here is to run a more efficient campaign, to get a high ROI, to get more conversions and it’s now clear that last click is not…it’s now clear that that is not a good model compared to these other… Compared to the other models, the more complex models are better. These are not causal models that will give you the actual attribution for the actual value of those clicks. This is just a hereistic model that’s the [inaudible 20:08] the order that different keywords were clicked. There is nothing causal about it. It might give you better results. It might not give you better results. And the interesting thing is, “How do you know if it gives you better results or not?” and we will get back to that later.
Lin: Yeah, Lars, I totally agree, in the sense that, if you kind of apply a certain set of rules that are sort of arbitrary, may not kind of generalize to a lot of the other scenarios, that should not necessarily work better than a very simple approach such as the last click ads. So what we propose here is actually a fundamentally different thing. So what we do is very flexible. We do not assume any rules beforehand. We will let the data to figure out the pattern in there, so we will dive into those details in the latter slides.
Frances: Great. So if we take a look at the analysis that you did, Lin, on your team, can you talk us through just at a very high level, what does ‘keyword assist’ mean? What were you looking for, when you started to study the attribution?
Lin: Yeah, absolutely. So Frances, I would actually like to kind of start this whole kind of introduction of the method keys with a very sort of, you know, like…example in here, so let’s take a look at the screen here, using this funnel. So let’s assume there is a user who is trying to complete the task of planning a vacation to Hawaii and let’s look at the things that she should also be thinking in order to make that task complete. So she should be thinking about making her tour booking and find the rental cars, and find the, to get [inaudible 22:01]. And if she eventually kind of converted on a keyword that is, let’s say ‘Kauai Vacation’, then she should also be kind of planning all those other things prior to that eventual conversion, so those are the possible signals that we factor into this model.
So on a high level, the model that we propose in the technical term is called the ‘Non-parametric Matching Model’. So what it does is, it’s very flexible frameworks to allow you to factoring all those possible signals prior to the eventual conversion. A very sharp distinction from the other operating peer on the market, is that it does not really just consider clicks per se; it also considers the other prior search pages, even if in those pages, there were no ad impressions like at all. So that’s kind of a very general framework, allowing you to kind of take in all those possible usable signals without leaving anyone out and then we will have the typical method, the Non-parametric Matching part to figure out the important part.
And then, like you can probably imagine that we pay a lot of attention to whether or not a particular path is kind of a very prevalent path that have been observed across multiple users. Let’s assume…kind of a super-duper kind of power user. She keeps searching the same terms like a million times, but that is not going to be the same, kind of a use case as kind of a million different users search on one single term, right? So we have definitely care a lot…I mean, specifically kind of consider the diversity in the user in terms of how we want to weigh those information. And in terms of the relatedness from the assist to the eventual conversion, we looked at the semantics, kind of the relatedness in terms of the word meaning and whether or not those queries are related at all and the timing is also another very important factor that we are taking into considerations.
If the assist is very much close to a eventual conversions, then we will be paying more attention to that and the latest great addition to the framework is allowing us to do the modeling top device, because let’s face it, now a lot of the users, they are kind of competing one task across multiple devices, right? They started a search on the PC and go to their mobile phones to continue doing that and eventually they complete the orders on the PC page again.
MJ: So just to recap for everyone listening in, what you just described is the framework that you use to conduct your experiment, and so this is in a framework that, you know, someone listening could go and do themselves. It’s quite complicated to take that huge amount of data. You need a lot of analyzing technology and power behind it, but what we wanted to make sure one understood was when we did that study and when Lin teamed at the study, these are all of the aspects that they looked at and for all the pieces of data that they collected and in order to kind of break it…like, previous understanding of what the search funnel looked….
Frances: Right. So this is an example we see on the screen right now, is a very sort of typical what people use to see as the search funnel.
Lin: Right, yeah. I think, Frances, what do you kind of have on the screen right here, is a very ideal role case, right? Yeah, so you expect kind of directly, you see everyone like starts from the initial query and they had a couple of intermediate queries and they eventually boomed. They ended up in a conversion and things like everything that is search along this line are kind of contributing towards the eventual conversion, things of like very important, as a very clean cut, so like the example that you see on the right hand side, like they start with their site search to Maui and eventually kind of ended up in conversion on the Kauai vacation. So what about the reality into real data, that we work on, things can be more fascinating.
MJ: This then, this image actually take a look at what really the search funnel, if you can even call it a funnel, which we will get into that as well, but what it actually looks like in real life? I mean, we all wish that customers just behaved calmly, rationally and normally and that that click funnel was very easy for us to follow in that slide, but as most of us are .., if you make users as well, I think if we are all real with ourselves, we know that that is not even how we behave.
Lin: Yeah.
MJ: So how do you then, how does this model helps us or help advertisers optimize their campaign better?
Lin: Yeah. So let’s do a reality check in here. So the screen that you see here is actually a slightly anonymized version of the real data that we’ve been working on and the branches that you see here refer to the individual user’s search path towards the same common destination, the same conversion on the particular department store coupon term and you can see the picture shows you three separate users like they start from the outside of the branch and move all the way into the eventual conversion, the same destination and they, each one of them go to different path; they search a couple of, you know, the department store related terms and they eventually end up in the same thing and the size of the bubble stands for the importance, according to the model of that particular assist keyword towards eventual conversion and the color of the bubble, as you can see, on the right hand side of the…indicates that we see both signals across different devices. It can be coming out of computer PC, the mobile-phones or the tablet in here. You know, what is the more interesting, what I mean by ‘Reality Check’, let’s also take a look at some seemingly kind of unusual terms in here.
Frances: Yeah. Here?
Lin: ‘Home Depot Brooklyn New York,’ so how is that related with the department store coupon? Things seem a little bit far fetched. So that’s actually one of the kind of the contribution of this model, so you know, if everything is kind of making sense and everything is kind of very intuitive, then probably the advertisers, they should already be paying some attention to that. Often times we somehow do not know as a researcher or as an analyst, analytical people, we do not really know the mechanism. I wish this one enters into the funnel, but as long as it makes a real contribution, we kind of give credit to that.
MJ: MJ here, Lin. Just kind of interesting, looking at all these terms and ‘Home Depot Brooklyn, New York’, I was thinking, “How could it really be related?” and I’m thinking, “May be spring and summer is approaching. People might be searching for a barbecue.” What do you think?
Lin: Yeah. I definitely…possible, right?
MJ: [Laughter] It is possible. And then one other quick clarifying question to you, Lin, is when you say these three branches represent three users, is there only three individual users or is three aggregated profiles of users?
Lin: In this particular visualization, they are actually three separate persons, but we will be showing you kind of a three thousand feet altitude view of the network view in the later slides and just keep aside and I would get all those things together, but here you can see as long as we can present that type of insight in front of some advertisers like MJ, you just kind of a possible explanation for that, you could be some other kind of a domain-specific knowledge that we don’t know, but the advertisers, they can make sense out of it and if we kind of offer them this type of insights, which they repeatedly did not necessarily pay attention to, that will broaden their kind of a horizon and then be able to focus on some ad campaigns, new ad campaigns that are, you know, going to be contributing to their business.
MJ: Great. So, like we’ve mentioned, it never…I mean, we wish it was a very simple, easy funnel, but it isn’t, so let me just mention that we, those were three individual users and the path they took to converting on that department stores site, and this is the very big picture of you of, you know, compilation of millions of searchers and users and what it actually looks like.
Lin: Yeah. So this is the kind of an aggregated view which I was alluding to like a minute ago and this, the branches that you see here are the dominant patterns, that have been aggregated across millions of users, that are kind of tracked on Bing Ads and those anonymous patterns that you see here in the graph, the way that you read those type of signals is going to be similar to the previous one. The only difference is those things do not correspond to the individual users any more. Those are all being aggregated across all the users that we track.
Lars: Yeah, Lin, I see there is a lot of brand terms here. As you alluded to before, there is a connection between this and the material we presented last month, where, I don’t know how many of you attended that, but we talked about the value of bidding on your brand terms, and here you can clearly see that brand terms upstream from the conversion query, contributes to increasing your potential to convert clicks.
Lin: Definitely a great callout, Lars. So a little bit kind of linking back to previous slide as well, so if let’s say, the ‘Home Depot’ term also appears here, that’s going to be a much much stronger story than kind of a coming out of a individual user story because if a whole depot ends up in here and we’re still talking about the department store coupon as the conversion and that will mean that ‘Home Depot’ is not just coming out of one [inaudible 33:05] credit kind of search path from one user; it actually can have many many users, that’s very credible to know, that the advertisers need to pay attention to.
MJ: Great. So just to kind of tie this all together, when you look at all of this data, you know, just kind of give us a recap and tell us how these assist keywords, you know, may have lower impressions, you may not be bidding on them, but they are doing their job.
Lin: Yeah, so I think overall, just to recap, this particular method that we’re offering in here, this will allow the advertisers to do a lot of new things which they previously were not capable of doing, so first abstract that I can think about is they allow you to do the keyword expansion in a very unique way. I guess at this point of time, a lot of advertisers bid like, really savvy about, do various kinds of a keyword expansion, like the ‘Broad Match Adoption’ and those type of things.
At the same time, this is a new type of a keyword expansion based on how much assisting work a particular keyword is helping in the eventual conversion and doing that allows you to kind of move towards some unexplored territories which we previously kind of ignored, but actually are very potent and at the same time, even if you kind of already did some work in kind of looking at some additional terms that could be possibly helping your business, and this promo model will allow you to make more sense out of your choice.
What I mean here is from the model, like the weight, the size of the bubble, things like that, it will allow you to understand which bidded keywords are more important than others, then you should find your bid according to the weight coming out of this model. So overall I think this is going to be a really competent kind of addition to your kind of whole rep. tour of doing the data analysis to understand what could be the things that are contributing to your conversion.
Frances: Great. So, Lin, thank you so much. So Lin basically just went over the model and that his team designed and all of the data that they were able to pull and analyze and we are able to pull this data for some of our larger advertisers, so if you have a Bing Ads account team, and you have a Bing Ads account manager, definitely chat with them because Lin’s team might be able to do this model specifically for your account and your data. Just chat with your account manager and they can get in touch with Lin’s team and see if this is something we can work on.
If however you don’t have a team or you want to try and do some of this on your own first, that’s what we’re going to go over now and now the way that you can do this on your own, unfortunately isn’t as sophisticated as what Lin’s team can do in terms of the breadth and scope, but you can still find some really interesting data and just be smarter about your assist keywords and how to optimize around them. So the first thing that you are going to want to do is get the report, so if you log into your Bing Ads account and you click on the ‘Reports’ tab, you can run your report for keywords. It’s just called the ‘Keyword Report’ and within that report, you can turn on ‘Assist’ as a column and that will help you and that will pull all of your assist data.
And so once you run the report, this is what it’s going to look like. You’re going to have an Excel document with keywords in one column, the convergence of those keywords so that’s the last click conversion, but it’s also going to tell you for that, you know, keyword number one, it had 220 conversions, but it also had 71 assists, so that’s quite a powerful keyword for you. Another example to look at number 10 – now this keyword, you might delete or turn off or ignore because it’s only driving 2 conversions, but it is driving almost 30 assist, and so this report is going to show you what keywords are important, what ones you might have been previously knowing, that actually are quite powerful.
And now Lars is going to help me explain to you how do you actually do that analysis to see which ones are going to be the ones that you want to spend time and money on. Now, I will hand it over to Lars. He will tell you how to set up your campaign, how to set up this experiment to do on your own, that will give you atleast a little bit of insight into which assisting keywords are going to be the most valuable for you.
Lars: Thank you Frances, so what you want to do is, get that report that Frances just went through and download your assist keywords, and then you want to group them in some sort of manner that makes sense to you because you want to create individual campaigns that only have assist keywords and if you have 1000 or 10,000 assist keywords, you don’t necessarily want 1000 or 10,000 campaigns, so you want to create groups that are manageable and makes sense. So for example think about the related keywords, similar keywords with those in the same campaign and once you do that, you pause those keywords on existing campaigns while you’re running the experiment. And once you have run your experiments, you have to run, collect data, analyze certain slides.
Frances: So once you do that, so you have collected your data, you have run your report, you have paused the assist keywords in any existing campaign, so that you can set up a campaign just for those assisted keywords, this is, Lars, you said, of a controlled environment, where you can see which assisted keywords are more helpful than others and then you need to let, you know, that campaign run for a couple of weeks and then you can collect enough data and then Lars is going to explain to you, once you have that data, how do you determine which keywords are statistically relevant, ones you should be paying attention to.
Lars: Yeah. So you want to use this student ‘t-test for this slide if you get increase in the metrics you care about, that could be conversions or that could be profit. You want to use the student ‘t-test’ to determine if there is actually a statistical significance difference. So we have some material to help you with the ‘t-test’, but beyond that you need to have access to the data that you want to analyze and there could be different types of metrics that you want to look at and so that would make sense, if our number of conversions, the total number of conversions that you get, you get an increase if you have assist keywords enabled compared to your assist keywords disabled as well as the profit that you get, like the value that you get on PL campaigns minus the cost of those campaigns.
So the value, you could find that in the templates but hopefully you have a sense of the value per conversion that you get. You can also see that value directed to UET dynamically. So if you are a retailer, you know your revenue, you want to know your profit margin, you can see that back through UET and you can get that report about from the Bing Ads backend as to actual volume of those conversions. And the [total] profit is the value minus the cost for those campaigns. One more thing to mention, you want to be able to geo-target because when we are going to run this experiment, we are going to do that. We are going to look at a set of DMAs where we go into ‘enable’ their assist keywords and we’re going to have a certain DMAs where we are going to ‘disable’ those assist keywords and compare those against each other.
Frances: So this is just, you know, like you would run any experiment, you need your control group and then you need the group where you are doing something differently, so that you can compare the two and see right which is going to be most important than which one’s are…
Lars: That’s right. Then there is also the ‘t-test’ Frances, we’re going to get back to that. We have a dozen templates that you can use so you don’t have to actually create the formulas by hand, so we have some templates, you can always get back to that in the later slide, but first of all, you need to identify pairs of network areas with similar customer profiles for your business, so you think about what makes sense to you, right? It could be based upon demographical composition, it could be based upon weather, right? If you sell surfing gear, you don’t want to compare…you don’t want to create the pair that’s like one is san Diego and another one is in [inaudible 42:26], right?
You want to pair these in a way that you have two and two metro areas that are very very similar, with regards to how the customers interact with your business in those metro areas and this slide just starts with as many as you can identify, upto 20. I would say 20 is probably minimum. If you have 30, it’s even better – 30 pairs of metro areas and then for each pair, so these are pairs that are similar, right? Two metro areas that are similar with regards to their customer profile for your business, but in this study, you randomly select one metro area into what we call ‘Treatment’ and the other one into ‘Control’. The ‘Treatment’ metro areas will get the assist keywords campaigns enabled.
MJ: So related to that, MJ here, Lars, a question for you that came in from Dan is: Should we limit the number of keywords in a campaign to 10 or 20 words as opposed to a 100, when we are doing experiment like this?
Lars: So we don’t have to do that. You can have 100’s of keywords in a campaign. It depends how granular you want to be, so what you really will get is, the more granular you go, the more information you get about that set of keywords, right? On the other hand, the larger you make the campaigns, the bigger effect you get from that campaign, so it gets easier to see if that campaign actually contributed positively or not. So there is pros and cons there.
MJ: Thanks Lars.
Lars: I mean, if you want to, you can run, you can put everything in the same campaign and that will still work…, but then you will get an answer for that whole set of assist keywords, contributing positively to the overall set of campaigns that you are running.
MJ: So in other words, if I am listening correctly is, if you want to isolate a few keywords and really know if they are affecting your campaigns, you don’t want to do hundreds of keywords. You would want to kind of be reasonable about it, so that you can actually get a better view.
Lars: Right.
MJ: Got it, thanks. Hope that helps you, Dan!
Lars: Okay, so once you have that set up, you want to measure the conversions by metro area before experiments in the same, with a large [inaudible 44:54]use, the profit, as the metric that you care about. You probably want to do both, right? Make sure that it is disabled, the assist keywords beforehand. You run this for a couple of weeks with the assist keywords disabled, so you’re not running those in either ‘Treatment’ or ‘Control’ metro areas. One thing to keep in mind though is that with regards to the conversion KPI, once you get your results, you will probably want to exclude the campaigns that have the assist keywords from the total number of conversions that you are counting because you add up assist keywords, right? You add assist keywords and by itself they are going to contribute to some conversions most likely.
MJ: Yeah.
Lars: And if you only are interested in how these can drive in conversions on down funnel keywords, you probably want to exclude the account with conversions from those campaigns, probably not with the initial value though. That’s a little bit different, right, because we have a cost, we have a value associated and you want to look at the total value that can drive in fulfilled campaigns. Then you want to count whole set of value from the assist keywords. I would see that once we get a little bit further on here, but you want to measure conversions by metro area before experiments, you create, then enable the geo-target of the assist keywords campaigns for the ‘Treatment’ DMAs, for the ‘Treatment’ metro areas and then you measure conversions by metro areas during the experiments, you know, before and then you have during the experiment.
MJ: And so once you’ve collected all this data, you would then run the same report as you ran previously, right? You download your…after you have run the experiment for a couple of weeks, you go back, download your keywords report, [inaudible 46:53].
Lars: That’s right. So you want to have it in the free period, which is before you enable the experiment, so step five here is enabling the experiment, so you spot your geo-targeted assist keyword campaigns for specific, only for the ‘Treatment’ metro areas.
MJ: Yeah, the ones you have nominated.
Lars: Yeah. Before they are totally disabled. You enable, then you run it the same time period after, so if you run it…you might want to look at two weeks before, two weeks after or four weeks before, four weeks after – those are probably reasonable amounts of times to look at.
MJ: And so, then, once you are having your data…
Lars: Yes, then we have, as you can see, you have these [assets] here, then the ‘Resource’ List, that you can download the ‘t-test’ plus a few customer details in the next slide, right? Yeah, so here you have three sets of column. You have the left set, the middle set and the right hand set, right? A little left hand set, you created them, for example, with number of conversions and you can do this… you can do the same things thinking about the value of the profit metrics we talked about, so let’s just speak the conversions for now, but it would work exactly the same way with the profit metric. You input the number of conversions that you have in the ‘Treatment’ metro areas which is the left ‘TO’ column and the ‘Control’ metro areas which is the right ‘TO’ column in that set of columns. That makes sense?
So the first step on the columns to the left, the first thing you want is the ‘Treatment’ metro areas, the number of conversions weighing there for each ‘Treatment’ metro area; then you have that paired up with that ‘Control’ metro area, those are the pairs, right? Each row has a pair. A pair metro area that’s similar, like we talked about previously, similar in composition with regards to how do you think about your customers. So your left one, you have the ‘Treatment’, the number of conversions in the ‘Treatment’ metro areas, the right one you have the number of conversions in the ‘Control’ metro areas and that’s before you start the experiment, two or four weeks before.
Then you start to experiment, then you have the next set of columns, the middle one. You do the same thing, but for that time period during the experiment, when you have your assist keyword campaigns enabled. Again, ‘Treatment’ and ‘Control’, left and right, you have the same set of metro areas that you have in the previous set of columns. Finally, in the right hand side, you get the calculated list. That is the calculated list; you don’t input anything there.
Those cells are already pre-populated with the right formulas to calculate the list, the list that you got from running those assist keywords and the ‘Treatment’ is the left one, ‘Control’ is the right one and then now you want to compare the list in ‘Treatment’ versus ‘Control’ and that’s what this ‘t-test’ does, which you have, which, again is automatically populated in that little black trained area, below these boxes.
Frances: So you’re looking for the formulas that are already built in there, are basically going to calculate your ‘t-test’ results, and then you just want to look at the ones that are statistically significant, so above the…whatever the ‘t-test’ that is your rate.
Lars: Yeah, so this is the overall ‘t-test’ results for this whole experiment. So, it’s not like you pick certain metro areas. Based upon the ‘t-test’ you can see if there is a statistically significant increase in whatever metric you care about, for example, the conversions in this example, by running the assist keywords. Now keep in mind, for conversions specifically, I would exclude the number of conversions like when you run the experiment, so I would exclude the conversions from the actual assist campaign for the reasons that I talked about previously, that they will probably drive some conversions by themselves and what you are probably interested in is just looking at the contribution of these keywords in driving conversions on other keywords, right? If that’s not the case, then you can include it. But if you’re only interested in looking at the assist power, they you want to exclude the actual conversions from those campaigns.
Frances: Yeah, so in this example, you can see that the average click increase was 6%, which is statistically relevant, so essentially overall this ‘t-test’ has shown you when you enabled these assist keywords, you were able to increase either the value of the conversions or whatever metric you’ve decided, you’ve been able to increase them significantly, and so the experiment shows you that those assist keywords are valuable and worth investing in or worth optimizing further.
Lars: Yeah. So the actual ‘t-test’ number that you get in this example, it looks like 0.04 – that basically means that there is only a 4% chance that these numbers are actually the same, but because of random distribution, you get some difference. But they are not really from a different set up; they are not really from a different distribution. You generally want that number to be lower than 0.05, and that’s what’s like a generally accepted threshold.
Frances: Okay.
Lars: Another thing, well, but with 95% certainty, these are actually different results in the ‘Treatment’ and ‘Control’.
Frances: Great. So let’s recap. MJ, I’m going to hand it over to you to pull a recap, and then we’ll have about five minutes to answer any of the unanswered questions. I know MJ has been answering questions as we’ve been chatting, but MJ, put this all together for us.
MJ: So I really wanted to make a reference to soccer as we did in the last webcast. If you listened to the science of brand bidding, we pondered the concept of how in sports, “Offence wins games but defence wins championships,” and so, in the spirit of that, I wanted to welcome you to ponder a very important concept with your keyword strategy as you listen throughout this webcast. Abby Wambach – she actually is the lead scorer in all of soccer; whether it’s men or women, she is the world’s leading in scoring and she says, “I’ve never scored a goal without getting a pass from someone else.” So that assist in soccer is super-important. So I just wanted to point a little, you know.
Frances: I like that. I love that. [Laughter]
MJ: But to close things down, let’s recap on the next slide there. Just make sure you take advantage of the keyword reporting tab. In the ‘Reports’ tab, there is a keywords report and it’s called ‘Assist’, so definitely download that report and make sure you set up a separate campaign when you’re going to be able to statistically test with the way that Lars had explained in the ‘t-test’, to make sure if that, you know, data is statistically relevant or not. Definitely have patience, because it may take a little bit of time and effort to run through the whole process but definitely dividends can pay off if you do so.
And then, you know, just the whole idea of making data-driven decisions is really what this entire industry is about, and that’s what makes us really super-excited, that we can pull these types of levers. And so I really welcome you to the…so let’s go into the Q & A. Of course, we do have some questions that we have left unanswered. One of the questions I know, it seems you touched on Lars, is how long should the testing period be? I know it’s kind of …but what would you start out with and then how many times, if we don’t get statistically relevant results, how far out would you go and then go all-quality day?
Lars: So there is not really a ‘one size fits all’ answer to that. You can think about also the length of your process funnel, like certain industries, process funnel could be months, right? For other industries, it would be very very short, and the longer your process funnel, is the longer you want to go, because you want to be able to capture those downstream conversions in a reasonable way. So I would say if your process funnel is really short, you might want to look at three to four weeks; if you don’t have significant results by that time, it might be because you didn’t had…you didn’t do it for long enough, but it could also be that those keywords are not really that valuable, right? You don’t really know. Only if you get a positive result, you can know for sure that it’s actually positive. If you get a negative result, it could be either because, well, there isn’t any effect or that you didn’t convert at the long term period, but for a short process funnel industry, probably two to four weeks is reasonable.
MJ: Great. Thanks very much. Awesome! We have another question from John, so not sure how far back this data goes and he mentions that they do have UET installed for quite some time now. What that means is that they could see the assisted data from, say, a year ago and the answer to that, I think, Frances is actually taking a quick look. We want to make sure we answer this accurately. I think it really depends on how you set up your UET and how far…I believe it will probably be a 180 days because a 180 days is how long that cookie would last, based on UETs, that’s as far back as they will go, so we’re at this moment, is six months is the answer, but we’ll double-check in on that. Thanks so much John.
Also for another question there around bidding on competitor’s brand terms. So we covered a little bit of…we’ve covered a lot of that in the previous webinar, but I saw a couple of questions come up. You are allowed to bid on your competitor’s brand terms. You’re allowed to bid on their brand name, you are allowed to bid on the brand term. Now this does vary by market, but in the US, in Canada, you cannot use your competitor’s name in your ad copy or in, say, your ad title.
But you are allowed to bid on them as like a competitor keywords strategy, so I saw a handful of questions there, people thinking you can’t do that. In the US and Canada, you can; there is a link on their blog to all of the editorial policies and if you’re in a different market, definitely double-check that because in some markets, it’s not allowed, but in the majority of people listening into there are local US, Canada and you can do that.
Lars: And…people do that a lot as well! May be especially small advertisers bidding on larger advertiser’s terms, but also lot of advertisers bidding on each other’s terms as well!
MJ: This next question, I believe, is referencing Lin’s study and it’s from Steven: Are these based on strictly on Bing searches or Yahoo and AOL as well when you did your study?
Lin: Steven, this is a great question. The study that I was showcasing you a while ago, that was based on the entire Bing network data, which will kind of span across both the Yahoo, Bing and also the AoL numbers.
MJ: Okay, great. Well, we want to respect everyone’s time. That’s all we have time for questions, but I do want to invite you guys to join us in our next webinar which is on March 15th at 11 am. I will be having Eric Couch as a guest speaker and he’s going to go into the ‘Science of Excel’ for PPC marketers, really cool tips, tricks, plug-ins and formulas and nested formulas and super-powerful stuff. If you are anyone listening, I highly encourage you to join. It’s been pretty powerful when we’ve been delivering this in person to customers, so there is a link to register right away. Again, March 15th at 11 am. Thanks everyone for joining. The next slide is just stay in touch with us, tweet any follow-up questions. Frances is passionate about that, so feel free to do that and then last but not the least, another way to stay in touch with us is tweet: #AskBingAds. Thanks everyone. Until next time, have some great campaign!
Frances: Thank you.
Lars: Thanks.
Lin: Thanks.
Related Articles:
Stop the wasted ad spend. Get more conversions from the same ad budget.
Our customers save over $16 Million per year on Google and Amazon Ads.