Metrics That Matter with Jim Sterne

Transcript

Hello, and welcome to Marketing Speak! I’m your host, Stephan Spencer, and today, I have the distinct pleasure of inviting Jim Sterne onto the show. Jim and I have known each other for, literally, decades. I remember in 1997, we co-presented a workshop together at Internet World UK and that that really dates us. However, I think we met in ’96, maybe even ’95.

That’s quite possible.

Yeah. Around that time was when you came out with your first book so you have ten books under your belt.

I do.

Jim has ten books on internet interactive marketing. His latest is The Devil’s Data Dictionary. He is a guru on metrics and anything that has to do with making smart decisions with data-he’s your guy. He’s an international consultant. He focuses on measuring the success of digital relationships with customers. He’s the founding president and current board chair of the Digital Analytics Association, formally known as the Web Analytics Association. Jim produces the eMetrics Summit, which I’ve spoken at before. It’s an incredibly fantastic event-definitely worth your time. He is just an all-around really good guy and an absolute guru in pretty much anything related to internet marketing. He’s probably written a book about it so welcome, Jim! It’s great to have you!

Thank you. After that, all I can say is, “Aww, shucks!”

Exactly!

And what a fun time we had way back when in those workshops. Because when you and I first met, it was sort of like, “Well, here’s an expert in the field and I’m going to sit down and learn from him and then I’m going to stand up and try to come up with something else to say,” and I remember, consistently, we were in violent agreement about everything. I think our audiences were kind of upset because they’d ask a question and one of us would answer and the other one would say, “Yeah, what he said!”

Yeah, yeah. Good times!

Yeah.

You’ve become “the” guy when it comes to metrics and analytics so let’s start there. Obviously, you’re just the guy…

I’m that guy.

Yeah. Let’s start with-what are the most important things that people should be measuring? What are the KPI’s that really make a difference and not the kind of “red herrings”?

The only correct answer to that is, it depends. Sorry, but it’s true.

Please elaborate.

In fact, it’s probably the most important thing to keep in mind because when you open up Google Analytics or any other tool, there is a splash page with charts and graphs. You look at those first charts and graphs, you go, “Oh, well, those must be the most important things since they put it on the front page,” and the answer is: No, that is absolutely not true. How many people came to your website today is actually not important at all. What’s important is, who are they; why are they there; are they being successful at what they’re trying to accomplish; are you being successful at what you’re trying to accomplish-and that’s where all of the “It depends,” comes into play. Here’s a whole bunch of data, what problem are you solving for? If I wanted to write a novel, I could just go to a dictionary and pull out a bunch of words-hmm, no! The industry problem we face is that the tools we’ve been given have default answers to questions we have not asked and we assume those are important questions, but they’re not so the important questions, the “it depends” part, and the KPI part is, what matters to you and that is immediately a political question. What is the goal of the organization? What is the hot button of the committee that’s holding reign over your budget? What is your immediate boss’ compensation package? These are the things that matter in what do you measure so if I’m running my own little website or, let’s say, I have a blog and I’m making a living selling ads on my blog, what do I care about? Oh, I care about page views. The more page views I have, the more real estate I can have, and the more inventory to sell ads against and if that’s all I care about, I’m going to end up writing one blog post that is one paragraph per page times twenty pages so you have to keep hitting next and in about ten minutes, all of your audience will leave you behind because that’s just too frustrating to consume.

Oh, I totally love those slide shows. I wish they would make like, 50 tips or 50 fun things or whatever, and put each one on a slide-I’m just so excited when I see that!

Especially when they take five to ten seconds for each page to load because there’s fifty ads on each page! That’s just so cool!

Yup! Love it!

The “it depends” really is the crux of it and that goes all the way up from the simplest, “Well, why do you have a website?” to “What are your corporate goals?” to “Why are you investing in this new technology?” and “Why are you in it?” That will tell you what you might measure and that will tell you which data is important to collect.

That makes a lot of sense. This reminds me of a book I read. It’s one of my favorite business books-other than yours, of course.

Aww, shucks!

That was Freakonomics. Freakonomics is all about figuring out what the incentives are that are baked into the system and aligning with those-or at least understanding them-because if you don’t understand them, you just don’t see things that are in plain sight like, teacher cheating in a Chicago public school system and things of that nature. Fascinating stuff! The compensation metrics of your boss is critically important because if you’re misaligned with what his or her incentives are, you’re off on a completely different track and you’re not going to last long in the organization.

If I’m a manager and you walk into my office with a grass, a chart, and a spreadsheet and you go, “Oh, look at this really cool correlation I found,” you just made me responsible for those numbers. You just said, “Hey, there’s something really important here. Isn’t that amazing! Don’t you see it?” and now, I have to look at the numbers and do your job, which is analyze it in real time while you’re hovering over me with this excited grin on your face and I now have to question, “Well, where did this data come from? How was it collected? How was it cleaned? How was it combined with other data? What assumptions are baked into it?” and you’ve put me at a disadvantage because I’m your boss and I hired you to do the all of that for me. What I want you to do is to come into my office and go, “I was looking at the data and I saw an interesting thing. It looks like whenever this happens, these are the results, and I’m thinking, maybe, if we change how this happens, we can increase our results,” and my response would be, “That’s pretty insightful. Can you think of an inexpensive test we could run to check that out?” and you would say, “Oh yeah, I could do that. I’ll be right back!” Now, we have an adult conversation and I’m not made to look like a fool for not being able to read that spreadsheet because I’m not a numbers guy. I’m a manager. I’m a boss. My job is to have six people come in and demand things of me and decide which of those demands makes the most sense. If the demand is, “Come and look at how hard I’ve worked at collecting this data and there are really interesting things I found inside on my way to get to the insight,” we’re having the wrong conversation. If you walk in and say, “Here’s an insight that I came up with,” and my experience with you is that your insights are pretty spot-on, I trust you to be able to handle the data, you’ve only been wrong a few times and hey, you’re human so that’s fine, but you didn’t impose and it’s not like you walked in with War and Peace in Russian and said, “Check out page 93!” I don’t read Russian. Tell me the insight and then I can decide whether that’s going to meet my personal goals for getting my bonus, my boss’s goals, the corporate goals, and if I can squeeze it in with all the other priorities I’m juggling. My favorite analogy is the doctor. When I walk into the doctor’s office and they say, “Well, I’ve got your blood test results here.” I don’t want him to just hand them to me and say, “What do you think?” I didn’t go to medical school! I don’t know! “Do I need surgery? Do I need to stop eating sugar? Do I need to get more exercise? Can I just take this pill?” If the answer is, “Well, yeah. Stop eating sugar and get more exercise and take this pill! “Great, doc. Thanks!” If the answer is, “Exercise, no sugar, and you need surgery,” I’m going to get a second opinion. “I trust you, but you’re talking about cutting me open. This is a serious investment. I want another opinion, but I am not going to go to medical school.” So, what’s the most important thing to measure? Well, that depends on what’s on my boss’ mind today.

Yeah, and this reminds me of another episode of Marketing Speak just recently from a few weeks ago. Aaron Ross-I interviewed him. He wrote From Impossible to Inevitable, which is a great book and a brand new book. There’s a whole section about the employee-boss relationship and how most employees are renting their jobs like you rent a car-you don’t treat it as if you own the car or you own the house…

Interesting.

There’s this amazing letter that the employee writes to his or her boss and it’s just spot on and then the employer writing back to the employee in a completely different perspective is like, same planet, but different worlds. And this is exactly what we’re talking about right now so if we can really nail this, we’re going to have a high-performing organization, not just great metrics that matter, but have a high-performing business, and that’s I mean-

I remember, especially in my 20’s, being so frustrated because my boss would tell me to go do X-Y-Z and I’d say, “Great, how do you want me to do it?” and my boss would say, “That’s not my job. My job is to tell you ‘Do it’ and your job is to figure out how,” which I thought was the most unfair thing setting me up for failure. I was just so frustrated and livid and I would come home and I would spout off to my wife about how unfair and how much extra work and I was going to have to invent stuff and my wife would say, “Well, why don’t we go the library this evening?” because that was before there was Internet and we’d look up some management books on how to do that stuff and I’d say, “Oh, well, it’s all right here!” She said, “Yeah, that’s called research. This is your job.” “ Oh, okay!” and I would come back and I would have completed it and if I completed it well and the boss liked it, everything was groovy and I got a bonus. If he didn’t like the answer, even if I had done the job well, if he didn’t like the answer, I was not as trusted next time to do it again so at first it was frustrating because it wasn’t fair that it was empowering and then, oh, we’re back on the “It’s not fair” part because if I said, “Yeah, your choices you made are not good choices,” then suddenly, my methodology was in question and I had to defend myself-and this is called, learning.

Yeah, you got to make sure your boss or your client look like the hero.

Exactly! That took me forever to learn because I was so interested in the truth and I should speak truth to power-not necessarily.

Yeah. Most importantly, protect that person’s job who hired you.

And then I had one boss who spelled it out much clearer and said that my job was to make sure that my boss got a promotion so that I could take his job. Oh, of course, I can do that! And so I had to study my boss’ boss to find out what his goals were so I could help my boss get promoted and if I did a good job at that, my boss just brought me up the next level and until finally, I realized I couldn’t work for other people anymore and needed to be an entrepreneur.

Yeah, but yet as a sought-after consultant, you still have that dynamic to deal with-it’s just you’re a contractor now instead of an employee, but you’re still looking after the person who hired you or signed the contract needs to look like the hero.

I had a good friend who was an electrical engineer for decades and got totally frustrated. Finally, he bought an engineering distribution company so he could sell electronics to companies and he could be his own boss and after about a year, we went out to lunch, I said, “So, how’s that going?” “Well, it’s okay.” “How’s that ‘I-don’t-have-a-boss’ thing?” “Oh, no!” he said, “I have 1500 bosses now!” Learned his lesson!

Yeah, I guess you’d have to run your own set of websites and not work for clients and then you, maybe, not have a boss.

No, your readers are your boss. If you run a website and nobody comes, you don’t have a job so we all got bosses.

Yeah. Your wife is your boss, right?

If you want to remain married.

Touché! Let’s kind of differentiate for our listeners the terms: metrics, analytics, and KPI’s. These words are kind of thrown around interchangeably and “analytics” was a term that Jared Spool on the episode where I interviewed him on Marketing Speak-here-has really railed. He railed on that term because people think of “analytics” as this Holy Grail of, “I’m going to get the answers I need,” and as we’re talking about these dashboards and things, we’re just putting the easiest data in front of us that are not the most compelling for our business.

Well, when you want to talk about definitions, I, of course, am going to go immediately to my latest book, The Devil’s Data Dictionary.

Well, let’s do it!

It says: Metrics-new and improved yardsticks deemed more reliable than imperial units. Analysis-when a calculation requires more than ten fingers. Analytics-same as analysis, but garnished with red pepper rings, micro greens, and chicory. And, probably my favorite in the book is: Artificial intelligence, which we’ll talk about eventually, but a definition for artificial intelligence is: fourth cocktail. This is looking at analytics and data by taking a big, poking fun at it because it’s just too darn serious all the time.

Yeah. At first, it’s kind of an easy definition for-not tongue-in-cheek, but it is kind of, let’s get serious for a moment.

You start with measurement and that’s just you know, I’m 6”1. That is a measurement and if you don’t compare it to anything, it’s kind of meaningless. Now, if you compare and contrast-if you put me in a room full of fourth graders I’m tall. If you put me in a roomful of NBL players, I’m short. If you compare how tall I am to how tall I was when I was seven, nine, and eleven years old-oh, well, now, we have metrics. Now, it went from a measurement to something we’re comparing. We can take that comparison and draw a graph. We can pull an average. We can compare it to other people and that’s metrics, so, taking a measurement and done it over time or comparatively to other segments and said, “Oh, well, the metrics say that I am ever so slightly taller than the average guy and I’m ever so slightly overweight than an average guy,” and now, we have something we can draw meaning from. Well, if you’re tall, that’s probably helpful and good. Our society deems tallness to be good for whatever reason, but the fact that I’m overweight, “Oh well, there’s something that we should talk about and work on,” and now that we have a goal, the measurements matter. I’m going to get on the scale every morning and now I can create a plan and look at results. Analytics is a different animal. Analytics is suggesting that, as I took measurements and lined them up against each other to create metrics, I’m going to take multiple metrics and line them up against each other, again, against a goal so I am going to measure how many people came to my website; I’m going to create metrics about the types of people so the people who came from search versus the people who came from display versus the people who came from email marketing, how do they act differently? That’s now a series of metrics that I can start acting against and say, Well, we’ve got seven segments and seven types of people, let’s treat them differently and see if we can improve the results whether it’s to sell more stuff, or convince more people to vote, or get more people to donate to the non-profit. If I can treat those different segments differently to get better outcomes, I am now engaging in analytics to look at multiple variables at the same time. Then, we get into big data, which has now become the annoying hype word of the decade, but that is simply, instead of taking three or four variables and comparing them in a spreadsheet, I’ve got a huge number of variables that I can’t manage on my own so I need a different kind of tool-a, quote, “big data tool” to help me look for correlations and look for causation. That is where we’re going to sprinkle some artificial intelligence and machine-learning on top because we’ve got more data than we can manage with more variables than we can fathom so it’s no longer human-the savant or the Rain Man cannot figure this out-and it takes a large number of computers working in parallel to look in see if this search term on that day produces more outcome because of the weather or because of some other variable that we hadn’t considered, but it’s there in the data. It’s there to be discovered, but it requires the computer to do that kind of discovery.

Right. We’re looking at analytics packages like Adobe Analytics and before we make a decision on which package is the best, we need to decide what are the things we care about the most and understand the differences between these terms, like, throwing around these terms interchangeably and we shouldn’t be. Now, once we have aligned our organization and our bosses’ incentives with our own and decided what we’re going to measure, now we can decide what analytics package makes the most sense.

I’m pretty sure that at this point we’ve stopped looking at features as a deciding factor on which kind of technology to get and instead, we’re now in that place of looking at recommendations from similar organizations who have used the products. In other words, if you’re going to go out and get a new accounting package for your company-well, they can all do accounting. There’s no question that all of the analytics tools can collect data, clean it, organize it, export it, and import other data. They all do tag management. They all do export to some sort of beautiful visualization tool. It’s no longer about features. It’s about the ecosystem of all of these. When you’ve got on one end, IBM and Hewlett Packard, and on the other end, Adobe, which used to be amateur and Google, and in the middle, here comes SalesForce.com, with a marketing cloud of their own, we’re now looking at, “Okay, well, I’m in the XYZ industry, what experience does that company have in working with my kind of field?” I mean, this is why the Web Analytics Association changed its name to the Digital Analytics Association-the problems we’re solving are much broader and they require an industry level knowledge, rather than just the technical bits and bytes anymore. That’s the path we’re headed on. That is a directional change that’s not going to stop all the way to machine-learning and self-programming computers. It’s, “Do you have knowledge of the industry?” that’s important. “Do you have knowledge of psychology?” is more important than, “Do you know what bits to twiddle?”

What do you say to the conspiracy theorists who say that with Google Analytics, you’re basically giving your data to the devil and they’re going to use that to adjust you in the rankings and do all sorts of other nefarious things?

I have two completely opposing views about that at the same time, proving that I’m human. One of them is, nonsense. Google can’t afford even the appearance of impropriety or their credibility would go completely. A conspiracy is also really difficult to manage, especially with that many people involved. People like to talk and they do. On the other hand, bias is built-in to the system by necessity because humans built it. Subtle decisions are made when you create a system and they are unintentional and invisible, but they absolutely exist. The people who are writing the algorithm to figure out a ranking of search results are bending over backwards to make sure that those search results are the best possible. If more people click on A than B then A must be the more correct answer. The user is imposing themselves on the algorithm and it becomes culturally shifted in the direction of the users. The more educated white males use Google, the less uneducated minority females will get value out of it-not by design, but by circumstance, so is that a conspiracy theory? No, it’s not, but is it a problem? You bet! Is there a solution to it? We’ll see.

One of the terms you mentioned a few minutes ago when we’re talking about features versus other things, you mentioned tag management. Can you define that for listeners?

Oh, sure! Where to start? You go to a web page and it’s going to start keeping track of what you do, how you do it, what you click on, how long you stay, and all of that stuff that makes us think about conspiracies. In order to do that, little bits of code have to fire behind the scenes while you are surfing. Different bits of code collect different kinds of information. Those different bits of code are fired or engaged through a tag or a page tag. If you go to the NFL’s website or you go to the Time Magazine website, there are maybe 15, 20, or 25 different page tags of different technologies collecting information in different ways. Managing those tags is a technical problem to solve. It is complex and so tools features have been created called Tag Management that allows you to put one tag on the page and then inside that tool manages all of these different collection agencies. There are probably 25 different little types of technology like that. You have to know tag management, you have to know search, and you have to know how algorithms work in general times 25 in order to do this technical job of collecting, managing, cleaning, organizing, and merging data together. At the moment, yeah, this is why we talk in terms of data scientists because it is complex. It started out simply and it got very hairy very quickly.

Because it matters where, for example, you’re firing these tags like, if it’s in the header or the footer, if the page hasn’t finished loading and then they disappear to another site and if it was in the footer, it may not have loaded yet. Or if it’s in the header. The tags are important for things like retargeting. Let’s define retargeting for our listeners too.

Probably one of the most annoying things that marketing people have invented so far on the internet. Retargeting is when you find a pair of shoes on any website and you put it in your shopping cart and you go, “Hmm, they’re too expensive!” or “Oh, I’ve got to go out to lunch,” or “Oh, I really don’t have any room in my closet for another pair of shoes,” so you abandon the shopping cart. For the next two months, those shoes follow you around and they are in every ad you see. The one that gets me? I’ve got a lovely digital camera from Amazon and for two months, Amazon tried to sell me exactly the same camera. Now, surely Amazon is smarter than the average bear. Surely Amazon should have been able to know that I bought that and told themselves so and sold me a tripod or a lens cap or a carrying case, I would have bought those things, but I already bought the camera so don’t sell it to me again please.

Yeah, pretty dumb, but when it’s done well, retargeting can generate a lot of revenue.

Well, this is the problem. This is why it’s so annoying for users. Even if it’s not done well, it generates revenue so the waste and the annoyance of showing me that ad is okay because some small percentage of people buy it again. Maybe, “Oh, it’s my brother’s birthday today. I like this camera. I’ll buy one for him!”

Even if it’s not done well, ads generate revenue so the waste and the annoyance of showing me that ad is okay because some small percentage of people buy it. Share on X

Yup.

It’s like, spam. People send out millions of spam emails because one bozo buys the thing and that pays for the 0.000002 cents that it cost to send it to me.

Yeah, wonderful! That’s why we have such great phrases like penis enlargement in our vernacular now.

Exactly!

Great! It’s a good point to bring in the discussion around correlation versus causation because the companies wrongly “causate” instead of “correlate” that, “Hey, this retargeting is working in its current form,” even though it’s been implemented in a brain-dead way. Because of the correlation and they’re assuming there’s causation there.

One of the definitions from Devil’s Data Dictionary is a little more serious than the rest, which is correlation. The definition is: Coincident; confidently mistaken for cause. If I look out my window and I see a whole bunch of people walking by with umbrellas, “Oh, look! They’re making it rain!” Every time I see those umbrellas, there’s rain coming down. Every time ice cream sales go up so do drownings so clearly, ice cream is dangerous. Well, actually no-there’s this temperature thing, right? You’re not seeing the whole picture. In fact, I would superstition almost. This is where analysts will always have a job. If I have the most high-powered computer and all of the data I could ever want to get my hands on, it will say there is a correlation between X and Y. There’s a correlation that says when the weather is bad, more people buy online than they do in the store, and as a human, I go, “Well, duh! Thank you, Captain Obvious! You’re right!” It is a causal factor, but I already built that into how I’m doing business. Twenty years ago, I knew that was true so tell me something I don’t know. “Well, whenever people buy blue socks, they also tend to buy Pepsi diet cola,” and I say, “Yeah, okay, that may correlate, but it actually has nothing to do with each other and it’s useless to me as a marketing person. Tell me something else.” “Well, when customer segment type 23 looks at pages 4, 7, and 12, they tend put more of these things in their shopping cart than those things and when they do, 75% are more likely to hit the purchase button.” “Okay, wait a minute. That’s interesting! I can use that information. I can put the right message in front of the right person at the right time with that information and that is a correlation I would never have thought to look up myself!’ It takes a computer crunching a whole bunch of data to say, “Oh, look! There is a relationship between these things,” but then has to turn to the human and say, “Is that important? Is that useful?” Let’s say the analyst is going to eventually get out of the bit twiddling technology business and get more into psychology and how the humans work and what is a valuable correlation that will help me achieve goals that my boss’ boss has set forth.

Right, and the non-intuitive correlations are the most interesting when you can apply it to improve your business. I remember, way back in the day, there’s something called Personified. It was talking about coming up with meaningful correlations. They found that Italian wine and pasta was a correlation, which was kind of obvious, but also white wine and bread, which is not obvious, but there is a very high correlation with that one too so if you’re Wal-Mart and somebody’s buying some white wine, you might want to offer them a deal on bread. Yeah.

Exactly! On the other side of the coin, there is a fabulous website called Spurious Correlations. Tyler Vigen is a brilliant guy who found a bunch of public databases and road algorithm that will look through all of these databases to find correlations. He has published a bunch of them and they’re all nonsensical. My favorite is, there is a very close correlation between drownings in public swimming pools and movies that Nicolas Cage has been in so clearly, Nicolas Cage is killing our children and there are charts and graphs. It’s Spurious Correlations, very google-able and very funny.

Yup, and so you have to reach statistical significance before any of this data is going to be believable, right?

Well, trustworthy.

Yeah. Right. Good differentiation there. In order to be able to trust that the data is real, we need to reach statistical significance. This is especially important when running tests. Let’s talk about the definition of statistical significance. Is that one standard deviation away? Two standard deviations away from being-or what?

Let’s talk about it in human terms rather than as mathematicians.

Okay.

I once yelled at my dog for barking and the dog stopped so clearly, yelling at the dog works every time-no, that only happened once and it never happened again so I have to find some other method that works consistently. If somebody says, “We did a poll,” and of course, this being the-I was going to say the election year as this is the decade of elections-pollsters are always out there and if I interview three people and they all say the same thing and I say, “Oh, well, clearly, that’s what everybody thinks,” that is too small a sample to be statistically significant. My website, the eMetrics Summit website, I get hundreds of people visiting my website a day-I’m saying that sarcastically. I only get hundreds of people a day so I can’t really do any heavy-duty statistical analysis because the people who show up, there’s just not enough of them to be statistically significant when you apply a higher level math so how do you know that you’ve reached statistical significance? Well, nowadays, the tools have the algorithms built-in that will tell you whether you reached a trustworthy number or not and how close to trustworthiness you are, but the poor man’s method is A-B-A testing. AB Testing is, I have two versions of the page or two versions of the offer and I’m comparing one to the other. How long and how many people do I have to show that to before I can trust the outcome? Well, the tools will tell me, but if I don’t have really sophisticated tools, I will do an A-B-A split that instead of just having two groups, I’ll have three groups. I’ll have the original A version, I’ll have the changed B version, and I’ll have a separate group that will show the original version too as well and I’m going to show those in the numbers will change and change and change and eventually, the first one and the last one will become the same because that’s the original version and people are responding to it equally in the group 1 and in the group 3. Now, I have statistical significance-poor man’s version-to say, “Oh, now I can look at the B version and see if it’s better or worse than the A version,” so statistical significance, I can get deep into the heavy math of it or I can just look at it from a logical perspective, but the real critical questions to ask whenever you’re looking at data of any kind is, “How big is my sample?” “How many people did we ask?” “How long did we let this test run?” and that’s where a smaller business has a serious problem. If I run a test on Amazon’s home page, I can get statistical significance in about an hour because a million people will see it in an hour and that is representative, but on the eMetrics.org website, I would have to run the test for six months to get enough people, but in six months too much has changed. I’m no longer measuring the same economic situation, the same competitive situation, and the same weather against what I started with at the end of the test so it is no longer valid.

And you’re basing the statistical significance on conversion and not just traffic because if you don’t get the conversions, let’s say, you get a lot of traffic, but no conversions and you’re trying to reach this statistical significance in terms of which one converts the best, you’re not going to hit the mark unless you get the conversion.

So, I have to always take this back to the logic and, again, using the small business as the example-well, in fact, I’ll use myself as an example. I put ads on Facebook and it drives a lot of traffic to eMetrics. I also put ads on LinkedIn and it drives about half as much traffic to eMetrics. Then I look at conversion-well, my product does not lend itself to a straight conversion analysis because it takes six to eight visits over a three-month period to decide whether you’re going to go to a conference: Who are the speakers? What are they talking about? What city is it in? What’s the price? Will my boss let me? Can I get approval for travel? Will my spouse let me? Can I figure out babysitting? I mean, there’s just a whole bunch of consideration going on so, what should I measure? My solution was, if you come to the eMetrics website and you look at the agenda, the speakers, and the price-three different pages-to me, that qualifies as a good engagement. You cared enough to find out how much it costs, that’s about the best thing I can measure in one visit.

Right.

The people who came from Facebook looked at one or two pages and never looked at price. The people who come from LinkedIn, “Oh, they’re looking at the price much more often!” so I’m shifting my money from Facebook over to LinkedIn, that’s called analysis. It’s not heavy-lifting from a numbers’ perspective, it’s brute force measurement.

Yeah, and when you’re dealing with a longer buy cycle, you have to come up with interim steps like milestones that matter.

Yes.

Let’s say you’re IBM and you’re selling some multi-million dollar consulting package to huge companies, white paper downloads could be that interim step or that milestone that you measure success against so this campaign drove X number of white paper downloads and this drove way more so this one is probably better-even better if you can calculate what that white paper download is worth. Based on the kind of the averages of how often people convert after say, 18 months into the big seven-figure consulting contract, maybe that white paper download on average are worth $100.

And so now you know how much to invest in promoting that white paper.

Yup.

It’s worth spending $20 because it’s worth $100 for the download.

On average, yeah. Exactly! Another thing that has that stymies a lot of marketers is correct attribution. They’re not sure that the thing that they paid for is what’s generating the leads or the sales, right? So, if this “last click,” “first click,” and all that-let’s dive into that for a bit. Can we, first, define attribution and then talk about the significance of it?

I’m spending money on marketing. I’m paying for search terms and I’m paying for content development that will generate organic traffic. I am buying display ads, I’m doing e-mail marketing-I’m spending money in all of these different areas and in a perfect simplistic world, I spend a dollar on each one of those and whichever one sends me back the most money, I just put all my money over there and I retire, but in reality, people respond to a multitude of touch points. They’re going to see a display ad five times before they even notice it. They’re going to get an e-mail and ignore it until the third time and because by that third time, they’ll have seen the fifth display ad, they’ll have read a blog post somewhere else; they’ll have seen a billboard or heard somebody discussing it in a podcast; and they’ll go, “Well, maybe that’s interesting. I will click on it now!” so who, where, and which dollar that I spent should get credit? How do I attribute the visit across all of that expenditure? The classic methods are, well, whatever they clicked on last was the deciding factor so that you get the credit. On the other hand, if you’re a branding kind of person, you believe that the first click, whatever caused them to click on, the first is the one that you get the credit because it really tipped the scale-that was the tipping point that got them in the funnel, if you will. There are those who say that the first clicks are the least valuable, but they still have value so we’re going to attribute a small amount to the first clicks and then more, more, and more and a large amount to the last click. There are some who say, “Well, I am going to parse this out by hand and I’m going to assume that the display ad had more impact, that the search results had even more impact, that the email wasn’t so important if it was the third thing, but it was very important if it was the fifth thing, and create this complex model that says, I believe, this is how our advertising and marketing is impacting the public.” Now, you can take all of those models and run real data against it and see which one more closely resembles the truth. If you find one that seems to be representative then you can use it as a predictor of what might happen and that will help you identify where to shuffle your money around. It is a seriously tough challenge and there are a dozen companies who are trying to apply some pretty heavy math to help create these models. However, “What is the impact of having a billboard on the highway? How on earth do you measure that?” is the marketer’s dilemma and always will be.

Yeah, and it’s one of the biggest dilemmas in SEO because when I’m trying to convince a prospect to become a client, they’re asking me, “Well, tell me about some of the results I’m likely to expect out of this engagement?” I’m like, “Well, it’s kind of like a leap of faith!” I mean, this is a difficult thing to track because let’s say, you’re doing a PR campaign and generating a lot of interest in your product through offline channels-you get written up in different magazines and industry journals and so forth that’s going to drive traffic via Google to your site-and I’m going to get credit for that, right? If you’re doing paid search, there will be some spill-over into SEO as well because sometimes, people will come back and they’ll click on the organic result even though it was the paid ad that generated the interest in the first place.

And all the research has shown that that is very true. The more you advertise, the more organic results you get because it’s branding. If I’m going to go look for shoes, did I see more ads for Adidas or Nike or something that nobody else has heard of, but their advertising just happened to click with how I feel about my feet, and that’s like, “Man, that’s my brand from now on!” so when I go search for shoes and that brand pops up, I go, “Oh, yeah I want to check that out! I saw the ads, but I haven’t thought to search for it, but that’s the one I’m really interested in!” and you cannot measure that-well, if you do enough marketing, or if you are Amazon, or if you’re Wal-Mart, you can throw a huge amount of money at it and you can have a competitive advantage over the rest of us, but for the rest of us, we’re dealing with what’s the difference between advertising, marketing, public relations, and partnerships to pick another analogy? There is no formula for getting that girl to go out on a date with you. It’s like, give her three compliments on her looks, bring her flowers twice, go out of your way to help her mother in the parking lot of the grocery store, and then absolutely, she would go out with you-no, it doesn’t work that way. It’s a mystery.

Yeah. I unfairly get credit for things that just happen because they’re working on other channels, but then I will lose the credit for things that I’ve done that are really powerful like let’s say, a link-building campaign just crushes it and generates a lot of link authority, but there’s very little direct click-through traffic from those links. But then the ranking across the board go up like it’s the rising tide that lifts all boats and all pages benefit from this link authority. I can’t, for certain, prove that that’s what happened. I can say, “Well, this is likely what happened but also, there was this thing that we just hired a PR firm two months prior, and that’s what I think made a big difference.”

And this is where all of this stuff becomes political.

Yeah.

The chief marketing officer who’s been in that job for 20 years got to that job because he was on Madison Avenue in the 60’s having three martini lunches and he does believe in television and that is his experience and it’s obvious. I mean, you have a million people watching the same show at the same time-show them an ad! That’s just common sense.

Right. Speaking of political, one of the ways that you can be political inside an organization is to show dashboards and depending on what the dashboard is, you can cause people to lose their jobs, get a promotion, or whatever and it’s all just how you spend the data.

Yeah. Lies, damn lies, and statistics, absolutely!

Yeah, so let’s talk about dashboards for a minute. What is your definition of a dashboard? What do you see dashboard being useful for? Where do you see them being misused?

Well, let’s again go to The Devil’s Data Dictionary since you asked for a definition of dashboard. A simplified display of only the most compelling metrics used to keep executives manicured pinkies out of the machinery. Also, the most succinct way to incite political discord and foster a misunderstanding of organizational objectives.

That sounds great! Where do we sign?

I have a deep distrust of dashboards because they imply that there are only certain things I need to look at and care about. Everybody’s familiar with the automobile. I need to know how fast I’m going, but actually I don’t. I can just go along with everybody else and I won’t have an accident and I won’t get a ticket. However, I want a speedometer. I definitely need to know how much gasoline I have in the tank, that’s critical, but I don’t need to know it every five seconds. Maybe an alert would be useful for that. If my car knew that every two weeks I drive from Los Angeles to San Francisco, it could give me an alert that says, “You’ve got three days before that trip to fill up your tank.” Oh, well, that would be just as useful as a dashboard. However, in business, if I take 10 or 20 things to put up on the wall, and I glance at them every day and say, “Well, everything’s going fine,” I am missing out on an opportunity and it is hiding problems. The worst thing is just a report that gets emailed every week. “Here’s the report!”-blah-and after three times, nobody looks at it anymore because who cares? Then, there’s the dashboard. I look at the dashboard, everything’s green, fine, and I can ignore it. Then, there are alerts that say something weird is happening and it could be good or bad, “There is a drop in traffic, go find out what’s wrong,” or “There’s a spike in traffic, go find out what’s right and see if you can repeat that.” It is the machine tapping you on the shoulder saying, “Hey, this is interesting! Go look at it!” Then, there’s automation that takes all of the above and makes decisions about it. If somebody comes to my website and they have seen the display ad before and they have clicked on an e-mail marketing campaign and they download the white paper, the machine can say, “Oh, therefore, I should send out this offer because statistically, they are more inclined to accept this offer than anybody else I could send it to!” Now, we’re getting really useful. The next step is predictive analytics where the machine says, “This is what happened in the past and based on the math, this is what’s likely to happen in the future,” and therefore, how you should spend your money or how you should spin your offer or which segment of your audience you should send that to, but a dashboard kind of allows me to stop paying attention. It allows me to stop asking questions. I can glance at it and go, “Okay, there’s no problem!” or “There’s no change!” or “Yeah, I see we’ve got mostly green and a couple of yellows that are normal and I know why and I don’t have to think about it,” instead of constantly being curious about what’s going on and asking new questions. To me, a dashboard is a reason to not think.

Yeah, it’s a crutch. It can give you some leverage if you want to do management by embarrassment.

Oh, yes! Read your Machiavelli.

Yeah. We’re about to run out of time here-one last question I wanted to ask of you is, what’s coming next? You’re great at prognosticating and reading tea leaves so, where are we heading?

Well, tea leaves and books-the one I’m cracking open at noon today is called, Only Humans Need Apply: Winners and Losers in the Age of Smart Machines by Tom Davenport and Julia Kirby. The fear of the singularity is that artificial intelligence will you take over the world and turn into Skynet and we’re all doomed, but certainly, the fear for analysts is that we’re out of a job. If the machine can find correlations and if the machine can figure out which correlations are valuable and make decisions on what messaging to send to what person at what time, then we’re in trouble so what’s coming is, the machines are going to get much smarter at the bit-twiddling, the data collection, and the data cleaning-the work that all of us, analysts, have to do that we really don’t like to do. It’s like, if you’re an oil painter, you want to have a canvas and brushes and you want to paint, but first, you have to build a frame, stretch the canvas, treat the canvas, you have to mix the colors, and you have to have the right materials that are clean enough. Now, finally at the end of the day, when you’ve done all this prep work, you can start being creative-no, I’m going to do that tomorrow. With data, it’s the same thing. We spent so much time cleaning, collecting, organizing, and validating that it’s frustrating. We don’t get enough time to just do the analysis so the machines are going to come along and help with all of that and that’s great-I can’t wait-but then, they’re going to take the next level up and start doing analysis. What does that mean for us humans? What’s over the horizon? The thing that I’m going to start digging into next is, what is the role of a human in collaboration with the machine? Instead of programming the machine, we teach the machine like pets or a child and what talents and skills are required to be a good mentor to an AI system? To me, that’s intriguing.

Yup.

The rest is logic, brute force, and hard work, and boy, is it a lot of work! There aren’t enough people. We still have more jobs begging for analysts than we have humans to do that and therefore, there is a marketplace for automation for the work we’re doing. An example of tag management is a great simple example. It is a tough, rudimentary problem that humans have had to solve and oh, we’ve got some tools that can help do that-great. Well, those tools get more sophisticated and how the human interaction with machines will have to get more sophisticated as well.

Yeah, there is a really great article series on the Medium.com that John Smart wrote about personal Sims and how life changing this is going to be in the near future. How we’re going to come to rely so much on our personal sim for our news and everything-all the input coming into us from the world will be filtered through our personal sims and they’re going to know us better than our significant others.

Well, I think the best way to think about that is, imagine if Google only works for you and you were freer by telling it everything that it knows already six, but you could trust it to negotiate for you. “I need to get new tires for my car and in my climate and in my price range, what’s the best?” and this personal sim-good way to describe it-goes out and negotiates with vendors on your behalf. “Oh, there’s another one, I can’t wait!” I will give up the flying car if I can have that today.

Yeah, amazing! It’s a brave, new world coming up. It’s exciting, but a little bit terrifying for all of us out there. How would somebody like take the next step with you? Like, sign up for the emetics Summit or get a hold of your book, The Devil’s Data Dictionary, or hire you as a consultant?

The Devils’ Data Dictionary is just for fun. If you’re in the data business, I encourage you. For the eMetrics Summit, the very next one is coming up June 20-22 in Chicago and that’s going to be great fun, entertaining, and educational because I have tried to find anybody I could who’s talking about machine-learning and marketing. The eMetrics Summit is becoming about marketing analytics today and tomorrow so you’ll learn a lot about what’s going on and network with people who really know what’s up, but then here’s a look into what some advanced people are doing and what it’s going to look like next-that’s eMetrics.org. I can be found at my consulting company, which is Target Marketing, so it’s Targeting.com. Of course, I can also be found on the Digital Analytics Association.

Very good. Thank you, Jim, so much for sharing all your wisdom and these insights into data. We really have to process the data analyzing and be thoughtful. We should not just look at dashboards and trust the analytics packages-we actually have to use our brains. Thanks for that reminder!

My pleasure, Stephan, and thanks! It’s always such a pleasure to chat with you.

Okay. Thank you, listeners, for listening. We’ll catch you on the next episode. This is your host, Stephan Spencer, signing off!

Important Links:

Your Checklist of Actions to Take

☑ Never stop asking questions. The data that you collect is only useful if you put it into action, find out what is working, what needs to be changed, and how you can better serve your clients.

☑ When running AB tests, create a third version-this will create more variety and more data on what your target audience responds to.

☑ For more data definitions and hilarious information, check out The Devil’s Data Dictionary!

☑ Don’t rely solely on a dashboard to give you data. While it’s a good start, you have to look further into the information provided by your analytics platform, to get a full picture.

☑ Create metrics around the types of people that come to your website, and where they came from. You can treat those different segments differently to get better outcomes.

☑ To reach statistical significance, include as many people as possible when you’re testing data or running ads.

☑ To find correlation that can help you, find a relationship between the pieces of data that are connecting, and then decide if it’s important or useful and can be acted upon.

☑ Don’t just look at the number of people that are coming to your website. Who are these people, and what is the goal of the organization? These questions can help you earn trust and boost sales.

☑ To create metrics, compare at least two pieces of data or factors.

☑ Research tag management tools to find the one that can best help you run tags on your website.

About Jim Sterne

Jim Sterne is an international consultant who focuses on measuring the success of digital relationships with customers. Sterne has written ten books on interactive marketing, is the Founding President and current Board Chair of the Digital Analytics Association and produces the the eMetrics Summits. Author of the Devil’s Data Dictionary. You can find Jim on Twitter @jimsterne.