According to Google's income statement to investors (on the Google website), Google earned over $6 billion from advertising during 2005 ($6,065,003,000 to be precise), and has earned over $4.6 billion in the first two quarters of this year. My estimate is that over 95 percent of this comes from AdWords; other sources claim 98 percent. This means Google's AdWords counting technology was responsible for at least $5.7 billion last year.
Given that such a vast amount of our money goes to Google on the say-so of a single web analytics system, it's extremely important to know how accurate that technology is.
It is well established that it is impossible for any web analytics technology to be 100 percent accurate. Most systems are accurate to within 95 percent. There are many reasons for this, which I've covered in earlier articles (see "Things That Throw Your Stats").
I have asked Google how accurate they consider the system to be. So far they have declined to provide any answer, so they haven't actually claimed 100 percent accuracy. Let's give Google the benefit of the doubt and consider the possibility that AdWords' click count is 99 percent accurate. This would make it the most accurate web analytics technology on earth by a long stretch. It would also mean it had mis-billed over $60 million last year. If it was 95 percent accurate, then it mis-billed $285 million last year.
My personal experience is that the accuracy of the billing system in Google AdWords varies. I run PPC campaigns for clients, and I always check Google's numbers with what our own software, InSite, says. Since we always run a unique tracking code on every Google ad, this is fairly easy. I have disputed numbers with Google several times over the last few years.
I'm not talking about click fraud here. Click fraud disputes do not involve a disagreement about the volume of traffic Google has relayed to your site. Click-fraud involves a disagreement about the validity of that traffic. Both sides agree on the numbers; they just disagree on whether all of it should be paid for.
The situations I'm concerned with are where Google AdWords claim to have sent more visitors to my site than I can verify for myself.
It's unreasonable to expect the numbers to match exactly. No two web analytics systems will ever agree perfectly. However, my experience of other systems suggests the disparity should not be too great. Overture/Yahoo PPC numbers consistently match mine to within two percent. Google AdWords usually match mine to within three percent. I think two percent is about as close as is possible, and three, or even four, percent is acceptable.
However, I do get concerned with the really big disparities. These are rare, but given the amount of money moving through AdWords, even rare disparities add up to a great deal of money. I have personally seen Google AdWords count 25 percent more traffic than I could verify on several occasions. This summer I had a situation in which the Google AdWords were claiming numbers ten times what I was seeing.
Here's what happened:
I recently ran a small test campaign for a new business. In the first two weeks Google AdWords claimed (and billed for) 96 clicks, but we only saw 11. When I queried this Google's AdWord Response Team said they checked and the numbers were correct. They didn't tell me how they'd checked the numbers, but they did point out I had a history of disputing their counts. They said this indicated my software was probably faulty. They invited me to use Google Analytics (once known as Urchin), which was, in their words, "one of the most powerful web analytics products on the market."
I was not convinced my software was faulty since it has always matched Yahoo/Overture, and usually matches Google AdWords, but if Google wanted to use their own system to check AdWords numbers I was willing to give it a try. I made sure to add all the tracking options Google recommended to ensure the Google Analytics count was as accurate as possible.
Having done everything Google asked of me -- by the book -- I waited a week for the numbers to cook. At the end of that time, Google Analytics told me I had 14 AdWord visits while the AdWords billing system itself claimed 59. My own software, InSite, also said 14 visits, the same count as Google Analytics.
On other metrics, such as traffic from native listings, or other sites, InSite matched Google Analytics to within three percent. I think this validates the accuracy of both InSite and Google Analytics. So the system that Google told me to use was telling me that Google AdWords were over-counting. I now had two different products telling me that Google AdWords was over-charging, and by a significant margin.
This article is not intended to be a rant against Google. I don't want to condemn them just because their AdWords system has made a mistake. I'm not even going to criticize them for insisting I use their software to check. They know nothing about mine, so why should they trust it? What is really important is how Google handles situations when a customer believes they have been over-charged. Given the volume of traffic AdWords handles, it would be incredible if the AdWords click-counting system never made a single mistake. However, given the amount of money at stake, it is very important Google handles situations like this in an appropriate manner.
My next step was to report my findings to Google's AdWord Response Team.
Their answer was straight-forward: "For privacy reasons I do not have access to your Analytics data. I've reviewed your account and found that you accrued charges for legitimate clicks made between July 28, 2006 and August 23, 2006 while your campaign was active. Therefore, we are unable to offer a refund for these charges."
This raises a number of questions: How did Google verify my charges were accurate? What is a "legitimate click"? How does Google explain the discrepancy between the two systems? Given that I'm still unhappy, what happens now?
My next article will explore Google's response, and then we'll look at the legal and business issues. In the interim, I'd like to hear from anyone else who has been in a similar situation. I'd like to get a handle on how big a problem this is. I've asked Google how often disputes occur, and the value of the disputed transactions, but they haven't provided an answer as of yet.
If you've felt Google AdWords has over-charged you, let me know. If you've disputed this with them, how have they responded? Do you feel Google had handled things appropriately, or did they leave you feeling frustrated? Email me with your story.
Brandt Dainow is an independent web analytics consultant. Read full bio.