Opened 5 years ago

Closed 15 months ago

Last modified 3 months ago

#517 closed Bug (fixed)

For a visit via a campaign, if the campaign keyword is not set, detect keyword from Referrer URL

Reported by: matt Owned by: matt
Priority: critical Milestone: 1.11 - Piwik 1.11
Component: Core Keywords:
Cc: Sensitive: no

Description (last modified by matt)

Bug report:

When a visitor comes and a campaign "name" is detected, there are sometimes no "keyword" set. Instead of leaving the value empty, we could try to fill up the gap and:

  • try detect the keyword from the Referrer URL if it is found there (ie. if a search engine was used)
  • if no keyword could be detected from the ref URL, set the "campaign keyword" as the domain hostname. This way it will give an easy breakdown of referrer website URL for campaigns.

This would improve the campaign tracking data quality.

Google Adwords tracking would also work better and give more accurate data.

Change History (31)

comment:1 Changed 5 years ago by matt (mattab)

  • Description modified (diff)

comment:2 Changed 5 years ago by matt (mattab)

  • Description modified (diff)

comment:3 Changed 5 years ago by matt (mattab)

  • Summary changed from Test the campaign feature with Adword campaigns to Test and documentation the campaign feature with Adword campaigns

comment:4 Changed 5 years ago by matt (mattab)

  • Milestone changed from Surviving The Wild to DigitalVibes

comment:5 Changed 5 years ago by matt (mattab)

  • Priority changed from major to critical

comment:6 Changed 5 years ago by matt (mattab)

  • Description modified (diff)

comment:7 Changed 5 years ago by matt (mattab)

  • Component changed from Core to Documentation

comment:8 Changed 5 years ago by MarcLindsay

No worries I'll give this a shot with a small adwords account I am playing with. Though I also have some more advanced idea's for tracking adwords without auto-tagging and thus giving more functionality in other area's.

Will test this out firstly and go from there.

comment:9 Changed 5 years ago by matt (mattab)

marc, sounds good; let us know if you find any bug or issues, this has never been tested, but we want this to work perfectly and with an easy to use documentation; I think we can assume that we need adwords to be auto tagged using landing keyword, etc.

comment:10 Changed 5 years ago by MarcLindsay

Hey Matt,

Not really. There are other less obvious to others, ways to track adwords that still keeps your data private.

See bots and others can crawl and track adwords adds and backwards engineer what you are bidding on. Do that every day over a period of time and you have a clear picture of what keywords are profitable for someone in what industry.

I'll test it out on auto-tagging and then discuss privately the other concepts and decide if you want to introduce those.

Of course there could be both options, auto-tagging and then the privacy option :D

Talk Soon
Marc

comment:11 Changed 5 years ago by alivenk

comment:12 Changed 5 years ago by domtop

comment:13 Changed 4 years ago by matt (mattab)

  • Priority changed from critical to major
  • Sensitive unset

comment:14 Changed 4 years ago by matt (mattab)

  • Milestone changed from 0 - Piwik 0.6.3 to 3 - Piwik 0.9 - Surviving The Wild

comment:15 Changed 4 years ago by matt (mattab)

  • Milestone changed from 3 - Piwik 0.9 - Surviving The Wild to Features requests - after Piwik 1.0

comment:16 Changed 3 years ago by matt (mattab)

example adwords url

http://www.google.co.uk/aclk?sa=L&ai=Ct4TzXV_9TL6QIYS9hAfg9oGfCuj09-QBwMGNhxi4hsLuVggAEAIoAlDA-7T6-P____8BYLueyoPcCsgBAakC1ql8VKZNuj6qBB5P0As2Ghmje5goPNu05sgS5wCcz4ejXah6M3nHdqg&sig=AGiWqtwqIsBBq929N9J0_AiRN6Mo0uHa4Q&adurl=http://www.itsaviours.com&rct=j&q=site%3Apcfixt.com%20computer%20repair%20falkirk 

comment:17 Changed 3 years ago by matt (mattab)

  • Component changed from Documentation to Core
  • Milestone changed from Feature requests to 1.2 - Piwik 1.2

From forum post

Generally, Adwords links seem to look like this:

http://www.google.com/aclk?sa=L&ai=CiUrQZYQYTbfCIMrgswaj9KjVApnw_9gB69247wnD4P8JCAAQASgDUPrYzh5gldqIgpgHyAEBqgQhT9CSfMdbwbHaUGieetZZj76iHm8iOC3ULeIvG3mH4k5qugUTCIOhx9G1jKYCFYbhzAodME9x38oFAA&rct=j&q=flights&ei=ZYQYTcPaDobDswawnsX7DQ&sig=AGiWqtwu0APuZWGM9C-z3t8c6j8f6irfjA&adurl=http://pixel1590.everesttech.net/1590/rq/3/s_1b6ee0e1a0059564b80fdcb8e19f29f3_2558612431/url%3Dhttp%253A//www.cheaptickets.com%253Fgcid%253DS16036x129%2526keyword%253Dcheap%252Btickets%252Bcom_ge%2526WT.mc_id%253Dec105%2526WT.mc_ev%253Dclick%2526WT.srch%253D1%2526DCSext.mc_kw%253Dcheap%252Btickets%252Bcom&cad=rja

http://www.google.com/aclk?sa=l&ai=CMkaoZYQYTbfCIMrgswaj9KjVAuz2g8gBzNXciRbD4PMYCAAQAigDUMHX2Kb4_____wFgldqIgpgHoAH608P1A8gBAakCHLcuXY2yhz6qBBxP0MITvVvCudvoax-w1S8ZuQgOZQKZVQJ1_pDNugUTCIOhx9G1jKYCFYbhzAodME9x38oFAA&rct=j&q=flights&ei=ZYQYTcPaDobDswawnsX7DQ&sig=AGiWqtw2ETvZZJV8d4SQdSSHAJqf3ACuXw&adurl=http://en.momondo.com/Default.aspx%3Faw%3Dde&cad=rja 

comment:18 Changed 3 years ago by matt (mattab)

  • Summary changed from Test and documentation the campaign feature with Adword campaigns to Detect Google Adwords referers as Campaigns (Search & Display network)

I made some test for a few days with an adwords account. Adwords reports 5 clicks and I see 5 clicks in the Piwik DB, good start. But 1 didnt have the referer set, 2 others were from google & Ask.com, 2 from Google display (adsense).

The only way to track Adwords properly is to have users enable auto tagging in their account:
http://adwords.google.com/support/aw/bin/answer.py?hl=en&answer=55590

Then landing page URLs have the parameter 'gclid' which is how we detect that the user is coming from adwords.

Alternatively (AND / OR), Users can 'manually' tag their landing pages, then they simply can use the standard Piwik campaign feature #604

Then we can look at the ref URL

  • If google (or Ask or other SE) then it is Google Search Network, with given keyword
  • If domain = googleads.g.doubleclick.net then it is Google Display Network

In the "Display" case, the ref URL looks like this:

	http://googleads.g.doubleclick.net/pagead/ads?client=ca-amazon-alexa_js&format=336x280_as&output=html&h=280&w=336&lmt=1294593285&channel=g-channel-14&ad_type=text&color_bg=ffffff&color_border=ffffff&color_line=00ff00&color_link=0033cc&color_text=333333&color_url=339900&oe=utf8&flash=10.0.32&hints=dospuntocerovision.com&url=http%3A%2F%2Fwww.alexa.com%2Fsiteinfo%2Fdospuntocerovision.com&adsafe=high&dt=1294593287457&shv=r20101117&jsv=r201106&saldr=1&correlator=123287498&frm=0&adk=3179527&ga_vid=1772644835.1294598&ga_sid=1294588&ga_hid=489613643&ga_fc=0&u_tz=0&u_his=3&u_java=1&u_h=800&u_w=1280&u_ah=760&u_aw=1280&u_cd=32&u_nplug=25&u_nmime=151&biw=1280&bih=577&eid=30143102&ref=http%3A%2F%2Fwww.bing.com%2Fsearch%3Fq%3Ddospuntocerovision.com%26pc%3Dconduit%26form%3DCONMHP%26ptag%3DA5DC7BAE8890B42A4A3F%26conlogo%3DCT2769726&fu=0&ifi=1&dtd=71&xpc=cO7JaJYx7F&p=http%3A//www.alexa.com

The parameter 'url' gives the URL where the click comes from. This is important to report it in Piwik since adwords advertisers would want to ban domains that have poor conversion rates from their Display pool.

Maybe a good way to report Adwords in Piwik would be:

  • on each page view in Piwik, check if URL contains gclid
    • if it does contain, AND there is no standard Piwik campaign parameters (OR GA campaign parameters), check referer:
    • if referer not empty and is googleads.g.doubleclick.net OR is a known search engine (big search engines have deals to display adwords on their search result pages)
      • we have detected a click from Adwords (ie. that we are paying for)
      • if this is a new visit OR if an existing visit with empty referer, then we record adwords as referer type 'campaign'
        • Set campaign = 'Google Adwords - Display Network', keyword = URL parameter from referer
        • OR set campaign = 'Google Adwords - Search Network', keyword = KWD detected in the referer

See google search network: http://adwords.google.com/support/aw/bin/answer.py?hl=en&answer=90956

and google display network: http://adwords.google.com/support/aw/bin/answer.py?hl=en&answer=117120

interesting page: http://www.google.com/support/forum/p/Google%20Analytics/thread?tid=3b79c0131cf3148d&hl=en

Tasks

  • update the Tracker algorithm to properly detect adwords search/display
  • Link the 'Campaigns' Referer report from the Referer Overview report
  • Ensure that logos are displayed in the campaign report like normal organic search
  • Create a FAQ answer to explain that this is working, that auto tagging must be enabled, etc.
  • Create a FAQ answer to explain how to track Campaigns in Piwik #1042
  • Update the Goal Tracking documentation and clarify that Goal conversions will be reported also for each Adword referer (ie. 'Adwords / keyword 'web analytics' has convertion rate 5%). Put a screenshot in the doc.

Any other suggestion / idea, please comment

comment:19 Changed 3 years ago by matt (mattab)

on a related note...

comment:21 Changed 16 months ago by matt (mattab)

  • Description modified (diff)
  • Milestone changed from 1.10 Piwik 1.10 to 1.9.3 - Piwik 1.9.3
  • Owner set to matt
  • Summary changed from Detect Google Adwords referers as Campaigns (Search & Display network) to For a visit via a campaign, if the campaign keyword is not set, detect keyword from Referrer URL

Claryfying & Restricting the scope of this ticket

This was the previous ticket description:

Piwik currently support campaigns and keywords for campaigns, but this support hasn't been tested with Adwords use case. This ticket is mainly about testing the support with real data, making sure that adword campaigns, when url tagging is enabled (see in adwords), piwik will detect the campaign name, the keyword from the referer or pick it up from the tagged URL, etc.

'''Draft'''

You need to enable auto tagging in your adwords campaign. "Google Analytics makes it easy to track your AdWords referrals effortlessly using the auto-tagging feature. Auto-tagging is enabled by default when you link your AdWords and Analytics accounts. To turn auto-tagging on or off at any time, update the checkbox in your Account Preferences from your AdWords My Account tab. Using auto-tagging allows you to bypass the steps to tag the URLs of your AdWords advertisements."

In Piwik, you can tag your links with two variables:

    * Campaign name (identifier of the campaign). This parameter is by default called piwik_campaign
    * Campaign detail (for example, the banner name). This parameter is by default called piwik_kwd 

When you enable auto tagging in adwords specifying the parameters names piwik_campaign and piwik_kwd, your visits coming from Adwords will be tracked in Referers > Campaigns. You will see how many visits, bounce rate, page views for each campaign and keyword. Also, if you have setup goals, you can see conversion rate and revenue per goal, for all your Adwords campaigns.

'''Parameter names'''

You can change the parameter names by setting the following in the config/config.ini.php and replacing by the name you wish to use.
{{{
; variable name to track the campaign name 
campaign_var_name		= piwik_campaign
; variable name to track the campaign detail (eg. keyword)
campaign_keyword_var_name	= piwik_kwd
}}}

'''Note: missing feature'''

"when i track campaigns via "piwik_campaign" from adwords i got only the data from the campaign query but not the real searched keyword (referer)." Explanation: currently when a campaign is detected, it doesn't try to load the keyword from the referer. This is a bug (or missing feature). What we can do is: when the campaign keyword is not specified in the landing URL, we can automatically load the keyword from the referer if found. 


'''Other'''

We also need a user guide on 
 * How to use campaigns in Piwik? (see [http://dev.piwik.org/trac/ticket/604 documentation example for tracking newsletter]) 

See also #476 for tracking adsense ads or yahoo content network.

comment:22 Changed 16 months ago by matt (mattab)

  • Description modified (diff)
  • Priority changed from major to critical

comment:23 Changed 16 months ago by matt (mattab)

  • Description modified (diff)

comment:24 Changed 15 months ago by capedfuzz (diosmosis)

(In [7811]) Refs #517, search for campaign keywords in referrer URL when not supplied by query param and use referrer hostname when no keywords in referrer URL and nothing supplied in query param.

Note: also includes quick tweak to integration test base for more descriptive error when expected file cannot be found.

comment:25 Changed 15 months ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from new to closed

(In [7812]) Fixes #517 if a campaign name is set, we will try to fill in the keyword with the publisher Url (when visitor came from adsense ad) or with user keyword (when coming from adwords)

comment:26 Changed 6 months ago by michalfapso

I have also problems with tracking AdWords campaigns. This issue is closed, but the problem seems to be not solved yet. I see lots of "googleads.g.doubleclick.net" referrers in my statistics. They are not detected as campaigns. The referrer URLs are of the form in #476. There are many parameters in the link, but parsing at least the "url=..." would be nice.

I would also like to help with coding, just tell me what would be the best way to implement this in Piwik.

Thanks a lot,
Michal

comment:27 Changed 6 months ago by michalfapso

I have found a workaround:

--- core/Tracker/Visit.php.orig	2013-10-14 10:42:00.000000000 +0200
+++ core/Tracker/Visit.php	2013-10-15 14:46:16.411174133 +0200
@@ -1703,6 +1703,12 @@
             }
         }
 
+        if (empty($campaignName) &&
+            strpos($this->refererHost, 'doubleclick') !== false
+        ) {
+            $campaignName = "AdWords";
+        }
+
         if (!empty($campaignName)) {
             $this->typeRefererAnalyzed = Piwik_Common::REFERER_TYPE_CAMPAIGN;
             $this->nameRefererAnalyzed = $campaignName;

The original code did not work because the pk_campaign parameter was not set for our AdWords campaigns. This way, every visit from doubleclick host is considered as the AdWords campaign and the referring sites are encoded as campaign keywords.

comment:28 Changed 6 months ago by matt (mattab)

thanks for the suggestion. Could you send me an example of a Adwords URL and an example of a Adsense URL ?

I think it would be best to set the campaign name to either Adwords or Adsense depending on the referrer URL.

comment:29 Changed 6 months ago by michalfapso

Thanks Matt, I sent you the URL examples to your email. Don't know if there is any secure info in the url parameters.

There is an url parameter "adsense_enabled", which is always true in our links, so I don't have any examples of Adwords (no Adsense) urls.

comment:30 Changed 6 months ago by matt (mattab)

In d5429865f7644099db4ef677d11cbed562b9484a:

Fixes #4261 Refs #517 Properly attribute to Adwords those visitors that come from a click on adwords

comment:31 Changed 3 months ago by matt (mattab)

see also #4468 when &gclid= parameter found, mark visitor as Adwords

Note: See TracTickets for help on using tickets.