Custom Variables support: new JS API and new reports

[matt]

FEATURE RELEASED in Piwik 1.2

See the documentation in: Web analytics - Tracking Custom Variables

The goal of this feature is to put visits into buckets (segments) via the Piwik JS API. Then access reports about these custom name/values just like other reports in Piwik.

Custom data: Use cases

• Count Logged in users VS logged out visitors (or Customers VS prospects)
• Count custom things per visit that are not included in Piwik standard metrics, eg. 'games played', 'comments left', or anything etc.
• Track position of keywords in google search page (using new parameter found in ref url recently after quick parsing of ref url in JS)

Custom data: New default reports

• Custom Variable: list all distinct names, for each list visits, pages, time on site, bounce rate, etc.
• On click on any variable name, list all distinct values for this custom variable with same metrics.

This new report could be displayed in "Visitors > Custom Variables"

Custom data: JS API piwik.js

• new .setCustomData(index, name, value)
• new .getCustomData(name)

We should allow for up to 3 custom variables (GA does 5 but 3 will be fine!).

This up to 3 (key,value) pairs will be stored in 1st party cookie.

For a given index, name should always be the same.

Scope of cookie is session scope (dies when browser dies).

The content of the cookie is passed to piwik.php (c1n=custom_name_1&c1v=custom_value_1 etc.?).

The name,value pairs should be limited in length to not excell the URL limit (they are send on each page view along with current URL, page name and possibly url ref and more).

Setting a custom var with the same slot in a subsequent page view will overwrite previous value set.

Also update PHP Tracker API with these new methods.

Custom data: Schema update

Add 6 fields to the log_visit table: custom_name_1, custom_value_1, custom_name_2, custom_value_2, etc.

name and value are the parameters to JS function piwik.setCustomVar(1, name, value)

Custom data: Tracker update

New visitor and existing visitor will set the custom variable if passed.

Custom data: Archiving

The new reports (see above) require couple new queries.

Custom data: API

New API function released to report the custom data names and values, similar to existing functions.

Performance impact

When no custom var is set, this should not affect the speed (ie. no performance regression in normal use case). Check this is the case.

For each custom variable, distinct values should be kept low whenever possible (eg. 10 or 20 disctinct values).

Nice to have

• Update Live! widget and visitor log to show the custom variables set

Please submit any feedback/question.

[vipsoft]

If I understand correctly, this busts the existing setCustomData() by moving the key-value storage into piwik.js.

What is "index"? Is that a namespace? (e.g., corresponds to the Plugin's name?) Shouldn't it also be present in getCustomData()?

I suggest we keep the existing json_encode() rather than splitting up into separate parameters.

[vipsoft]

I think there are a variety of scope for custom data:

• this tracking request
• this page
• this session
• persistent

[matt]

Having several scopes would be a feature request, for now this proposal is implementing the 'session' scope (similar to GA session scope)

[vipsoft]

The current implementation provides the first two scopes.

[matt]

Replying to vipsoft:

The current implementation provides the first two scopes.

It provides 'potential' page level scope, if the custom data was recorded (which is not the case at this stage). However this ticket will actually implement it for the session level (ie. update schema, new report & API). I think we agree :)

See interesting article at  http://www.roirevolution.com/blog/2010/12/ecommerce-with-custom-variables.php

[matt]

(In [3780]) Refs #1984 - New plugins CustomVariables

• Now possible to track up to 5 Custom variables per Visit. Also, Goal conversions will be reported 'By custom variable name & value'
• New report in Visitors > Custom Variables, new CustomVars API
• Updated schema
• Updated PiwikTracker PHP api to allow setting the name,value pairs

Code style

• Refatoring some API code, and Archiving queries
• Changing text from 'segment' to 'dimension' as this is better description + we want to build actual segmentation later
• removing getJs calls in some plugins since they only included sparkline.js, moved to CoreHome

[matt]

To finish the ticket, this is pending first party cookie implementation JS API updates required for the following:

• setCustomVariables(nameId, name, value, persistVisitor = false)
  • nameId, name and value are all required parameters and non empty
  • if persistVisitor, stores the name,value pairs in the _pk_id cookie so they are persisted across visits
  • if persistVisitor is false, which is the default, the custom variables will be set in the _pk_ses cookie, and be reset on each visit.

• getCustomVariable( nameId )
  • Returns the custom variable in _pk_ses cookie, or in _pk_id cookie (in this order in case the value is defined in both)
  • Useful to build like a state machine in the cookies, for example to set a visitor status in Javascript, where the status would evolve based on the previous status (visitor -> engaged -> buyer -> customer).

• Documentation for these new methods + put up a documentation with example use case for the Custom Variable feature

• Show custom variables in Live!

[vipsoft]

In setCustomVariables, what's the difference between nameId and name?

Assuming nameId, name, or value can contain '.', it looks like a change in the #409 requirements (i.e., store an associative array in the cookie).

[matt]

basically, nameId and name must always match, so that we always record the custom variable name,value in the same 'slot' in the database. This is similar to GA implentation.

nameId is integer, name and value can contain '.', but maybe we can just replace by _ or something. But yes it is probably similar to storing an array, how do you implement the cookie storage at the moment?

[vipsoft]

At the moment, the cookie storage isn't abstracted because in #409, we are only storing simple values (dot-separated, in the case of _pk_id).

[vipsoft]

(In [3783]) refs #409 - first party cookies

• API changes:
  • added: setCookieNamePrefix(cookieNamePrefix)
  • added: setCookieDomain(domain)
  • added: setCookiePath(path)
  • added: setVisitorCookieTimeout(timeout) - defaults to 2 years since last page view
  • added: setSessionCookieTimeout(timeout) - defaults to 30 minutes since last activity
  • added: setReferralCookieTimeout(timeout) - defaults to 6 months from the first visit
  • added: setConversionAttributionFirstReferer(enable)
  • added: getVisitorId()
    • for asynchronous tracking, use:

      	var visitorId;
      
      	_paq.push(function () {
      		visitorId = this.getVisitorId();
      	});
      

• Cookie notes:
  • The default cookie path is '/'. This might be viewed as a potentially insecure default because it allows cookies to be shared across directories on the same domain. (Again, see the social network example.) This is unfortunately, a necessity. If we leave the path blank, the behaviour is undefined (i.e., browser or browser-version dependent). For example, earlier versions of Firefox would default to '/'; later versions default to the origin path.
  • I was hoping to avoid this, but I added a hash to the cookie content similar to GA's setAllowHash(). This is needed for two reasons:
    1. Cookies are uniquely identified by the tuple (key,domain,path). Hashing only the domain is a bug. (See "social network website" use case.)
    2. There's a long-standing cookie+subdomain bug in Firefox (Gecko) dating back to 1.0 that leaks cookies from "example.com" (not ".example.com") to "xyz.example.com". @see  https://bugzilla.mozilla.org/show_bug.cgi?id=363872
• changed internal setCookie() method to take expiry time in milliseconds (was days)
• removed internal dropCookie() method as it was never used

@todo Missing unit tests and cross browser testing

refs #739 - piwik.js improvements

• jslint 2011-01-09
• new unit tests (integrated jslint, is_a functions, sha1(), utf8_encode(), etc)
• use ECMAScript String.substring() instead of non-standard (although widely supported) String.substr()
• implement domainFixup() so "example.com" and "example.com." are equivalent

• API changes:
  • added: killFrame() - a frame buster
  • added: redirectFile( url ) - redirect if browsing off-line, aka file: buster; url is where to redirect to
  • added: setHeartBeatTimer( delay ) - send heart beat 'delay' milliseconds after initial trackPageView(); set to 0 to disable
  • removed: piwik_log() - legacy tracking code; see trackLink()
  • removed: piwik_track() - legacy tracking code; see trackPageView()
  • removed: setDownloadClass() - deprecated; see setDownloadClasses()
  • removed: setLinkClass() - deprecated; see setLinkClasses()

refs #752 - track middle mouse button clicks (via mousedown+mouseup pseudo-click handler); defaults to tracking true "clicks"

• API changes:
  • modified: addListener( element, enablePseudoClickHandler = false )
  • modified: enableLinkTracking( enablePseudoClickHandler = false )

refs #1984 - custom variables vs custom data

@todo These are just stubs.

• API changes:
  • added: setCustomVar(slotId, key, value, opt_scope) - scope is 1 (visitor), 2 (sesson), 3 (page)
  • added: getCustomVar(slotId)
  • added: deleteCustomVar(slotId)

• API changes for consistency:
  • added: setCustomVar(slotId, obj, opt_scope)
  • added: setCustomData(key, value)
  • for the equivalent of deleteCustomData(), use:

        tracker.setCustomData(null);
    

[matt]

(In [3787]) Refs #1984

• records custom variables in a session cookie
• the data is sent to piwik as stringified json
• and recorded in the cookie as json. I had to add json_parse which is quite large (!!) but maybe we can remove some code inside it that we don't use (since we just want to parse strings and int... ?)

Still to do:

• make the deleteCustomVariable work on server side (currently is ignored)
• more testing
• setCustomVar, keep/change?

[vipsoft]

(In [3790]) refs #1984

• missing var declarations (e.g., unqualified 'name' would refer to global scope window.name; so better to avoid using it)
• syntax error (missing + operator in constructing request)
• fix comments to match implementation

[matt]

(In [3882]) Refs #1984

• removing setCustomData feature
• updated JS doc piwik.org
• deprecated setCustomData getCustomData and the customData parameter to few functions. Now plugin developers can use the Custom Variables array

[matt]

(In [3883]) Refs #1984 - Deleting custom variables feature, adding doc in piwik.org

[vipsoft]

I would have appreciated a heads-up before [3882]. setCustomVariable() doesn't implement request/page scope; deprecating customData wasn't discussed and doing so, breaks the unit tests in tests/javascript/index.php.

[matt]

(In [3917]) Refs #1984 - Displaying custom variables

[vipsoft]

(In [3939]) refs #409:

• always use Crockford's JSON module (renamed to JSON2) to workaround broken "native implementations"
• add JSON unit tests
• revert [3893] and [3900]; rewrite getVisitorId() per comment:80
• refactor browser feature detection for fingerprinting (used to generate uuid)
• setDomains() now takes either '*.domain' or '.domain'
• Safari emits warnings for Content-Length and Connection as "unsafe headers" in XHR POST request

refs #1984:

• partially revert [3882] in order for the unit tests to run
• fix inconsistency in getCustomVariable() depending on whether it is loaded from memory or from a cookie

refs #2078 Webkit bug ("Failed to load resource") when link target is the current window/tab

• requires further discussion because the workaround may not be desirable behavior, i.e.,

  if ((new RegExp('WebKit')).test(navigatorAlias.userAgent)
      && (!sourceElement.target.length || sourceElement.target === '_self')
      && linkType === 'link')
  {
      // open outlink in a new window
      sourceElement.target = '_blank';
  }
  

[hass]

Follow up issue  http://dev.piwik.org/trac/ticket/2206 because of setCustomData() do not exists anymore.

[hass]

Documentation needs update because of

setDomains() now takes either '*.domain' or '.domain'.

[hass]

Documentation also needs update for:

added: setCustomVar(slotId, key, value, opt_scope) - scope is 1 (visitor), 2 (sesson), 3 (page)

[hass]

I mean "opt_scope" need to be documented.

[vipsoft]

Thanks hass.

Deprecating setCustomData() was a bit premature. setCustomData() is still needed for unit tests, so it still exists in piwik.js. Also, setCustomVariable() doesn't implement opt_scope yet.

[hass]

Ok, but setCustomData() need to be kept - not only for unit tests, please.

I have integrated setCustomVariables in the Piwik for Drupal module, but still have some questions.

1. Are there any length limitations like we have them in Google Analytics? I have currently taken over the code from GA, what brings a limitation of 64 chars per variable including the "name". I guess Google have this limitation because of URL length. You may have not documented it, but URLs may become really long and than the server URL limits may lose data if the tracker url is toooo long. Many servers have a URL length limit of 2048/4096 bytes for GET requests.

2. Compared to Google Analytics - are Piwik Custom variables now "visitor" or "session" or "page"? I documented as visitor in the module, but from confusing comments above it may be "session". Can someone confirm what it really is and may document this in the JS API docs, please?

[vipsoft]

name and value are limited to 100 chars each

use setRequestMethod('POST') to workaround GET limits

scope of custom variable is equivalent to GA "session" scope (visit)

[matt]

I updated the Custom Variables documentation with the following

Custom Variables are stored in a first party cookie that is valid only for the current Visit. When the same Visitor returns to your site, the Custom Variables will be empty. If you want to persist a Custom Variable to a Visitor, for example "UserType = Customer", you must call the Javascript function setCustomVariable( index, name, value ) at least once during each Visit.

Custom Variables names and values are limited to 100 characters in length each. We recommend to store values that are as small as possible to ensure that the Piwik Tracking request URL doesn't go over the URL limit for the webserver or browser.

[hass]

Sorry for being nitpicking, but 100 chars after URL encoding or before?

THX

[matt]

100 chars limit before URL encoding. But in any case please don't push the limit, we highly recommend to use values as small as possible