Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom Variables support: new JS API and new reports #1984

Closed
mattab opened this issue Jan 8, 2011 · 30 comments
Closed

Custom Variables support: new JS API and new reports #1984

mattab opened this issue Jan 8, 2011 · 30 comments
Assignees
Labels
Critical Indicates the severity of an issue is very critical and the issue has a very high priority. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone

Comments

@mattab
Copy link
Member

mattab commented Jan 8, 2011

FEATURE RELEASED in Piwik 1.2

See the documentation in: Web analytics - Tracking Custom Variables

The goal of this feature is to put visits into buckets (segments) via the Piwik JS API. Then access reports about these custom name/values just like other reports in Piwik.

Custom data: Use cases

  • Count Logged in users VS logged out visitors (or Customers VS prospects)
  • Count custom things per visit that are not included in Piwik standard metrics, eg. 'games played', 'comments left', or anything etc.
  • Track position of keywords in google search page (using new parameter found in ref url recently after quick parsing of ref url in JS)

Custom data: New default reports

  • Custom Variable: list all distinct names, for each list visits, pages, time on site, bounce rate, etc.
  • On click on any variable name, list all distinct values for this custom variable with same metrics.

This new report could be displayed in "Visitors > Custom Variables"

Custom data: JS API piwik.js

  • new .setCustomData(index, name, value)
  • new .getCustomData(name)

We should allow for up to 3 custom variables (GA does 5 but 3 will be fine!).

This up to 3 (key,value) pairs will be stored in 1st party cookie.

For a given index, name should always be the same.

Scope of cookie is session scope (dies when browser dies).

The content of the cookie is passed to piwik.php (c1n=custom_name_1&c1v=custom_value_1 etc.?).

The name,value pairs should be limited in length to not excell the URL limit (they are send on each page view along with current URL, page name and possibly url ref and more).

Setting a custom var with the same slot in a subsequent page view will overwrite previous value set.

Also update PHP Tracker API with these new methods.

Custom data: Schema update

Add 6 fields to the log_visit table: custom_name_1, custom_value_1, custom_name_2, custom_value_2, etc.

name and value are the parameters to JS function piwik.setCustomVar(1, name, value)

Custom data: Tracker update

New visitor and existing visitor will set the custom variable if passed.

Custom data: Archiving

The new reports (see above) require couple new queries.

Custom data: API

New API function released to report the custom data names and values, similar to existing functions.

Performance impact

When no custom var is set, this should not affect the speed (ie. no performance regression in normal use case). Check this is the case.

For each custom variable, distinct values should be kept low whenever possible (eg. 10 or 20 disctinct values).

Nice to have

  • Update Live! widget and visitor log to show the custom variables set

Please submit any feedback/question.

@robocoder
Copy link
Contributor

If I understand correctly, this busts the existing setCustomData() by moving the key-value storage into piwik.js.

What is "index"? Is that a namespace? (e.g., corresponds to the Plugin's name?) Shouldn't it also be present in getCustomData()?

I suggest we keep the existing json_encode() rather than splitting up into separate parameters.

@robocoder
Copy link
Contributor

I think there are a variety of scope for custom data:

  • this tracking request
  • this page
  • this session
  • persistent

@mattab
Copy link
Member Author

mattab commented Jan 9, 2011

Having several scopes would be a feature request, for now this proposal is implementing the 'session' scope (similar to GA session scope)

@robocoder
Copy link
Contributor

The current implementation provides the first two scopes.

@mattab
Copy link
Member Author

mattab commented Jan 9, 2011

Replying to vipsoft:

The current implementation provides the first two scopes.

It provides 'potential' page level scope, if the custom data was recorded (which is not the case at this stage). However this ticket will actually implement it for the session level (ie. update schema, new report & API). I think we agree :)

See interesting article at http://www.roirevolution.com/blog/2010/12/ecommerce-with-custom-variables.php

@mattab
Copy link
Member Author

mattab commented Jan 19, 2011

(In [3780]) Refs #1984 - New plugins CustomVariables

  • Now possible to track up to 5 Custom variables per Visit. Also, Goal conversions will be reported 'By custom variable name & value'
  • New report in Visitors > Custom Variables, new CustomVars API
  • Updated schema
  • Updated PiwikTracker PHP api to allow setting the name,value pairs

Code style

  • Refatoring some API code, and Archiving queries
  • Changing text from 'segment' to 'dimension' as this is better description + we want to build actual segmentation later
  • removing getJs calls in some plugins since they only included sparkline.js, moved to CoreHome

@mattab
Copy link
Member Author

mattab commented Jan 19, 2011

To finish the ticket, this is pending first party cookie implementation JS API updates required for the following:

  • setCustomVariables(nameId, name, value, persistVisitor = false)
    • nameId, name and value are all required parameters and non empty
    • if persistVisitor, stores the name,value pairs in the _pk_id cookie so they are persisted across visits
    • if persistVisitor is false, which is the default, the custom variables will be set in the _pk_ses cookie, and be reset on each visit.
  • getCustomVariable( nameId )
    • Returns the custom variable in _pk_ses cookie, or in _pk_id cookie (in this order in case the value is defined in both)
    • Useful to build like a state machine in the cookies, for example to set a visitor status in Javascript, where the status would evolve based on the previous status (visitor -> engaged -> buyer -> customer).
  • Documentation for these new methods + put up a documentation with example use case for the Custom Variable feature
  • Show custom variables in Live!

@robocoder
Copy link
Contributor

In setCustomVariables, what's the difference between nameId and name?

Assuming nameId, name, or value can contain '.', it looks like a change in the #409 requirements (i.e., store an associative array in the cookie).

@mattab
Copy link
Member Author

mattab commented Jan 19, 2011

basically, nameId and name must always match, so that we always record the custom variable name,value in the same 'slot' in the database. This is similar to GA implentation.

nameId is integer,
name and value can contain '.', but maybe we can just replace by _ or something. But yes it is probably similar to storing an array, how do you implement the cookie storage at the moment?

@robocoder
Copy link
Contributor

At the moment, the cookie storage isn't abstracted because in #409, we are only storing simple values (dot-separated, in the case of _pk_id).

@robocoder
Copy link
Contributor

(In [3783]) refs #409 - first party cookies

  • API changes:
    • added: setCookieNamePrefix(cookieNamePrefix)
    • added: setCookieDomain(domain)
    • added: setCookiePath(path)
    • added: setVisitorCookieTimeout(timeout) - defaults to 2 years since last page view
    • added: setSessionCookieTimeout(timeout) - defaults to 30 minutes since last activity
    • added: setReferralCookieTimeout(timeout) - defaults to 6 months from the first visit
    • added: setConversionAttributionFirstReferer(enable)
    • added: getVisitorId()
      • for asynchronous tracking, use:
    var visitorId;

    _paq.push(function () {
        visitorId = this.getVisitorId();
    });
  • Cookie notes:

    • The default cookie path is '/'. This might be viewed as a potentially insecure default because it allows cookies to be shared across directories on the same domain. (Again, see the social network example.) This is unfortunately, a necessity. If we leave the path blank, the behaviour is undefined (i.e., browser or browser-version dependent). For example, earlier versions of Firefox would default to '/'; later versions default to the origin path.
    • I was hoping to avoid this, but I added a hash to the cookie content similar to GA's setAllowHash(). This is needed for two reasons:
      1. Cookies are uniquely identified by the tuple (key,domain,path). Hashing only the domain is a bug. (See "social network website" use case.)
      2. There's a long-standing cookie+subdomain bug in Firefox (Gecko) dating back to 1.0 that leaks cookies from "example.com" (not ".example.com") to "xyz.example.com". @see https://bugzilla.mozilla.org/show_bug.cgi?id=363872
  • changed internal setCookie() method to take expiry time in milliseconds (was days)

  • removed internal dropCookie() method as it was never used

    @todo Missing unit tests and cross browser testing

refs #739 - piwik.js improvements

  • jslint 2011-01-09
  • new unit tests (integrated jslint, is_a functions, sha1(), utf8_encode(), etc)
  • use ECMAScript String.substring() instead of non-standard (although widely supported) String.substr()
  • implement domainFixup() so "example.com" and "example.com." are equivalent
  • API changes:
    • added: killFrame() - a frame buster
    • added: redirectFile( url ) - redirect if browsing off-line, aka file: buster; url is where to redirect to
    • added: setHeartBeatTimer( delay ) - send heart beat 'delay' milliseconds after initial trackPageView(); set to 0 to disable
    • removed: piwik_log() - legacy tracking code; see trackLink()
    • removed: piwik_track() - legacy tracking code; see trackPageView()
    • removed: setDownloadClass() - deprecated; see setDownloadClasses()
    • removed: setLinkClass() - deprecated; see setLinkClasses()

refs #752 - track middle mouse button clicks (via mousedown+mouseup pseudo-click handler); defaults to tracking true "clicks"

  • API changes:
    • modified: addListener( element, enablePseudoClickHandler = false )
    • modified: enableLinkTracking( enablePseudoClickHandler = false )

refs #1984 - custom variables vs custom data

@todo These are just stubs.

  • API changes:
    • added: setCustomVar(slotId, key, value, opt_scope) - scope is 1 (visitor), 2 (sesson), 3 (page)
    • added: getCustomVar(slotId)
    • added: deleteCustomVar(slotId)
  • API changes for consistency:
    • added: setCustomVar(slotId, obj, opt_scope)
    • added: setCustomData(key, value)
    • for the equivalent of deleteCustomData(), use:
    tracker.setCustomData(null);

@mattab
Copy link
Member Author

mattab commented Jan 21, 2011

(In [3787]) Refs #1984

  • records custom variables in a session cookie
  • the data is sent to piwik as stringified json
  • and recorded in the cookie as json. I had to add json_parse which is quite large (!!) but maybe we can remove some code inside it that we don't use (since we just want to parse strings and int... ?)

Still to do:

  • make the deleteCustomVariable work on server side (currently is ignored)
  • more testing
  • setCustomVar, keep/change?

@robocoder
Copy link
Contributor

(In [3790]) refs #1984

  • missing var declarations (e.g., unqualified 'name' would refer to global scope window.name; so better to avoid using it)
  • syntax error (missing + operator in constructing request)
  • fix comments to match implementation

@mattab
Copy link
Member Author

mattab commented Feb 13, 2011

(In [3882]) Refs #1984

  • removing setCustomData feature
  • updated JS doc piwik.org
  • deprecated setCustomData getCustomData and the customData parameter to few functions. Now plugin developers can use the Custom Variables array

@mattab
Copy link
Member Author

mattab commented Feb 13, 2011

(In [3883]) Refs #1984 - Deleting custom variables feature, adding doc in piwik.org

@robocoder
Copy link
Contributor

I would have appreciated a heads-up before [3882]. setCustomVariable() doesn't implement request/page scope; deprecating customData wasn't discussed and doing so, breaks the unit tests in tests/javascript/index.php.

@mattab
Copy link
Member Author

mattab commented Feb 17, 2011

(In [3917]) Refs #1984 - Displaying custom variables

@robocoder
Copy link
Contributor

(In [3939]) refs #409:

  • always use Crockford's JSON module (renamed to JSON2) to workaround broken "native implementations"
  • add JSON unit tests
  • revert [and 3900; rewrite getVisitorId() per comment:80
  • refactor browser feature detection for fingerprinting (used to generate uuid)
  • setDomains() now takes either '*.domain' or '.domain'
  • Safari emits warnings for Content-Length and Connection as "unsafe headers" in XHR POST request

refs #1984:

  • partially revert [3882] in order for the unit tests to run
  • fix inconsistency in getCustomVariable() depending on whether it is loaded from memory or from a cookie

refs #2078 Webkit bug ("Failed to load resource") when link target is the current window/tab

  • requires further discussion because the workaround may not be desirable behavior, i.e.,
if ((new RegExp('WebKit')).test(navigatorAlias.userAgent)
    && (!sourceElement.target.length || sourceElement.target === '_self')
    && linkType === 'link')
{
    // open outlink in a new window
    sourceElement.target = '_blank';
}

@anonymous-matomo-user
Copy link

Follow up issue #2206 because of setCustomData() do not exists anymore.

@anonymous-matomo-user
Copy link

Documentation needs update because of

setDomains() now takes either '*.domain' or '.domain'.

@anonymous-matomo-user
Copy link

Documentation also needs update for:

added: setCustomVar(slotId, key, value, opt_scope) - scope is 1 (visitor), 2 (sesson), 3 (page)

@anonymous-matomo-user
Copy link

I mean "opt_scope" need to be documented.

@robocoder
Copy link
Contributor

Thanks hass.

Deprecating setCustomData() was a bit premature. setCustomData() is still needed for unit tests, so it still exists in piwik.js. Also, setCustomVariable() doesn't implement opt_scope yet.

@anonymous-matomo-user
Copy link

Ok, but setCustomData() need to be kept - not only for unit tests, please.

I have integrated setCustomVariables in the Piwik for Drupal module, but still have some questions.

  1. Are there any length limitations like we have them in Google Analytics? I have currently taken over the code from GA, what brings a limitation of 64 chars per variable including the "name". I guess Google have this limitation because of URL length. You may have not documented it, but URLs may become really long and than the server URL limits may lose data if the tracker url is toooo long. Many servers have a URL length limit of 2048/4096 bytes for GET requests.
  2. Compared to Google Analytics - are Piwik Custom variables now "visitor" or "session" or "page"? I documented as visitor in the module, but from confusing comments above it may be "session". Can someone confirm what it really is and may document this in the JS API docs, please?

@robocoder
Copy link
Contributor

name and value are limited to 100 chars each

use setRequestMethod('POST') to workaround GET limits

scope of custom variable is equivalent to GA "session" scope (visit)

@mattab
Copy link
Member Author

mattab commented Mar 20, 2011

I updated the Custom Variables documentation with the following

Custom Variables are stored in a first party cookie that is valid only for the current Visit. When the same Visitor returns to your site, the Custom Variables will be empty. If you want to persist a Custom Variable to a Visitor, for example "UserType = Customer", you must call the Javascript function setCustomVariable( index, name, value ) at least once during each Visit.

Custom Variables names and values are limited to 100 characters in length each. We recommend to store values that are as small as possible to ensure that the Piwik Tracking request URL doesn't go over the URL limit for the webserver or browser.

@anonymous-matomo-user
Copy link

Sorry for being nitpicking, but 100 chars after URL encoding or before?

THX

@mattab
Copy link
Member Author

mattab commented Mar 21, 2011

100 chars limit before URL encoding. But in any case please don't push the limit, we highly recommend to use values as small as possible

@mattab mattab added this to the Piwik 1.2 milestone Jul 8, 2014
@yagavi
Copy link

yagavi commented Sep 10, 2015

how to show my sql table in piwik dashboard..which file i have to edit?

@kanzy21
Copy link

kanzy21 commented Jan 8, 2018

Can anyone provide a example for custom variables such as-- getting a username or email as custom variable on form submit

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Critical Indicates the severity of an issue is very critical and the issue has a very high priority. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Development

No branches or pull requests

5 participants