Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Ping requests to report a better 'time on page' #2041

Closed
mattab opened this issue Jan 21, 2011 · 63 comments
Closed

Support Ping requests to report a better 'time on page' #2041

mattab opened this issue Jan 21, 2011 · 63 comments
Assignees
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone

Comments

@mattab
Copy link
Member

mattab commented Jan 21, 2011

Currently, the last page view is set with a 'time on page' of zero seconds.

Instead, we could regularly ping the Piwik server with a request that notifies Piwik that the user is still on the website.

This request should not record new page view, only increase the visit_total_time counter.

Also, we should only throw this request when the user is actually looking at the page, not doing something else in some other tabs.

The risk with this feature is that instead of improving the report accuracy, we end up inflating numbers: a lot of users leave pages open in other tabs for hours and don't do anything about it. We should heavily test to ensure we never throw a request on a page where the user didn't interact with recently. Some interactions we can maybe detect: mouse click, mouse move, etc.

Of course, we should not make everything slow by adding handlers on mouse move, etc.

Also, once this feature is implemented, a new checkbox should be added to the Advanced section in the 'Tracking Code' admin page. Checking it will generate JS code that enables this feature.

@robocoder
Copy link
Contributor

(In [3788]) fixes #739 - addEventListeners() on mouseup and mousedown; good catch, Matt!

  • utf8_encode() - required to convert multibyte char to string of bytes; for unit testing, it needs to be a separate function
  • killFrame() - would be called after a trackPageView(); used to prevent one's site from being frame by third-parties; there are tradeoffs (as you point) out, so it is not enabled by default

refs #2041 - setHeartBeatTimer - uses setTimeout, so it happens only once (vs setInterval is recurring); only happens on the landing page; intended to be used to improve bounce rate report; changed it to send ping=1 in the request

@robocoder
Copy link
Contributor

I reworked setHeartBeatTimer() in [3830]. It now takes an initalPingDelay and recurringHeartBeatDelay as parameters. By default, this is disabled.

@robocoder
Copy link
Contributor

The initial ping request contains "ping=1". Use this to improve bounce rates and visit time.

Subsequent heartbeat requests (if recurringDelay != 0) contain "hb=1".

I'm going to defer adding event handlers as there are issues with propagating some events like keyclicks.

Mental notes to self:

  • if you only send a beat when there's activity within the last n seconds, then on average, you'll expect the total time to be overstated by n/2 secs
  • to offset/compensate, you can instead send a beat if there's been activity in the last n/2 seconds
  • if there's no activity for n secs, do you stop checking for activity or stop sending beats?
  • what if activity resumes again? how long is too long ago to to resume beats?

@mattab
Copy link
Member Author

mattab commented Feb 3, 2011

how do you define 'activity'? Will the ping be sent if user is looking at another tab, another window, or even away from computer?

@robocoder
Copy link
Contributor

activity: mousemove, click, mouseup, mousedown, mousewheel, scroll, DOMMouseScroll, resize, keypress, keydown, keyup, focus, and blur

  • need this many because of browser differences an the possibility of interference from other eveny handlers
  • handler simply sets the lastActivity timestamp

Yes, ping would be sent. That's why there's a check for activity. If there's no activity, I propose to no longer check.

I'll commit my changes after some more testing.

@robocoder
Copy link
Contributor

(In [3838]) refs #2041 - setHeartBeatTimer() now takes the minimumVisitLength and the heartBeatPeriod (time between pings) as parameters; this is simpler than initial ping + recurring hb; requests contain "ping=1".

also updated to jslint 2011-02-03

@robocoder
Copy link
Contributor

(In [3839]) refs #2041 - rollback buggy n/2 logic

@robocoder
Copy link
Contributor

(In [3840]) refs #2041 - undo previous optimizaton to ping on first page of a new session

@robocoder
Copy link
Contributor

(In [3841]) refs #2041 - add guard

@robocoder
Copy link
Contributor

Client-side changes are done, so it would be nice to have the server-side change done in time for 1.2 but we can roll into the next milestone.

@mattab
Copy link
Member Author

mattab commented Feb 9, 2011

How often are the activity tests done? every second? Just thinking of keeping JS performance overhead low with this new feature.

I wont be able to do this in time for 1.2, so it will go in 1.3. Is pinging disabled by default? thx

@robocoder
Copy link
Contributor

Yes, disabled by default; frequency is user-defined.

@mattab
Copy link
Member Author

mattab commented Feb 17, 2011

Before enabling this by default I think we could try it on some heavy websites, that are using GA, other libraries (jquery eg.) and doing Javascript heavily. Then we could see if Piwik adds any unexpected performance overhead.

For example it would be nice to say: Piwik will query a few JS variables every second to assess if the visitor is active or not, but will not conflict with any other library or Javascript and should not affect performance at all.

What do you think? for example, have you tried it on a a running piwik loading the dashboard full of widgets.

@mattab
Copy link
Member Author

mattab commented Apr 28, 2011

I think, disabled by default, but easy to enable, this would make it a really cool feautre. Also, should be made available via the 'advanced' javascript tag generator see #1845

@anonymous-matomo-user
Copy link

Just wondering where we are on this. I'm in the process of switching several sites from GA to Piwik, and this feature is pretty important to me. Does 1.x mean it's going to be in an indefinite release between now and 2?

@mattab
Copy link
Member Author

mattab commented Oct 5, 2011

Polrbear, yes currently not roadmapped for 1.7 but it might change

@anonymous-matomo-user
Copy link

I'm interessed too by this feature. I have activate it on my website, but the presentation in the server side, isn't perfect (the ping is show as a new page visit).

I hope, that it can be made for 1.7.

Thanks.

@mattab
Copy link
Member Author

mattab commented Dec 16, 2011

Another bug report: http://forum.piwik.org/read.php?2,83913,page=1#msg-84060

NOTE: This feature does not work currently

@mattab
Copy link
Member Author

mattab commented Sep 17, 2012

Proof Reading the upcoming book about Piwik, it's really good.

it gives me an idea related to this feature:

  • we should add a new parameter at the trackPageView function level, that users could use to trigger events/pageviews that will allow to disable the default "lower bounce rate". When "pinging" the page, we don't want to lower bounce rate by default, other users might want to do this as well in some cases.

We could allow for this other addon: send a fake Pageview 15seconds after arrival: http://analytics.blogspot.com/2012/07/tracking-adjusted-bounce-rate-in-google.html - might require the ping query so it doesn't inflate Page view counts, but changes Bounce rates for all other metrics.

@mattab
Copy link
Member Author

mattab commented Dec 6, 2012

Note from Anthon:

The original thinking was, fire a ping once the page has been open some minimum amount of time, and keep pinging as long as there's some mouse activity. Basically, the goal would be to report lower bounce rates (ie if the first and only page visited) and more accurate time on site for the last page visited on a site.

The Page Visibility and traceCallback changes means the ping isn't sent until a tab gains focus and has been open the minimum amount of time. Further pings require some activity while the tab is open.

We probably need a way to restart the heartbeat if the tab regains visibility. Also, the inactivity logic doesn't work with passive activities (eg streaming media), but I think could be addressed by adding an API for players. (Are you using event tracking for this?)

This obviously isn't feature complete, so if anyone has ideas on how this should work or could work better, feel free to share your thoughts.

@mattab
Copy link
Member Author

mattab commented Dec 14, 2012

I'm very interested in this feature but unfortunately time is missing to implement it for now, unless someone jumps in and produces a patch we can use! Or if you can sponsor the feature please get in touch at sponsors at piwik.org

@kevinoid
Copy link

kevinoid commented Jan 9, 2014

Great feature, looking forward to it. (Just commenting to add myself to the CC list.)

@mattab
Copy link
Member Author

mattab commented May 25, 2014

See also: #5208 Detect when a user is actively browsing or using the webpage, for more accurate Time on page

@mattab
Copy link
Member Author

mattab commented Jun 3, 2015

Summary of discussion so far:

  • piwik.js: sendRequest mechanism can be reused
  • need to decide on "ping" strategy eg. ping first time after 20s, then ping less and less often eg. [20, 40, 60, 100, 160, 260, ....] ?
  • we need to ping only when the page is actively viewed (we need to detect active tab/window somehow, maybe Page Visibility API can do this?)
  • define & document a new Tracking API parameter, eg ping=1 (or better name!)

Maybe you have more thoughts, cc @diosmosis

@hpvd
Copy link

hpvd commented Jun 3, 2015

thanks for the summary. Hopefully it's ok when I would like to ask to clarify the target even more:

  1. will this feature be an additional thing to other already available "site is active" signals like events (see user posts of robocoder, masteranalyze and me) so it mainly brings new active window detection?
    And Piwik host will combine this and notice which signal came "last"?

  2. or will it use these available signals to determine site is active, add active window detection and send a ping if there is any activity?

  3. or nothing of these ;-)

Many thanks!

@mattab
Copy link
Member Author

mattab commented Jun 3, 2015

@hpvd currently there is no feature - the code related to heartbeat timer / configHeartBeatTimer in piwik.js is actually unused we could/should remove it as well!

@masteranalyze
Copy link

Hey guys,
Matt,hpvd,i think the best method is 1 " will this feature be an additional thing to other already available "site is active" signals like events (see user posts of robocoder, masteranalyze and me) so it mainly brings new active window detection?
And Piwik host will combine this and notice which signal came "last"? "

This is how it should be .
That way based on events,and user actions,we will have accurate TIMING+Bouncing rates!

About the : need to decide on "ping" strategy eg. ping first time after 20s, then ping less and less often eg. [20, 40, 60, 100, 160, 260, ....] ? ,ping strategy,i think from 15seconds ,to 15 seconds is okay or even from 20 seconds to 20 seconds,but this also can depends on user shared enverimonths,if the user haves Shared hosting,and this ping in eats lots of resources from the server,cpu,ram,i think their should be an recomandation for shared ,vps and dedicated server.
An dedicated server can have no problem to handle from 15 seconds to 15 seconds,but an shared server might have an problem,and maybe is good to put it from 30 or 60 seconds .
Anyhow an minimum value should be 15,and not less,and we let the user to decide.

we need to ping only when the page is actively viewed (we need to detect active tab/window somehow, maybe Page Visibility API can do this?)

Exactly,we need to ping in,only when the page is actively viewed,and this can be done with the events,that we talked more earlier.If they are no events,basically the page is maybe left in the tab for hours,and user is watching an youtube clip,so it`s not in your page tab,so basically this will just inflate the numbers witch is not good.

The events should also be there,and easily accesibile,as options in piwik,and we let the user freedom to decide,maybe some of us only needs some of the events,or all,it depends from case to case.

@matt : define & document a new Tracking API parameter, eg ping=1 (or better name!)

I think tracking api parameter, ping =1 ,is an good name,but what 1 will represent ?
1 will represent that PING is actively ?Or it will ping from 1 second to another ?
Basically 1 should Active Ping In,and 0 - Disabled Ping IN.

Hopefully we will see this in the new piwik 2.14.

@hpvd
Copy link

hpvd commented Jun 8, 2015

"1) = additional thing to other already available "site is active" signals"
that's also my preferred way, and was the reason for creating #7607

(maybe one should taking into account the date this ticket about supporting ping requests was created/ started: "21 Jan 2011" - at this time there was no great "event tracking" available within piwik)

@masteranalyze
Copy link

so this will work,basically on 15 seconds to 15 seconds to the end user pings?
or via events triggers ?

I agree,hpvd,there is no great "Event tracking" available in piwik.

@hpvd
Copy link

hpvd commented Jun 18, 2015

@masteranalyze now standard event tracking is fine for me.
What I was describing was the time the ticket was started...in 2011 there was no event tracking at all: so you had no chance to determine if there is someone on the site.
If you are using event tracking today, you got lots of activity signals from each site during the usage by the visitor. So the ping only needs to be an additional possibility to determine if there is activity.
But of course the signals from the events first have to be included in the determination of active/passive...
@ALL <= do we need an extra ticket for this??
or should it stay be part of #7607?

@masteranalyze
Copy link

@hpvd is clearly that from that time to today,piwik did have an great evolution,but i think it can be even more great,as except the fact the anaytics are private,there is one more good thing and an big advantage for PIWIK - Server Side Tracking,witch is not possible neither in Google Analytics,neither in StatCounter,neither in Clicky or any other stats available on the market today.

The event tracking we have today,of course is more good,that what it was missing in 2011,i totally agree with you,but nowdays for every event tracking,you need to add one more line code into the javascript,one more for another event and so on,it will be alot more easily,if in the event tracking back -end it will be events with options : YES ,NO ,and addiacently the options to it,so everybody can Enable or use them,right now Events are not quite so easily setup for someone who does not write code.
This is for example for : ADJUSTED BOUNCED RATE,but if you dont add it to your code of tracking,it wont track this event,and mostly your bounce rate will be higher,and time spend on the page less.
script
setTimeout("_paq.push(['trackEvent', 'time spent', '>15sec', 'good reading'])",15000);
/script
Maybe we can make an list of all this,scripts for events how they should be,and further on,they can be implemented and be available from backend Event Tracking!

"If you are using event tracking today, you got lots of activity signals from each site during the usage by the visitor." - i totally agree with you,with the pointing of that only people who setup the events and puts additionally script,can track the events,or if the software your using,haves integer piwik,you will have those events,else not.

" So the ping only needs to be an additional possibility to determine if there is activity.
But of course the signals from the events first have to be included in the determination of active/passive..." - like i said and i agree with you,that this ping in,normaly should be an option from backend,that everybody to be able to setup up,for example on 15 seconds,on 30seconds,to be the ping in .
And we also do needs all the Events,in Back END,to check the user activity,if he is active or passive.
In administration of piwik,we don`t have Events ,to be easily setup,with yes or no,and to not have to add,more and more code,into the piwik original tracking code.

hpvd,create an new ticket for PIWIK EVENTS SCRIPTS,and there we can put an list of EVENTS,and the accordingly script,but afterwards,how we manage to put them into the backend of piwik?!
What will be the tracking script for this action : Social actions on page engagement:
Here is the code for google analytics that can be easily converted to piwik to track this events:
http://www.optimizesmart.com/social-interactions-tracking-through-google-analytics/ ?!
I think if we all contribute and make an list,afterwards,that list can be complected along the time!

@hpvd
Copy link

hpvd commented Jun 18, 2015

@masteranalyze
a ticket for collecting standard events and the js code snippets belonging to them
is a great idea for a future improvement - please start the ticket!!

@masteranalyze
Copy link

@hpvd i opened this #8141 ,for posting colection of standard events and js code.
The thing is,how we,do implement them in piwik back-end to be easily accesibile?Even for non-coders.

@diosmosis
Copy link
Member

Fixed in #8069.

@robocoder
Copy link
Contributor

@diosmosis \o/

@jgebal
Copy link

jgebal commented Jul 11, 2015

Hi all,
+1 to the entire idea.
I'm using Piwik for my blog and I continuously see high bounce rates and no time on page for single page views.
And a short suggestion.
The feature is complex so don't try to make it perfect. Just do something.
If the discussion takes 4,5 years, there is something terribly wrong.
Make it Lean, use inspect and adapt. Release it with option to disable/enable the feature and people will give you feedback how it works what works what does not work.
Don't try to figure it all by yourself :)
The lack of any kind of real metric on single page views is really annoying.
Cheers and thanks for all the great job done on Piwik.

@ReessKennedy
Copy link

+1 to this as well. And agreed with @jgebal.

This would be really great option to enable!

@mattab
Copy link
Member Author

mattab commented Jul 22, 2015

@ReessKennedy feel free to post in the other issue: #8225 (this issue here is closed)

@mattab
Copy link
Member Author

mattab commented Dec 18, 2015

Feature is now documented at Accurately measure the time spent on each page

@hpvd
Copy link

hpvd commented Dec 18, 2015

many thanks for documenting this!
Is there a setting how often website send this alive information?
not every body needs it every second, but every 60seconds maybe to long for others...
and of course you may need to find an individual balance between accuracy and server-load

@VincentSC
Copy link

This is documented in the changelog: http://developer.piwik.org/changelog

On 18/12/15 11:43, hpvd wrote:

many thanks for documenting this!
Is there a setting how often website send this alive information?
not every body needs it every second, but every 60seconds maybe to
long for others...
and of course you may need to find an individual balance between
accuracy and server-load


Reply to this email directly or view it on GitHub
#2041 (comment).


V.G.Hindriksen MSc., CEO
StreamComputing BV - http://www.streamcomputing.eu
Location: Amsterdam, Netherlands, Europe
Cell: +31 6 45400456
NEW PHONE NUMBER: +31 854865760
KvK-number (Chamber of Commerce): 61901873
Rabo Bank (IBAN/SEPA): NL89RABO0113885571 - BIC: RABONL2U
OpenCL Newsletter: http://bit.ly/OpenCLnewsletter
OpenCL Twitter : http://twitter.com/StreamComputing
OpenCL Blog : http://www.streamcomputing.eu/blog
PGP Public key : http://bit.ly/VincentHindriksenPGP


@hpvd
Copy link

hpvd commented Dec 18, 2015

thanks for pointing there.
_To use this feature use tracker.enableHeartBeatTimer(); or _paq.push(['enableHeartBeatTimer']);. By default, a ping request will be sent every 15 seconds. You can specify a custom ping delay (in seconds) by passing an argument, eg, tracker.enableHeartBeatTimer(10); or paq.push(['enableHeartBeatTimer', 10]);.
If this is working fine it should be added to docu as well..

@mattab
Copy link
Member Author

mattab commented Dec 20, 2015

@hpvd
Copy link

hpvd commented Dec 21, 2015

Thanks very much!

@masteranalyze
Copy link

good work @mattab and the team!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Development

No branches or pull requests