Opened 3 years ago

Last modified 3 weeks ago

#2041 new New feature

Support Ping requests to report a better 'time on page'

Reported by: matt Owned by:
Priority: normal Milestone: 2.x - The Great Piwik 2.x Backlog
Component: Core Keywords:
Cc: Sensitive: no

Description (last modified by capedfuzz)

Currently, the last page view is set with a 'time on page' of zero seconds.

Instead, we could regularly ping the Piwik server with a request that notifies Piwik that the user is still on the website.

This request should not record new page view, only increase the visit_total_time counter.

Also, we should only throw this request when the user is actually looking at the page, not doing something else in some other tabs.

The risk with this feature is that instead of improving the report accuracy, we end up inflating numbers: a lot of users leave pages open in other tabs for hours and don't do anything about it. We should heavily test to ensure we never throw a request on a page where the user didn't interact with recently. Some interactions we can maybe detect: mouse click, mouse move, etc.

Of course, we should not make everything slow by adding handlers on mouse move, etc.

Also, once this feature is implemented, a new checkbox should be added to the Advanced section in the 'Tracking Code' admin page. Checking it will generate JS code that enables this feature.

Change History (31)

comment:1 Changed 3 years ago by vipsoft (robocoder)

(In [3788]) fixes #739 - addEventListeners() on mouseup and mousedown; good catch, Matt!

  • utf8_encode() - required to convert multibyte char to string of bytes; for unit testing, it needs to be a separate function
  • killFrame() - would be called after a trackPageView(); used to prevent one's site from being frame by third-parties; there are tradeoffs (as you point) out, so it is not enabled by default

refs #2041 - setHeartBeatTimer - uses setTimeout, so it happens only once (vs setInterval is recurring); only happens on the landing page; intended to be used to improve bounce rate report; changed it to send ping=1 in the request

comment:2 Changed 3 years ago by vipsoft (robocoder)

I reworked setHeartBeatTimer() in [3830]. It now takes an initalPingDelay and recurringHeartBeatDelay as parameters. By default, this is disabled.

comment:3 Changed 3 years ago by vipsoft (robocoder)

The initial ping request contains "ping=1". Use this to improve bounce rates and visit time.

Subsequent heartbeat requests (if recurringDelay != 0) contain "hb=1".

I'm going to defer adding event handlers as there are issues with propagating some events like keyclicks.

Mental notes to self:

  • if you only send a beat when there's activity within the last n seconds, then on average, you'll expect the total time to be overstated by n/2 secs
  • to offset/compensate, you can instead send a beat if there's been activity in the last n/2 seconds
  • if there's no activity for n secs, do you stop checking for activity or stop sending beats?
  • what if activity resumes again? how long is too long ago to to resume beats?

comment:4 Changed 3 years ago by matt (mattab)

how do you define 'activity'? Will the ping be sent if user is looking at another tab, another window, or even away from computer?

comment:5 Changed 3 years ago by vipsoft (robocoder)

activity: mousemove, click, mouseup, mousedown, mousewheel, scroll, DOMMouseScroll, resize, keypress, keydown, keyup, focus, and blur

  • need this many because of browser differences an the possibility of interference from other eveny handlers
  • handler simply sets the lastActivity timestamp

Yes, ping would be sent. That's why there's a check for activity. If there's no activity, I propose to no longer check.

I'll commit my changes after some more testing.

comment:6 Changed 3 years ago by vipsoft (robocoder)

(In [3838]) refs #2041 - setHeartBeatTimer() now takes the minimumVisitLength and the heartBeatPeriod (time between pings) as parameters; this is simpler than initial ping + recurring hb; requests contain "ping=1".

also updated to jslint 2011-02-03

comment:7 Changed 3 years ago by vipsoft (robocoder)

(In [3839]) refs #2041 - rollback buggy n/2 logic

comment:8 Changed 3 years ago by vipsoft (robocoder)

(In [3840]) refs #2041 - undo previous optimizaton to ping on first page of a new session

comment:9 Changed 3 years ago by vipsoft (robocoder)

(In [3841]) refs #2041 - add guard

comment:10 Changed 3 years ago by vipsoft (robocoder)

  • Milestone changed from Feature requests to 1.2 Piwik 1.2

Client-side changes are done, so it would be nice to have the server-side change done in time for 1.2 but we can roll into the next milestone.

comment:11 Changed 3 years ago by matt (mattab)

How often are the activity tests done? every second? Just thinking of keeping JS performance overhead low with this new feature.

I wont be able to do this in time for 1.2, so it will go in 1.3. Is pinging disabled by default? thx

comment:12 Changed 3 years ago by vipsoft (robocoder)

Yes, disabled by default; frequency is user-defined.

comment:13 Changed 3 years ago by matt (mattab)

Before enabling this by default I think we could try it on some heavy websites, that are using GA, other libraries (jquery eg.) and doing Javascript heavily. Then we could see if Piwik adds any unexpected performance overhead.

For example it would be nice to say: Piwik will query a few JS variables every second to assess if the visitor is active or not, but will not conflict with any other library or Javascript and should not affect performance at all.

What do you think? for example, have you tried it on a a running piwik loading the dashboard full of widgets.

comment:14 Changed 3 years ago by matt (mattab)

  • Milestone changed from 1.2 Piwik 1.2 to 1.3 - Piwik 1.3
  • Priority changed from normal to major

comment:15 Changed 3 years ago by matt (mattab)

  • Priority changed from major to critical

I think, disabled by default, but easy to enable, this would make it a really cool feautre. Also, should be made available via the 'advanced' javascript tag generator see #1845

comment:16 Changed 3 years ago by matt (mattab)

  • Milestone changed from 1.5 - Piwik 1.5 to 1.x - Piwik 1.x
  • Priority changed from critical to major

comment:17 Changed 3 years ago by PolrBear

Just wondering where we are on this. I'm in the process of switching several sites from GA to Piwik, and this feature is pretty important to me. Does 1.x mean it's going to be in an indefinite release between now and 2?

comment:18 Changed 3 years ago by matt (mattab)

Polrbear, yes currently not roadmapped for 1.7 but it might change

comment:19 Changed 2 years ago by phoenix

I'm interessed too by this feature. I have activate it on my website, but the presentation in the server side, isn't perfect (the ping is show as a new page visit).

I hope, that it can be made for 1.7.

Thanks.

comment:20 Changed 2 years ago by matt (mattab)

Another bug report: http://forum.piwik.org/read.php?2,83913,page=1#msg-84060

NOTE: This feature does not work currently

comment:21 Changed 21 months ago by matt (mattab)

  • Milestone changed from 1.x - Piwik 1.x to 1.9 Piwik 1.9

comment:22 Changed 19 months ago by matt (mattab)

Proof Reading the upcoming book about Piwik, it's really good.

it gives me an idea related to this feature:

  • we should add a new parameter at the trackPageView function level, that users could use to trigger events/pageviews that will allow to disable the default "lower bounce rate". When "pinging" the page, we don't want to lower bounce rate by default, other users might want to do this as well in some cases.

We could allow for this other addon: send a fake Pageview 15seconds after arrival: ​http://analytics.blogspot.com/2012/07/tracking-adjusted-bounce-rate-in-google.html - might require the ping query so it doesn't inflate Page view counts, but changes Bounce rates for all other metrics.

Last edited 16 months ago by matt (previous) (diff)

comment:23 Changed 17 months ago by matt (mattab)

Note from Anthon:

The original thinking was, fire a ping once the page has been open some minimum amount of time, and keep pinging as long as there's some mouse activity. Basically, the goal would be to report lower bounce rates (ie if the first and only page visited) and more accurate time on site for the last page visited on a site.

The Page Visibility and traceCallback changes means the ping isn't sent until a tab gains focus and has been open the minimum amount of time. Further pings require some activity while the tab is open.

We probably need a way to restart the heartbeat if the tab regains visibility. Also, the inactivity logic doesn't work with passive activities (eg streaming media), but I think could be addressed by adding an API for players. (Are you using event tracking for this?)

This obviously isn't feature complete, so if anyone has ideas on how this should work or could work better, feel free to share your thoughts.

Last edited 17 months ago by matt (previous) (diff)

comment:24 Changed 17 months ago by matt (mattab)

  • Milestone changed from 1.10 Piwik 1.10 to Feature requests
  • Priority changed from major to critical

I'm very interested in this feature but unfortunately time is missing to implement it for now, unless someone jumps in and produces a patch we can use! Or if you can sponsor the feature please get in touch at sponsors at piwik.org

comment:25 Changed 14 months ago by capedfuzz (diosmosis)

  • Description modified (diff)

comment:26 Changed 12 months ago by matt (mattab)

  • Milestone changed from Feature requests to 2.0 - Piwik 2.0
  • Priority changed from critical to major

comment:27 Changed 8 months ago by matt (mattab)

  • Priority changed from major to normal

comment:28 Changed 7 months ago by matt (mattab)

  • Milestone changed from 2.0 - Piwik 2.0 to Piwik 2.1

Piwik 2.0 full focus mode: Moving normal + low priorities to Piwik 2.1 milestone.

comment:29 Changed 3 months ago by kevinoid

Great feature, looking forward to it. (Just commenting to add myself to the CC list.)

comment:30 Changed 3 months ago by matt (mattab)

  • Milestone changed from 2.1 - Piwik 2.1 to 2.x - Piwik 2.x
  • Priority changed from normal to major

comment:31 Changed 3 weeks ago by matt (mattab)

  • Priority changed from major to normal
Note: See TracTickets for help on using tickets.