Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Propose Page speed reports, Load time analytics #1700

Closed
robocoder opened this issue Sep 14, 2010 · 69 comments
Closed

Propose Page speed reports, Load time analytics #1700

robocoder opened this issue Sep 14, 2010 · 69 comments
Assignees
Labels
Critical Indicates the severity of an issue is very critical and the issue has a very high priority. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone

Comments

@robocoder
Copy link
Contributor

In Piwik 1.12 we are shipping Page Speed Report in Piwik! A new column "Average Response time" is displayed in the Pages Urls and Page title reports. This column is the time it takes for server to generate the response and for the visitor to download it.

The historical view of the time can be seen via Row evolution feature.

The average time across all pages is plotted on Visitor > Overview.

Time tracking works for all modern browsers, who most implement the Dom timing API.

http://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html

Enjoy this new simple page speed report in Piwik.

@mattab
Copy link
Member

mattab commented Apr 28, 2011

See also #1985

@mattab
Copy link
Member

mattab commented May 4, 2011

This was released in GA: http://www.google.com/support/analyticshelp/bin/answer.py?hl=en&answer=1205784&topic=1120718

We could do something similar, by doing:

  • Custom variables per page (in development), use a custom var per page to track the page load time for this page. Report the page load time for all URLs and Page Titles (as a new metric)
  • Custom Variables per visit. Track, as a Custom Var, the running average page load time of all pages in the visit.

Regarding "how to track the page load speed", we could either fire a different beacon (like GA does), or maybe approximate the page load speed by stopping the timer when the asynchronous beacon is fired? But, it might be not accurate enough to do this approximation.

@robocoder
Copy link
Contributor Author

Reference http://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/timing-overview.png

We wouldn't capture every perf metric. I think we can condense it two main ones (similar to "drive/road" and "curb" time metrics):

  • time for the network request and response (ie responseEnd - navigationStart)
  • time for the page content to load resources and render (ie loadEventStart - domLoading)

The second metric can be approximated using the naive method shown in the spec example 1.

@mattab
Copy link
Member

mattab commented May 6, 2011

It would be great to tackle this before Piwik 2.0... increasing priority so we can figure out the implementation plan

@robocoder
Copy link
Contributor Author

I'll post my idea later tonight.

@robocoder
Copy link
Contributor Author

Attachment: changes to piwik.js
1700.patch

@robocoder
Copy link
Contributor Author

I've attached a patch for proposed (minimum) changes to piwik.js.

  • I used a separate tracking request to record the timing because the trackPageView may be called before the dom has completed, and it may too early to pass the timings as custom variables.
  • This could be extended to sample a percentage of page views, eg
// theoretically, sample 10% of page views
if (Math.random() % 10 != 0) { return; }

Usage:

<head>
<script type="text/javascript">
var domLoading = new Date().getTime();
...
piwikTracker.enableTiming(domLoading);

or asynchronous method:

<head>
<script type="text/javascript">
var _paq = [];
_paq.push['enableTiming', new Date().getTime()];
</script>

@mattab
Copy link
Member

mattab commented May 10, 2011

it looks very interesting!

For user friendliness and ease of use and maintenance, I'm not sure about asking users to add the "var domLoading = ..." at the top of the page.

If we don't pass this value, timing will only work for recent browsers. I think that's good enough, rather than introduce an API input that we may not want to maintain in the future. What do you think?

For the php side, I'm working on Custom Var per page, so we could create a "fake pageview" with 2 custom vars that could maybe "update" the previous page view row rather than create a new one... That would make reporting easier I think

@robocoder
Copy link
Contributor Author

Replying to matt:

For user friendliness and ease of use and maintenance, I'm not sure about asking users to add the "var domLoading = ..." at the top of the page.

If we don't pass this value, timing will only work for recent browsers. I think that's good enough, rather than introduce an API input that we may not want to maintain in the future. What do you think?

Sure, timing for newer browsers only would be easier.

For the php side, I'm working on Custom Var per page, so we could create a "fake pageview" with 2 custom vars that could maybe "update" the previous page view row rather than create a new one... That would make reporting easier I think

Are you talking about page-level custom vars?

@mattab
Copy link
Member

mattab commented May 16, 2011

Are you talking about page-level custom vars?

Yes, #2432

@mattab
Copy link
Member

mattab commented Jun 14, 2011

OK for limiting this to new browsers implementing timing APIs. These are the browsers we want to spend most of our time on since these are the ones we can easily time and profile.

Aim to provide new interesting reports:

  • Average page load time overall, and overall Bar chart of loading times
  • For each Page URL, (and/or page title?), Average load time breakdown (Bar graph), min/max
  • Show response time for each Country also? What other dimensions are relevant?

Implementation proposal:

  • Piwik.js
    • select a sample of visits, eg. 10%?
    • Send the timing request, in a different Piwik request, with a custom variable of type page, with a custom name eg. _pkspd for load time value
    • the PHP client also support this feature via trackTiming (similar to GA)
For example if the jQuery library took 20 milliseconds to load from the Google Content Network, and you wanted data to be sent for 50% of visitors, you would call:
_gaq.push([_trackTiming, jQuery, Load Library, 20, Google CDN, 50]);
  • PHP Tracker code modifications
    • Modify Action logging mechanism to UPDATE the previous page view row, rather than insert a new page view for this "fake action". We do not want to change bounce rates. The request for an action with this custom variable set is the indicator
  • PHP Archiving Code & API Output
    • Modify the Actions report, to also count the total time loading page for all visits on the page, using the custom variable value for these rows that had it. Add the metrics (or count + sum)
  • UI: Add a new report in UI Actions, to show the Page Speed report page
    • Hide load time column from other the Actions reports

I think this would be a fantastic new feature to have.

@mattab
Copy link
Member

mattab commented Sep 16, 2011

This feature is becoming increasingly important for Web Analytics and Piwik should have this very interesting report :)

@mattab
Copy link
Member

mattab commented Dec 15, 2011

New metrics in the site speed reports in GA: it looks very interesting!
http://analytics.blogspot.com/2011/12/greater-insights-from-site-speed-report.html

Tracking Javascript
Can we track the following metrics as custom variables scope page?



    Avg. Redirection Time - the time spent in redirection before fetching this page. If there are no redirects, the value for this metric is expected to be 0.
    Avg. Domain Lookup Time - the average amount of time spent in DNS lookup for this page.
    Avg. Server Connection Time - the time needed for the user to connect to your server.
    Avg. Server Response Time - the time for your server to respond to a user request, including the network time from the users location to your server.
    Avg. Page Download Time - the time to download your page.

Note: we don't have to track them all, maybe we should only track the general aggregated value for all visits rather than per-page value since it'd be too much data overhead to store in the Actions reports.

@robocoder
Copy link
Contributor Author

All part of the Navigation Timing spec (linked above in ticket description). Not in my patch, but could be added.

The GA implementation has site and per page averages.

@anonymous-matomo-user
Copy link

Hello, have there been any more developments on this? At the company I work for I have been asked to develop a Piwik plugin which will track and display page loading metrics. The plugin will most likely become open-source once it is done. So, in order not to duplicate effort, apart from the attached patch is there any more info or code on the ticket subject?

@robocoder
Copy link
Contributor Author

Banezaklan: feel free to develop and contribute this feature

@mattab
Copy link
Member

mattab commented Aug 17, 2012

This feature is just too important and too good not to assign the highest priority ;)

@anonymous-matomo-user
Copy link

Anybody working on this? I am new to piwik but would be happy to contribute if someone could point me in the right direction. We need this functionality ourselves. Is the best place to start the example plugin?

@anonymous-matomo-user
Copy link

My bad! Educated myself a little and figured out where it stands.

Replying to nnnnathann:

Anybody working on this? I am new to piwik but would be happy to contribute if someone could point me in the right direction. We need this functionality ourselves. Is the best place to start the example plugin?

@anonymous-matomo-user
Copy link

Hi,

I have applied the patch uploaded by vipsoft and it seems to work, at least the part where 'nettime' and 'domtime' values are sent to piwik server. My question is what now? How can I view this data that in the Dashboard UI ? Is a new plug-in needed for that?

Thanks,
Horatiu

@anonymous-matomo-user
Copy link

Hi Horatiu,

I am about 1000% naive when it comes to Piwik, and the first thing I did was try to hack a page speed number into the Actions section using the implementation proposal by matt as a guide.

I didn't want to submit a patch to SVN as this solution is about as hacky as it gets, but it might give you some insight into where things happen, especially if you are new like me:

https://github.com/nnnnathann/piwik/compare/master...page_load

Hope that helps!

Would love to spend some time to get this working the right way, Piwik is pretty awesome

@anonymous-matomo-user
Copy link

Hi nnnnathann,

Thanks for your info, I have applied your patch and it works! However, as you said, it's kind of hard to interpret the data since the values of the Custom Variable don't
have the URL of the page analyzed. I will try to find some solution to this.

Thanks again,
Horatiu

@anonymous-matomo-user
Copy link

Hello all,
Some time ago we started to work on a plugin to enable piwik to record the page loading metrics. I am uploading what we did so far.
Note that, this is still work in progress and I'm attaching the plugin as it is now. It is not ready for production yet!
What remains to be done is proper archiving of the data from the custom table which plugin creates and displaying of analyzed data preferably as part of standard Piwik reports.
At the moment, there is only one simple analysis of the data and is displayed as a part of the plugin. It is pulling directly from the custom table so it will get slow soon on very busy websites.
We're kind of stuck with the archiving part as we don't understand completely what and how to archive in order get the best reports later.
All ideas and help is welcome!

Notes: Example JS script to be added to monitored website is in the archive.

@anonymous-matomo-user
Copy link

Attachment: Site Performance plugin (work in progress)
SitePerformance.zip

@anonymous-matomo-user
Copy link

Any progress on this?

@mattab
Copy link
Member

mattab commented Jan 4, 2013

No progress yet, we are waiting for a company to sponsor the development on this feature! please contact us for more info

@halfdan
Copy link
Member

halfdan commented Mar 11, 2013

Anthon's patch to piwik.js is great. I think however that the API should rather be piwikTracker.enableTiming(0.1) to track 10% of page views. We should also use the HTML5 Web Performance API to get the data. Supporting newer browser only is ok for this feature. http://www.html5rocks.com/en/tutorials/webperformance/basics

@timo-bes
Copy link
Member

In 65458d0: refs #1700 basic performance analytics: handle server generation time for each page and page title

CORE

  • formatting sub-second times
  • getColumn() method on data table array (in order to behave the same as the regular data table class)
  • data tables can store in their meta data, which columns are empty (this is used in order to dynamically hide the new "generation time" column)
  • ViewDataTable and Api.getProcessedReport act according to the empty column meta data

SCHEMA

  • new column custom_float_1 in log_link_visit_action
  • new version to apply the change

TRACKER

  • Piwik_Tracker::setGenerationTime
  • tracking parameter "generation_time_me"
  • value is stored in new custom_float_1 column
  • the log importer can handle a group "generation_time_micro" which can be used in a custom log format. _micro is used because apache can log the time in microseconds but piwik processes milliseconds.
  • note: extension of JS tracker still missing

ACTIONS PLUGIN

  • for pages and page titles, add new columns sum_time_generation and nb_hits_with_time_generation to the blob archives
  • if they are set, compute avg_time_generation on the API level. if not, remove the columns and mark them as empty in the data table meta data.
  • show new column "avg. generation time" in the pages and page titles reports

plus TESTS for everything

@timo-bes
Copy link
Member

In bdf12d2: refs #1700 fixing an error introduced in the previous commit (thanks peter for the report).
added a little debugging output to datatable_manager.

@timo-bes
Copy link
Member

In f56c9d6: refs #1700 performance tracking in piwik.js

  • by default, send performance.timing.responseEnd - performance.timing.requestStart as generation time with the tracking request
  • if the performance.timing API is not supported by the browser, don't send the metric
  • new method disablePerformanceTracking()
  • new method setGenerationTimeMs(generationTime) to set the generation after measuring it on the server side

@mattab
Copy link
Member

mattab commented Apr 4, 2013

flushing the templates worked... Strange I think it's the first time I had to manually delete compiled templates on upgrade on the demo... maybe we introduced a regression somewhere else.

It's working anyway, very nice!

@mattab
Copy link
Member

mattab commented Apr 4, 2013

Reopening as we found a bug, eg. on the demo:

https://demo.piwik.org/index.php?module=CoreHome&action=index&date=2013-04-04&period=day&idSite=1#module=Actions&action=indexPageTitles&date=2013-04-04&period=day&idSite=1

Some records show an average generation time of "12 years".

Some page views have a max generation time of 412 years and 90 days.

There could be a bug in the DOMTiming API for some browser, or a bug in the tracker or arcihving mechanism.

We should:

  • investigate why on demo we have such high values (is it a bug in our code or in some browsers?)
  • Add a maximum value of maybe 10 minutes for the generation_time to prevent wrong values from affecting the reports

@mattab
Copy link
Member

mattab commented Apr 5, 2013

Also,

  • Could we show the avg generation time in the visitor log ?

@timo-bes
Copy link
Member

timo-bes commented Apr 5, 2013

Could you run this on the demo server to find out more about the records with high generation times?
What exactly are the high numbers and do they occur in certain browsers only?

SELECT
  action.name,
  link.server_time,
  link.custom_float,
  visit.config_os,
  visit.config_browser_name,
  visit.config_browser_version
FROM piwik_log_link_visit_action AS link
LEFT JOIN piwik_log_action AS action
  ON link.idaction_url = action.idaction
LEFT JOIN piwik_log_visit AS visit
  ON visit.idvisit = link.idvisit
WHERE link.idsite = 1 AND 
  link.server_time >= "2013-04-04" AND 
  link.server_time <= "2013-04-05"
ORDER BY custom_float DESC

The generation time is performance.timing.responseEnd - performance.timing.requestStart. Maybe this happens if responseEnd is 0. We should catch this case in JS anyway.

@timo-bes
Copy link
Member

timo-bes commented Apr 5, 2013

In 9c7209b: refs #1700: generation time in visitor log. using tooltip on the action urls and titles.

@timo-bes
Copy link
Member

timo-bes commented Apr 5, 2013

I was just wondering, would the median maybe be better than the average?

@mattab
Copy link
Member

mattab commented Apr 5, 2013

In d646c5f: Refs #1700 Ignore requests with high generation time

@mattab
Copy link
Member

mattab commented Apr 5, 2013

Attachment: showing requests with wrong performance time
output.png

@mattab
Copy link
Member

mattab commented Apr 5, 2013

I think average is OK. See screenshots. I added check to ignore requests longer than 1 hour, which should fix it going forward!

@anonymous-matomo-user
Copy link

Hi, not sure if this is helpful. But when I made a hack to measure response time (discreet steps), I used the following javascript to work a bit better whenever profiling time is not available. As you can see there is one for loading the page itself, and one for the entire roundtrip...

<head>
  <script type="text/javascript">var responseStart = new Date().getTime();</script>
</head>

and

<script type="text/javascript">
/* wait till everything is loaded before start tracking so that we can measure page speed */
window.onload = function () {
  try {
    var piwikTracker = Piwik.getTracker(pkBaseURL + "piwik.php", 4);
    if (responseStart) {
      elapsed_time = Math.ceil(Math.max(0, new Date().getTime - responseStart));
      piwikTracker.setCustomVariable (1, 'PageLoadTime', _get_value(elapsed_time), "page" );
    }

    // measure roundtrip from GET request till page load end
    // this will only work in modern browsers, we need to either use a cookie at unload
    // like New Relic does, or use server time in some way...
    navigationStart = window.performance       ? performance.timing.navigationStart       :
                      window.msPerformance     ? msPerformance.timing.navigationStart     :
                      window.webkitPerformance ? webkitPerformance.timing.navigationStart :
                                                 0;
    if (navigationStart) {
      elapsed_time = new Date().getTime() - navigationStart;
      piwikTracker.setCustomVariable (2, 'RoundtripLoadTime', _get_value(elapsed_time), "page" );
    }

    piwikTracker.discardHashTag(true); // Ignore fragment part
    piwikTracker.trackPageView();
    piwikTracker.enableLinkTracking();
  } catch( err ) { window.console && console.log(err.message); }
};
function _get_value(elapsed_time) {
  return elapsed_time > 10000 ? " >  10 sec" :
         elapsed_time >  8000 ? " >   8 sec" :
         elapsed_time >  7000 ? " >   7 sec" :
         elapsed_time >  6000 ? " >   6 sec" :
         elapsed_time >  5000 ? " >   5 sec" :
         elapsed_time >  4000 ? " >   4 sec" :
         elapsed_time >  3000 ? " >   3 sec" :
         elapsed_time >  2000 ? " >   2 sec" :
         elapsed_time >  1000 ? " >   1 sec" :
         elapsed_time >   900 ? " > 0.9 sec" :
         elapsed_time >   800 ? " > 0.8 sec" :
         elapsed_time >   700 ? " > 0.7 sec" :
         elapsed_time >   600 ? " > 0.6 sec" :
         elapsed_time >   500 ? " > 0.5 sec" :
         elapsed_time >   400 ? " > 0.4 sec" :
         elapsed_time >   300 ? " > 0.3 sec" :
         elapsed_time >   200 ? " > 0.2 sec" :
         elapsed_time >   100 ? " > 0.1 sec" :
                                " ~   0 sec";
}

@timo-bes
Copy link
Member

timo-bes commented Apr 5, 2013

@matt Looks like all the high values come from one visitor... I have no idea why this would happen. Thanks for the patch, that should work.

@nicomen you can use piwikTracker.setGenerationTimeMs to use your current method of measurement. In Piwik Core, we were looking for a way that does not require modifications of the tracked site. If you want to do better, just use piwikTracker.setGenerationTimeMs. The time can also be measured on the server side to be even more accurate.

@anonymous-matomo-user
Copy link

@ezdesign: ah cool thanks ;)

(Although I still think it might be useful to be able to have both traffic/bandwidth speed including the whole roundtrip, and the actual finishing of loading a page. And maybe support for other timings, so maybe it should have been a set of custom variables that are averaged rather than aggregated?)

@mattab
Copy link
Member

mattab commented Apr 10, 2013

I think this is now fixed.

The result is beautiful, the perfect V1 of an awesome feature. Well done Timo and thanks to the customer!

@timo-bes
Copy link
Member

In 1d91fd2: refs #1700 time tracking

  • bug fix: min/max generation time didn't work in nested data tables because the column aggregation operations were not passed to sub tables
  • ui improvement: very small values were shown as 0s. use three decimal places if value is smaller than 10ms (e.g. display 2ms as 0.002s instead of 0s)

@mattab
Copy link
Member

mattab commented Apr 13, 2013

Added to the User Tracking API doc the new param:

  • generation_time_ms -- Average generation time, in milliseconds. This value is used to process the "Avg. generation time" column, in the Page URL and Page Title reports, as well as a site wide running average of the speed of all pageviews. Note: when using the Javascript tracker this value is set to (Time for server to generate response + Time for client to download response).

@timo-bes
Copy link
Member

We still need to document the JS tracker changes, see my comment about that.

Should we add that to the JS tracker doc or write a separate page for performance analytics?

@mattab
Copy link
Member

mattab commented Apr 14, 2013

I added to the js tracker doc the following:

  • setGenerationTimeMs(generationTime) - By default Piwik uses the browser DOM Timing API to accurately determine the time it takes to generate and download the page. You may overwrite the value by specifying a milliseconds value here.

I think you're right, creating a new small user guide for Page Speed report. I created the page at: Site Speed - feel free to improve it (but no need to discuss the "Disable" feature since we dont want users to disable)

Btw I also found a bug:

  • the 'Avg Generation time' metric does not appear in the Metrics picker in the Visitors>Overview graph.

@timo-bes
Copy link
Member

In 0a1a5c1: refs #1700 adding avg. generation time to visitors > overview

@timo-bes
Copy link
Member

Nice documentation, Matt. Well done!

@mattab
Copy link
Member

mattab commented May 7, 2013

In 6803f67: Refs #1700 important: renaming generation_time_ms to gt_ms for keeping parameters name short.
Updated doc at: http://piwik.org/docs/tracking-api/reference/
and http://piwik.org/docs/page-speed/

@mattab
Copy link
Member

mattab commented Sep 23, 2013

See also ticket #4173 to Add Page Generation Time for Site Search requests

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Critical Indicates the severity of an issue is very critical and the issue has a very high priority. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Development

No branches or pull requests

5 participants