Opened 4 years ago

Closed 13 months ago

Last modified 7 months ago

#1700 closed New feature (fixed)

Propose Page speed reports, Load time analytics

Reported by: vipsoft Owned by: EZdesign
Priority: critical Milestone: 1.12 - The Great 1.x Backlog
Component: Core Keywords:
Cc: Sensitive: no

Description (last modified by matt)

In Piwik 1.12 we are shipping Page Speed Report in Piwik! A new column "Average Response time" is displayed in the Pages Urls and Page title reports. This column is the time it takes for server to generate the response and for the visitor to download it.

The historical view of the time can be seen via Row evolution feature.

The average time across all pages is plotted on Visitor > Overview.

Time tracking works for all modern browsers, who most implement the Dom timing API.

http://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html

Enjoy this new simple page speed report in Piwik.

Attachments (3)

1700.patch (2.9 KB) - added by vipsoft 3 years ago.
changes to piwik.js
SitePerformance.zip (5.3 KB) - added by banezaklan 19 months ago.
Site Performance plugin (work in progress)
output.png (96.9 KB) - added by matt 13 months ago.
showing requests with wrong performance time

Download all attachments as: .zip

Change History (75)

comment:1 Changed 4 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:2 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)
  • Summary changed from Web Timing spec; performance metrics to Navigation Timing spec; performance metrics

comment:4 Changed 3 years ago by matt (mattab)

This was released in GA: http://www.google.com/support/analyticshelp/bin/answer.py?hl=en&answer=1205784&topic=1120718

We could do something similar, by doing:

  • Custom variables per page (in development), use a custom var per page to track the page load time for this page. Report the page load time for all URLs and Page Titles (as a new metric)
  • Custom Variables per visit. Track, as a Custom Var, the running average page load time of all pages in the visit.

Regarding "how to track the page load speed", we could either fire a different beacon (like GA does), or maybe approximate the page load speed by stopping the timer when the asynchronous beacon is fired? But, it might be not accurate enough to do this approximation.

comment:5 Changed 3 years ago by vipsoft (robocoder)

Reference http://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/timing-overview.png

We wouldn't capture every perf metric. I think we can condense it two main ones (similar to "drive/road" and "curb" time metrics):

  • time for the network request and response (ie responseEnd - navigationStart)
  • time for the page content to load resources and render (ie loadEventStart - domLoading)

The second metric can be approximated using the naive method shown in the spec example 1.

comment:6 Changed 3 years ago by matt (mattab)

  • Milestone changed from Feature requests to 1.x - Piwik 1.x

It would be great to tackle this before Piwik 2.0... increasing priority so we can figure out the implementation plan

comment:7 Changed 3 years ago by vipsoft (robocoder)

I'll post my idea later tonight.

Changed 3 years ago by vipsoft (robocoder)

changes to piwik.js

comment:8 Changed 3 years ago by vipsoft (robocoder)

I've attached a patch for proposed (minimum) changes to piwik.js.

  • I used a separate tracking request to record the timing because the trackPageView may be called before the dom has completed, and it may too early to pass the timings as custom variables.
  • This could be extended to sample a percentage of page views, eg
    // theoretically, sample 10% of page views
    if (Math.random() % 10 != 0) { return; }
    

Usage:

<head>
<script type="text/javascript">
var domLoading = new Date().getTime();
...
piwikTracker.enableTiming(domLoading);

or asynchronous method:

<head>
<script type="text/javascript">
var _paq = [];
_paq.push['enableTiming', new Date().getTime()];
</script>

comment:9 follow-up: Changed 3 years ago by matt (mattab)

it looks very interesting!

For user friendliness and ease of use and maintenance, I'm not sure about asking users to add the "var domLoading = ..." at the top of the page.

If we don't pass this value, timing will only work for recent browsers. I think that's good enough, rather than introduce an API input that we may not want to maintain in the future. What do you think?

For the php side, I'm working on Custom Var per page, so we could create a "fake pageview" with 2 custom vars that could maybe "update" the previous page view row rather than create a new one... That would make reporting easier I think

comment:10 in reply to: ↑ 9 ; follow-up: Changed 3 years ago by vipsoft (robocoder)

Replying to matt:

For user friendliness and ease of use and maintenance, I'm not sure about asking users to add the "var domLoading = ..." at the top of the page.

If we don't pass this value, timing will only work for recent browsers. I think that's good enough, rather than introduce an API input that we may not want to maintain in the future. What do you think?

Sure, timing for newer browsers only would be easier.

For the php side, I'm working on Custom Var per page, so we could create a "fake pageview" with 2 custom vars that could maybe "update" the previous page view row rather than create a new one... That would make reporting easier I think

Are you talking about page-level custom vars?

comment:11 in reply to: ↑ 10 Changed 3 years ago by matt (mattab)

Are you talking about page-level custom vars?

Yes, #2432

comment:12 Changed 3 years ago by matt (mattab)

  • Summary changed from Navigation Timing spec; performance metrics to Propose Page speed reports, Load time analytics

OK for limiting this to new browsers implementing timing APIs. These are the browsers we want to spend most of our time on since these are the ones we can easily time and profile.

Aim to provide new interesting reports:

  • Average page load time overall, and overall Bar chart of loading times
  • For each Page URL, (and/or page title?), Average load time breakdown (Bar graph), min/max
  • Show response time for each Country also? What other dimensions are relevant?

Implementation proposal:

  • Piwik.js
    • select a sample of visits, eg. 10%?
    • Send the timing request, in a different Piwik request, with a custom variable of type page, with a custom name eg. _pkspd for load time value
    • the PHP client also support this feature via trackTiming (similar to GA)
      For example if the jQuery library took 20 milliseconds to load from the Google Content Network, and you wanted data to be sent for 50% of visitors, you would call:
      _gaq.push([‘_trackTiming’, ‘jQuery’, ‘Load Library’, 20, ‘Google CDN’, 50]);
      
  • PHP Tracker code modifications
    • Modify Action logging mechanism to UPDATE the previous page view row, rather than insert a new page view for this "fake action". We do not want to change bounce rates. The request for an action with this custom variable set is the indicator
  • PHP Archiving Code & API Output
    • Modify the Actions report, to also count the total time loading page for all visits on the page, using the custom variable value for these rows that had it. Add the metrics (or count + sum)
  • UI: Add a new report in UI Actions, to show the Page Speed report page
    • Hide load time column from other the Actions reports

I think this would be a fantastic new feature to have.

comment:13 Changed 3 years ago by matt (mattab)

  • Priority changed from normal to major

This feature is becoming increasingly important for Web Analytics and Piwik should have this very interesting report :)

comment:14 Changed 3 years ago by matt (mattab)

  • Milestone changed from 1.x - Piwik 1.x to 1.7 Piwik 1.7

comment:15 Changed 2 years ago by matt (mattab)

New metrics in the site speed reports in GA: it looks very interesting!
http://analytics.blogspot.com/2011/12/greater-insights-from-site-speed-report.html

Tracking Javascript
Can we track the following metrics as custom variables scope page?


    Avg. Redirection Time - the time spent in redirection before fetching this page. If there are no redirects, the value for this metric is expected to be 0.
    Avg. Domain Lookup Time - the average amount of time spent in DNS lookup for this page.
    Avg. Server Connection Time - the time needed for the user to connect to your server.
    Avg. Server Response Time - the time for your server to respond to a user request, including the network time from the user’s location to your server.
    Avg. Page Download Time - the time to download your page.

Note: we don't have to track them all, maybe we should only track the general aggregated value for all visits rather than per-page value since it'd be too much data overhead to store in the Actions reports.

comment:16 Changed 2 years ago by vipsoft (robocoder)

All part of the Navigation Timing spec (linked above in ticket description). Not in my patch, but could be added.

The GA implementation has site and per page averages.

comment:17 Changed 2 years ago by banezaklan

Hello, have there been any more developments on this? At the company I work for I have been asked to develop a Piwik plugin which will track and display page loading metrics. The plugin will most likely become open-source once it is done. So, in order not to duplicate effort, apart from the attached patch is there any more info or code on the ticket subject?

comment:18 Changed 2 years ago by vipsoft (robocoder)

Banezaklan: feel free to develop and contribute this feature

comment:19 Changed 20 months ago by matt (mattab)

  • Milestone changed from 1.9 Piwik 1.9 to 1.8.x - Piwik 1.8.x
  • Priority changed from major to critical

This feature is just too important and too good not to assign the highest priority ;)

comment:20 follow-up: Changed 20 months ago by nnnnathann

Anybody working on this? I am new to piwik but would be happy to contribute if someone could point me in the right direction. We need this functionality ourselves. Is the best place to start the example plugin?

comment:21 in reply to: ↑ 20 Changed 20 months ago by nnnnathann

My bad! Educated myself a little and figured out where it stands.

Replying to nnnnathann:

Anybody working on this? I am new to piwik but would be happy to contribute if someone could point me in the right direction. We need this functionality ourselves. Is the best place to start the example plugin?

comment:22 Changed 20 months ago by hneamtu

Hi,

I have applied the patch uploaded by vipsoft and it seems to work, at least the part where 'nettime' and 'domtime' values are sent to piwik server. My question is what now? How can I view this data that in the Dashboard UI ? Is a new plug-in needed for that?

Thanks,
Horatiu

comment:23 Changed 20 months ago by nnnnathann

Hi Horatiu,

I am about 1000% naive when it comes to Piwik, and the first thing I did was try to hack a page speed number into the Actions section using the implementation proposal by matt as a guide.

I didn't want to submit a patch to SVN as this solution is about as hacky as it gets, but it might give you some insight into where things happen, especially if you are new like me:

https://github.com/nnnnathann/piwik/compare/master...page_load

Hope that helps!

Would love to spend some time to get this working the right way, Piwik is pretty awesome

comment:24 Changed 20 months ago by hneamtu

Hi nnnnathann,

Thanks for your info, I have applied your patch and it works! However, as you said, it's kind of hard to interpret the data since the values of the Custom Variable don't
have the URL of the page analyzed. I will try to find some solution to this.

Thanks again,
Horatiu

comment:25 Changed 19 months ago by banezaklan

Hello all,
Some time ago we started to work on a plugin to enable piwik to record the page loading metrics. I am uploading what we did so far.
Note that, this is still work in progress and I'm attaching the plugin as it is now. It is not ready for production yet!
What remains to be done is proper archiving of the data from the custom table which plugin creates and displaying of analyzed data preferably as part of standard Piwik reports.
At the moment, there is only one simple analysis of the data and is displayed as a part of the plugin. It is pulling directly from the custom table so it will get slow soon on very busy websites.
We're kind of stuck with the archiving part as we don't understand completely what and how to archive in order get the best reports later.
All ideas and help is welcome!

Notes: Example JS script to be added to monitored website is in the archive.

Changed 19 months ago by banezaklan

Site Performance plugin (work in progress)

comment:26 Changed 16 months ago by nicomen

Any progress on this?

comment:27 Changed 16 months ago by matt (mattab)

No progress yet, we are waiting for a company to sponsor the development on this feature! please contact us for more info

comment:28 Changed 14 months ago by matt (mattab)

  • Milestone changed from 1.12.x - Piwik 1.12.x to 1.12 - Piwik 1.12

comment:29 Changed 14 months ago by halfdan

Anthon's patch to piwik.js is great. I think however that the API should rather be piwikTracker.enableTiming(0.1) to track 10% of page views. We should also use the HTML5 Web Performance API to get the data. Supporting newer browser only is ok for this feature. http://www.html5rocks.com/en/tutorials/webperformance/basics

comment:30 Changed 13 months ago by EZdesign (BeezyT)

  • Component changed from New Plugin to Core
  • Owner set to EZdesign

comment:31 Changed 13 months ago by EZdesign (BeezyT)

In 65458d099ae4a0ddb1923b7f6e110c5c0a9d3207:

refs #1700 basic performance analytics: handle server generation time for each page and page title

CORE

  • formatting sub-second times
  • getColumn() method on data table array (in order to behave the same as the regular data table class)
  • data tables can store in their meta data, which columns are empty (this is used in order to dynamically hide the new "generation time" column)
  • ViewDataTable and Api.getProcessedReport act according to the empty column meta data

SCHEMA

  • new column custom_float_1 in log_link_visit_action
  • new version to apply the change

TRACKER

  • Piwik_Tracker::setGenerationTime
  • tracking parameter "generation_time_me"
  • value is stored in new custom_float_1 column
  • the log importer can handle a group "generation_time_micro" which can be used in a custom log format. _micro is used because apache can log the time in microseconds but piwik processes milliseconds.
  • note: extension of JS tracker still missing

ACTIONS PLUGIN

  • for pages and page titles, add new columns sum_time_generation and nb_hits_with_time_generation to the blob archives
  • if they are set, compute avg_time_generation on the API level. if not, remove the columns and mark them as empty in the data table meta data.
  • show new column "avg. generation time" in the pages and page titles reports

plus TESTS for everything

comment:32 Changed 13 months ago by EZdesign (BeezyT)

In bdf12d27b0eada820acd5210c94b21ab26466f95:

refs #1700 fixing an error introduced in the previous commit (thanks peter for the report).
added a little debugging output to datatable_manager.

comment:33 Changed 13 months ago by EZdesign (BeezyT)

In f56c9d6548778aea63e872ec125f4867e62def1c:

refs #1700 performance tracking in piwik.js

  • by default, send performance.timing.responseEnd - performance.timing.requestStart as generation time with the tracking request
  • if the performance.timing API is not supported by the browser, don't send the metric
  • new method disablePerformanceTracking()
  • new method setGenerationTimeMs(generationTime) to set the generation after measuring it on the server side

comment:34 follow-up: Changed 13 months ago by EZdesign (BeezyT)

A minimal version of this feature that supports only the generation time is complete at this point.

Here's a proposal for part of the documentation:

  • By default, Piwik uses the performance timing API that is available in most modern browsers. If it is available, piwik.js will measure the time between the start of the request and the end of receiving the response and track it as generation time. This number is the best approximation for the server generation time that is available on the client side. It is only approximate because it includes the transfer time.
  • Since the API is not available in all browsers, the performance data is not set on all pageviews. The value shown in Piwik is the average of all pageviews where a value for the time is set.
  • If you want to analyze the real server generation time, you have to measure it on the server and use piwikTracker.setGenerationTimeMs(generationTimeInMilliSeconds) to tell Piwik the real value. The call to setGenerationTimeMs() has to take place before calling trackPageView().
  • You can also disable the entire feature by using piwikTracker.disablePerformanceTracking(). This method also has to be called before calling trackPageView().
  • If you want to import the generation time from the server logs, specify a custom log format with the group "generation_time_micro" for the generation time in microseconds.

For more details about browser support, see http://caniuse.com/nav-timing

comment:35 Changed 13 months ago by EZdesign (BeezyT)

In 611922c647f8dfefffb472bd077d5c77ab99509b:

refs #1700 better time formatting for milliseconds

comment:36 Changed 13 months ago by matt (mattab)

In b10464e230fe19fc4dc3b1746fb2448b25b43308:

  • Small tweaks to translation refs #1700
  • rename custom_float_1 DB field to custom_float (further time tracking can be tracked as events)
  • fix regression missing auth in Live
  • remove extra comma in javascript which breaks IE parsing

comment:38 Changed 13 months ago by matt (mattab)

Great work Timo on this one (code + tests look nice)!

It's a simple implementation, approximating the Page Speed to the Server generation+Response download, but it's a agreat approximate of the server speed as "experienced by visitors".

Feedback/Review:

  • Can you add general "Avg. generation time" in Visitors>Overview as sparkline + archived metric? Overall per-website site speed will be of huge value!!
  • Can you update the doc at: https://github.com/piwik/piwik/blob/master/misc/log-analytics/README.md Maybe add a new section and showcase an example of a custom log + Custom command line used to import the time.
  • import_logs script should support micro (your use case) but also milli (the std use case will be expected to work). Please also add a new test case for milli... ;)
  • Thx for for the doc for JS. I will update the JS doc shortly before release.

comment:39 Changed 13 months ago by matt (mattab)

In db0dab9235a5ac0dd1216708a58f17483e3b4213:

Refs #1700
Fix notice when there is no data
Delete sum_time_generation column from output + Fix tests. This data point can be obtained by AVG_TIME * HITS_WITH_TIME so not so useful in output

comment:40 Changed 13 months ago by matt (mattab)

  • Description modified (diff)

comment:41 Changed 13 months ago by matt (mattab)

Couple suggestions to make reports better:

  • in the Reports, I think it's better to show 2 decimals so make two changes:
    • show 3.456s as 3.46s
    • show 876ms as 0.87s

This will allow easy visual comparison and highlighting of "bad" columns. 2 decimals avoids clutter.

  • in the Datatable for Page Urls/Page Titles, when hovering on the avg. response time value, show a tooltip "Average based on 198 hits" showing the nb_hits_with_response_time. This helps explaning sometimes some very high values, based on 1 or 2 hits.
Last edited 13 months ago by matt (previous) (diff)

comment:42 Changed 13 months ago by EZdesign (BeezyT)

In fb5f11a5ca82b15eb419803e6e00ce6dfba515d0:

refs #1700 performance analytics

  • adding avg_time_generation to Actions.get + integration tests
  • adding sparkline for average generation time to Visitors > Overview
  • changing number formatting to 0.XXs instead of XXXms + test cases
  • tooltip for reports with avg. generation time: "average based on X hit(s)"
  • log import: support generation_time_milli (not only generation_time_micro)
  • example for importing generation time from logs in read me

comment:43 Changed 13 months ago by EZdesign (BeezyT)

All review comments are implemented now.

OK to close the ticket?

comment:45 Changed 13 months ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from new to closed

In 1cfa989fd1bb9324a8a329652fe7f069fe91396a:

Beautify the time in Metadata reports + tests
Small update to log analytics readme
Fixes #1700

Awesome work Timo!

comment:46 Changed 13 months ago by EZdesign (BeezyT)

In 30f06e28d6ff1a1d366f598f657fb8484a9ae062:

refs #1700: new metrics min_time_generation, max_time_generation

  • DataTable_Row::sumRow has a new parameter $aggregationOperations which allows treating columns as min or max (not only sum)
  • The aggregation operations can be set on any data table or passed to ArchiveProcessing_Period::archiveDataTable()
  • The Actions plugin uses the mechanism to aggregate the new metrics as min and max
  • The metrics are in the API output but not in processed reports
  • The min/max values are shown in the tooltip that appears when hovering the average generation times
  • Integration test updates

comment:47 Changed 13 months ago by matt (mattab)

In f61c0196e33d8ecc5c4e967ce1a8199a11477de2:

Very nice commit!

Refs #1700

  • Refactoring + adding missing parameters to other sumRow() calls
  • Fixing tests on PHP 5.4 / 5.5

comment:48 Changed 13 months ago by EZdesign (BeezyT)

Thanks, Matt. I think now we're done here...

comment:49 Changed 13 months ago by matt (mattab)

Looking good.

But on the demo running latest code, the tooltip on hover, are not displayed, not sure why... I re-generated the archive for today but didn't help.
The data is there in the XML: http://demo.piwik.org/index.php?module=API&method=Actions.getPageUrls&format=XML&idSite=7&period=day&date=2013-04-04&flat=0&filter_limit=100&token_auth=anonymous

comment:50 Changed 13 months ago by EZdesign (BeezyT)

Did you delete tmp/assets/*?

comment:51 Changed 13 months ago by matt (mattab)

I just did to be sure, but this is not necessary as it's done during the upgrade process. still not working. I checked and the tooltip PHP code is the latest one.

comment:52 Changed 13 months ago by EZdesign (BeezyT)

The JS code is there too. But the span.cell-tooltip is missing. It sould be rendered in datatable_cell.tpl. Can you check that the template starts with {assign var="tooltipIndex" value=$column|cat:"_tooltip"} and that the template cache has been flushed?

comment:53 Changed 13 months ago by matt (mattab)

flushing the templates worked... Strange I think it's the first time I had to manually delete compiled templates on upgrade on the demo... maybe we introduced a regression somewhere else.

It's working anyway, very nice!

comment:54 Changed 13 months ago by matt (mattab)

  • Resolution fixed deleted
  • Status changed from closed to reopened

Reopening as we found a bug, eg. on the demo:

https://demo.piwik.org/index.php?module=CoreHome&action=index&date=2013-04-04&period=day&idSite=1#module=Actions&action=indexPageTitles&date=2013-04-04&period=day&idSite=1

Some records show an average generation time of "12 years".

Some page views have a max generation time of 412 years and 90 days.

There could be a bug in the DOMTiming API for some browser, or a bug in the tracker or arcihving mechanism.

We should:

  • investigate why on demo we have such high values (is it a bug in our code or in some browsers?)
  • Add a maximum value of maybe 10 minutes for the generation_time to prevent wrong values from affecting the reports

comment:55 Changed 13 months ago by matt (mattab)

Also,

  • Could we show the avg generation time in the visitor log ?

comment:56 Changed 13 months ago by EZdesign (BeezyT)

Could you run this on the demo server to find out more about the records with high generation times?
What exactly are the high numbers and do they occur in certain browsers only?

SELECT
  action.name,
  link.server_time,
  link.custom_float,
  visit.config_os,
  visit.config_browser_name,
  visit.config_browser_version
FROM piwik_log_link_visit_action AS link
LEFT JOIN piwik_log_action AS action
  ON link.idaction_url = action.idaction
LEFT JOIN piwik_log_visit AS visit
  ON visit.idvisit = link.idvisit
WHERE link.idsite = 1 AND 
  link.server_time >= "2013-04-04" AND 
  link.server_time <= "2013-04-05"
ORDER BY custom_float DESC

The generation time is performance.timing.responseEnd - performance.timing.requestStart. Maybe this happens if responseEnd is 0. We should catch this case in JS anyway.

comment:57 Changed 13 months ago by EZdesign (BeezyT)

In 9c7209b7267d659dc1ecd6a5c4a9fba7ae6f0a8f:

refs #1700: generation time in visitor log. using tooltip on the action urls and titles.

comment:58 Changed 13 months ago by EZdesign (BeezyT)

I was just wondering, would the median maybe be better than the average?

comment:59 Changed 13 months ago by matt (mattab)

In d646c5f43d5fa54b4b86b9cf4cc389be30c92ba0:

Refs #1700 Ignore requests with high generation time

Changed 13 months ago by matt (mattab)

showing requests with wrong performance time

comment:60 Changed 13 months ago by matt (mattab)

I think average is OK. See screenshots. I added check to ignore requests longer than 1 hour, which should fix it going forward!

comment:61 Changed 13 months ago by nicomen

Hi, not sure if this is helpful. But when I made a hack to measure response time (discreet steps), I used the following javascript to work a bit better whenever profiling time is not available. As you can see there is one for loading the page itself, and one for the entire roundtrip...

<head>
  <script type="text/javascript">var responseStart = new Date().getTime();</script>
</head>

and

<script type="text/javascript">
/* wait till everything is loaded before start tracking so that we can measure page speed */
window.onload = function () {
  try {
    var piwikTracker = Piwik.getTracker(pkBaseURL + "piwik.php", 4);
    if (responseStart) {
      elapsed_time = Math.ceil(Math.max(0, new Date().getTime - responseStart));
      piwikTracker.setCustomVariable (1, 'PageLoadTime', _get_value(elapsed_time), "page" );
    }

    // measure roundtrip from GET request till page load end
    // this will only work in modern browsers, we need to either use a cookie at unload
    // like New Relic does, or use server time in some way...
    navigationStart = window.performance       ? performance.timing.navigationStart       :
                      window.msPerformance     ? msPerformance.timing.navigationStart     :
                      window.webkitPerformance ? webkitPerformance.timing.navigationStart :
                                                 0;
    if (navigationStart) {
      elapsed_time = new Date().getTime() - navigationStart;
      piwikTracker.setCustomVariable (2, 'RoundtripLoadTime', _get_value(elapsed_time), "page" );
    }

    piwikTracker.discardHashTag(true); // Ignore fragment part
    piwikTracker.trackPageView();
    piwikTracker.enableLinkTracking();
  } catch( err ) { window.console && console.log(err.message); }
};
function _get_value(elapsed_time) {
  return elapsed_time > 10000 ? " >  10 sec" :
         elapsed_time >  8000 ? " >   8 sec" :
         elapsed_time >  7000 ? " >   7 sec" :
         elapsed_time >  6000 ? " >   6 sec" :
         elapsed_time >  5000 ? " >   5 sec" :
         elapsed_time >  4000 ? " >   4 sec" :
         elapsed_time >  3000 ? " >   3 sec" :
         elapsed_time >  2000 ? " >   2 sec" :
         elapsed_time >  1000 ? " >   1 sec" :
         elapsed_time >   900 ? " > 0.9 sec" :
         elapsed_time >   800 ? " > 0.8 sec" :
         elapsed_time >   700 ? " > 0.7 sec" :
         elapsed_time >   600 ? " > 0.6 sec" :
         elapsed_time >   500 ? " > 0.5 sec" :
         elapsed_time >   400 ? " > 0.4 sec" :
         elapsed_time >   300 ? " > 0.3 sec" :
         elapsed_time >   200 ? " > 0.2 sec" :
         elapsed_time >   100 ? " > 0.1 sec" :
                                " ~   0 sec";
}

comment:62 Changed 13 months ago by EZdesign (BeezyT)

@matt Looks like all the high values come from one visitor... I have no idea why this would happen. Thanks for the patch, that should work.

@nicomen you can use piwikTracker.setGenerationTimeMs to use your current method of measurement. In Piwik Core, we were looking for a way that does not require modifications of the tracked site. If you want to do better, just use piwikTracker.setGenerationTimeMs. The time can also be measured on the server side to be even more accurate.

comment:63 Changed 13 months ago by nicomen

@EZdesign: ah cool thanks ;)

(Although I still think it might be useful to be able to have both traffic/bandwidth speed including the whole roundtrip, and the actual finishing of loading a page. And maybe support for other timings, so maybe it should have been a set of custom variables that are averaged rather than aggregated?)

Last edited 13 months ago by nicomen (previous) (diff)

comment:64 Changed 13 months ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from reopened to closed

I think this is now fixed.

The result is beautiful, the perfect V1 of an awesome feature. Well done Timo and thanks to the customer!

comment:65 Changed 13 months ago by EZdesign (BeezyT)

In 1d91fd2dd63cd7fcbe8a9d462370d909904eebe7:

refs #1700 time tracking

  • bug fix: min/max generation time didn't work in nested data tables because the column aggregation operations were not passed to sub tables
  • ui improvement: very small values were shown as 0s. use three decimal places if value is smaller than 10ms (e.g. display 2ms as 0.002s instead of 0s)

comment:66 Changed 12 months ago by matt (mattab)

Added to the User Tracking API doc the new param:

  • generation_time_ms -- Average generation time, in milliseconds. This value is used to process the "Avg. generation time" column, in the Page URL and Page Title reports, as well as a site wide running average of the speed of all pageviews. Note: when using the Javascript tracker this value is set to (Time for server to generate response + Time for client to download response).

comment:67 in reply to: ↑ 34 Changed 12 months ago by EZdesign (BeezyT)

We still need to document the JS tracker changes, see my comment about that.

Should we add that to the JS tracker doc or write a separate page for performance analytics?

comment:68 Changed 12 months ago by matt (mattab)

I added to the js tracker doc the following:

  • setGenerationTimeMs(generationTime) - By default Piwik uses the browser DOM Timing API to accurately determine the time it takes to generate and download the page. You may overwrite the value by specifying a milliseconds value here.

I think you're right, creating a new small user guide for Page Speed report. I created the page at: Site Speed - feel free to improve it (but no need to discuss the "Disable" feature since we dont want users to disable)

Btw I also found a bug:

  • the 'Avg Generation time' metric does not appear in the Metrics picker in the Visitors>Overview graph.

comment:69 Changed 12 months ago by EZdesign (BeezyT)

In 0a1a5c137eec747eee54b5984110fe6a22fe01f1:

refs #1700 adding avg. generation time to visitors > overview

comment:70 Changed 12 months ago by EZdesign (BeezyT)

Nice documentation, Matt. Well done!

comment:71 Changed 12 months ago by matt (mattab)

In 6803f67666055d04fb0f511e6f28b991a48c4515:

Refs #1700 important: renaming generation_time_ms to gt_ms for keeping parameters name short.
Updated doc at: http://piwik.org/docs/tracking-api/reference/

and http://piwik.org/docs/page-speed/

comment:72 Changed 7 months ago by matt (mattab)

See also ticket #4173 to Add Page Generation Time for Site Search requests

Note: See TracTickets for help on using tickets.