Opened 4 years ago

Closed 2 years ago

Last modified 16 months ago

#1454 closed New feature (fixed)

Add sparklines and stat for Page views, Downloads and Outlinks separately

Reported by: matt Owned by:
Priority: major Milestone: 1.12.x - Piwik 1.12.x
Component: Core Keywords:
Cc: Sensitive: no

Description

Currently Piwik reports total number of actions, which includes page views, downloads and outlinks.

While this stat is useful, it would also make sense to report specifically about each action, and have a sparkline+stat for each of the following

  • Page views
  • Downloads
  • Clicks on outlinks

Change History (53)

comment:1 Changed 4 years ago by WaKKa

It will by useful!

comment:2 Changed 3 years ago by matt (mattab)

  • Milestone changed from Features requests to Piwik 1.x

Very useful indeed, comparing pages such as marsa-alam.htm and standard downloads / clicks is useful.

comment:3 Changed 3 years ago by matt (mattab)

Also in the API we should return nb_actions and nb_downloads nb_outlinks nb_hits

comment:4 Changed 3 years ago by matt (mattab)

  • Milestone changed from 1.x - Piwik 1.x to 1.7 Piwik 1.7

Pretty useful and must have!

comment:5 follow-up: Changed 3 years ago by EZdesign (BeezyT)

While working on this, I found a tracker bug which was visible in the ecommerceOrderWithItems testcase. I'll document it now and commit the fix/tests later.

The test case works as follows (only relevant parts):

  • Visitor views 4 pages in a visit
  • In a second visit, he views 6 pages (3 above the loop + 3 in the first loop with offset=0)
  • In a third visit, he views 3 pages (in the loop with offset=5)
  • In a fourth visit, he views 3 more pages (in the loop with offset=24)
  • After the loop, an order is placed with a time offset that leads to a fifth visit. Another order is placed in that visit.
  • Then, an order is placed for another idSite, which leads to a sixth visit

Number of actions should be 4 / 6 / 3 / 3 / 0 / 0, since actions are the sum of pageviews, outlinks and downloads. The computed actions are 4 / 6 / 3 / 4 / 1 / 1. This is because the fourth visit starts with doTrackEcommerceOrder and the fifth and sixth start with doTrackEcommerceOrder. All the others start with doTrackPageView. When a new visit is created, visit_total_actions is set to 1, which is only correct if the first thing that is tracked is an action.

The fix is to check whether we are tracking an action and set visit_total_actions to 0 or 1. I have this fix in my working copy. It will be commited with the other tests for this ticket.

comment:6 Changed 3 years ago by EZdesign (BeezyT)

This ticket requires some schema changes: visit_total_(hits|downloads|outlinks) in log_visit.

I added them to core/Db/Schema/Myisam.php and created an Update 1.6-rc2.php in core/Updates. This doesn't trigger the updater. What do I have to do?

comment:7 Changed 3 years ago by matt (mattab)

To trigger update, change core/Version.php to a new version number, then create the update file for this version number. It will trigger the update.

But in this case you dont need new fields in the schema, which creates performance overhead. To process page/downloads/outlinks, there is a simpler way and also that is more efficient. You can get the total number of downloads/oulinks/page views after processing the datatables in: http://dev.piwik.org/trac/browser/trunk/plugins/Actions/Actions.php#L463

This is similar to what is done in the Referers plugins to process the number of referers from each type (campaign, websites,etc.): http://dev.piwik.org/trac/browser/trunk/plugins/Referers/Referers.php#L477

comment:8 in reply to: ↑ 5 Changed 3 years ago by matt (mattab)

The fix is to check whether we are tracking an action and set visit_total_actions to 0 or 1. I have this fix in my working copy. It will be commited with the other tests for this ticket.

It sounds good!

comment:9 Changed 3 years ago by matt (mattab)

To get the downloads/outlinks/actions sum, for each table, you could do something like

$downloads = array_sum($dataTable->getColumn(Piwik_Archive::INDEX_PAGE_NB_HITS)) ?

comment:10 Changed 3 years ago by EZdesign (BeezyT)

Thanks for the feedback. I still think it would be good to add hits/downloads/outlinks to the schema.

  1. It doesn't really create performance overhead. Increasing the counts is done in the same query as increasing the number of actions.
  1. It's faster to aggregate. The aggregation for archiving takes place in the same query as the other VisitsSummary metrics. On the other hand, the sum of rows (one row being one url of which there can be a lot) will be calculated for 30 days to display a sparkline.
  1. Using the Actions archives would be a break in concept. All the other VisitsSummary metrics are calculated in one query and don't depend on the other plugins (since they are the core metrics). Loading the Actions report in the VisitsSummary API would look a little clumsy, wouldn't it?

These are the reasons why I would stick with this approach. If you're still conviced it would be better another way, I'll refactor. A compromise could be to add another report to the Actions plugin that returns the sums. This would partition related metrics in two reports (the ones we are talking about and the VisitsSummary) but it's still better than loading the Actions report in the VisitsSummary plugin. Nevertheless, I'd stick with counting the hits/downloads/outlinks in the database.

comment:11 Changed 3 years ago by matt (mattab)

The performance overhead is because we add 3 fields to the log_visit table, which cause overhead of 8b * N visits. Not much, but log_* tables must stay as small as possible.

Of course you are right that it is not ideal because VisitsSummary would then required Actions archives to display all sparklines. I think this limitation is preferred than adding fields to log_visit.

The Downloads/outlinks/pages metrics would be aggregated during archiving (as a "numeric" archive). So, for a sparkline of 30 days, the performance is the same as if you would process these metrics in VisitsSummary. Indeed it is important that all metrics are pre-processed. So the array_sum() would be during archiving only (when the datatable have just been processed). Then VisitsSummary controller would call Actions_API::getDownloadsCount for example. Does it sound good?

comment:12 Changed 2 years ago by EZdesign (BeezyT)

How do we add this to the UI? We could just add three more sparklines in the visitors overview or we could introduce some kind of groups in the overview. Any ideas?

Also, it would be cool to display pageviews and unique pageviews (same for downloads and outlinks) in one graph. I think we don't need a separate sparkline for the unique metrics, this would probably be too much.

comment:13 Changed 2 years ago by EZdesign (BeezyT)

(In [5386]) refs #1454 fix for tracking of nb_actions: only track pageviews, outlinks and downloads

comment:14 Changed 2 years ago by EZdesign (BeezyT)

(In [5387]) refs #1454 archiving (unique) pageviews, (unique) downloads and (unique) outlinks

comment:15 Changed 2 years ago by EZdesign (BeezyT)

The unit tests don't run on my machine so I can't add the new expected API output. Is this only happening locally or are the tests broken in trunk?

comment:16 Changed 2 years ago by EZdesign (BeezyT)

(In [5388]) refs #1454 adding pageviews, downloads and outlinks to the ui

comment:17 Changed 2 years ago by EZdesign (BeezyT)

[5388] is a first attempt at the UI. I would like to make it similar to row evolution but that's hard to do for 12 metrics... Maybe someone has an idea!?

comment:18 Changed 2 years ago by EZdesign (BeezyT)

(In [5389]) refs #1454 ui improvements

comment:19 Changed 2 years ago by EZdesign (BeezyT)

(In [5390]) refs #1454 ui improvements (this time from the right working copy)

comment:20 Changed 2 years ago by EZdesign (BeezyT)

(In [5391]) refs #1454 ui improvements (revertig 5389)

comment:21 Changed 2 years ago by EZdesign (BeezyT)

In [5389], I commited the updated files from the wrong working copy. The changes are reverted in [5391]. The actual UI improvements are in [5390]: I made the visits overview more compact by combining metrics.

comment:22 Changed 2 years ago by matt (mattab)

(In [5392]) refs #1454 Updating expected test files after adding few fields in the VisitsSummary output

comment:23 Changed 2 years ago by matt (mattab)

Feedback:

  • Great idea to display and plot both metrics :)
  • Backward compatibility UI: the "pageviews" metrics should display the old "Actions" count if the pageviews count is not available (ie. for dates before the upgrade to piwik 1.7). I realize this is NOT consistent and partly incorrect (but only when there were downloads/outlinks). It is much better than display zeros for all previous data. Also, if we do not do that, there would be no way to access the previous page views count, which is a very basic metric that we must make available :)
  • Test should work. On my box, it takes 800s to run tests/integration/Main.test.php - this is VERY painful. However, it should work. Hopefully at some point I will spend some time trying to make tests faster so it's easier for developers...

comment:24 Changed 2 years ago by EZdesign (BeezyT)

Regarding backward compatibility: could we manipulate the existing archives to add the new metrics during the update? We would just have to sum the values for all archived action tables and would not need to use the logs.

comment:25 Changed 2 years ago by matt (mattab)

Probably doable, but would potentially take a long time to select all archives (potentially thousands) and update it...

I think it's OK to leave the old data as it is but just make sure the UI uses Nb_Actions when available, if nb_pageviews is not available for a given day?

comment:26 Changed 2 years ago by EZdesign (BeezyT)

(In [5393]) refs #1454 new action metrics are only accessed by actions plugin, bounce rate fix, changed color of second line chart series, updated expected test files

comment:27 Changed 2 years ago by EZdesign (BeezyT)

(In [5394]) refs #1454 backward compatibility: show nb_actions if finer metrics are not available, german translation

comment:28 Changed 2 years ago by matt (mattab)

(In [5395]) refs #1454

  • rename API to Actions.get
  • EN translation fix
  • tests update, fix build?

comment:29 Changed 2 years ago by EZdesign (BeezyT)

(In [5396]) refs #1454 making german and english translations consistent

comment:30 Changed 2 years ago by matt (mattab)

(In [5399]) Fixing more test refs #1454

comment:31 Changed 2 years ago by matt (mattab)

(In [5400]) Fixing more test - refs #1454
VisitsSummary API does NOT call Actions API now.

comment:32 Changed 2 years ago by EZdesign (BeezyT)

(In [5404]) refs #1454 finally fixing the build?

comment:33 Changed 2 years ago by EZdesign (BeezyT)

(In [5405]) refs #1454 different csv encoding on the build server

comment:34 Changed 2 years ago by EZdesign (BeezyT)

(In [5406]) refs #1454 different csv encoding on the build server - again

comment:35 follow-up: Changed 2 years ago by EZdesign (BeezyT)

i added a file to integration/processed by mistake. should it be in svn-ignore?

comment:36 Changed 2 years ago by EZdesign (BeezyT)

(In [5407]) refs #1454 removing mistakenly added processed integration tests

comment:37 Changed 2 years ago by EZdesign (BeezyT)

(In [5408]) refs #1820 metrics picker, refs #1454 using Api.get for cross-plugin evolution graphs

comment:38 Changed 2 years ago by EZdesign (BeezyT)

(In [5409]) refs #1454 report meta data update

comment:39 Changed 2 years ago by EZdesign (BeezyT)

The visitors overview now works without the ugly cross-plugin API calls. It uses the brand new API.get which combines the *.get methods.

Also, there is a metrics picker in the visitors overview evolution graph that makes all metrics from VisitsSummary.get and Actions.get selectable.

comment:40 in reply to: ↑ 35 Changed 2 years ago by EZdesign (BeezyT)

Replying to EZdesign:

i added a file to integration/processed by mistake. should it be in svn-ignore?

I think it is in svn-ignore, isn't it? It just wasn't ignored by my git-svn setup.

comment:41 Changed 2 years ago by EZdesign (BeezyT)

(In [5410]) refs #1454 export and dashboard widget fix

comment:42 Changed 2 years ago by EZdesign (BeezyT)

(In [5414]) refs #1454 API.get has no plugin prefix for metrics anymore

comment:43 Changed 2 years ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from new to closed

(In [5416]) Fixes #1454

  • By default, show all columns merged
  • thanks timo for last patches now looks consistent

comment:44 Changed 2 years ago by EZdesign (BeezyT)

(In [5418]) refs #1454 now that API.get shows all metrics by default, we can export the full report when the export icon below the graph is clicked

comment:45 Changed 2 years ago by EZdesign (BeezyT)

(In [5419]) refs #2714 additional column translations and number format fix for csv exports, refs #1454 metadata for API.get report (needed for csv column translations)

comment:46 Changed 2 years ago by EZdesign (BeezyT)

(In [5420]) refs #2714, #1454 test fix for previous commit

comment:47 Changed 2 years ago by JulienM (JulienMoumne)

To fix a bug I'm having in the PDFReports plugin, I'd like to understand the purpose of the new API.get mentioned in comment:39.

Could you elaborate on its rationale?

I'm no expert in archiving and data aggregation so don't hesitate to put it in layman's terms.

comment:48 Changed 2 years ago by EZdesign (BeezyT)

@Julien: API.get combines the get methods of other plugins (e.g. Actions.get / VisitsSummary.get). You can get the data from the *.get reports in a single report. This makes comparing metrics from different .get methods possible, because the comparison takes place on one report only. The method introduces no additional data, it's just a proxy for the other methods. In PDFReports, you probably don't need to worry about it.

comment:49 Changed 2 years ago by JulienM (JulienMoumne)

Thanks for the clarification.

PDFReports plugin creation and edit forms use the output of 'API.getReportMetadata' to display the list of available reports.

The new API is returned in 'API.getReportMetadata'. It is therefore selectable when creating/editing mail reports.

Is this expected ?

comment:50 Changed 2 years ago by EZdesign (BeezyT)

Since the report doesn't contain new data it probably souldn't be selectable in the PDFReports plugin. If you want, take a look at the API output and see for yourself...

comment:51 Changed 2 years ago by matt (mattab)

julien, please add an exception to exclude this report from the PDF/Email list. thx!

comment:52 Changed 2 years ago by JulienM (JulienMoumne)

(In [5436]) refs #2706, #1454 API category reports excluded from PDFReports Plugin forms

comment:53 Changed 16 months ago by matt (mattab)

  • Milestone changed from 1.8.x Piwik 1.8.x to 1.9.x - Piwik 1.9.x

Milestone 1.8.x Piwik 1.8.x deleted

Note: See TracTickets for help on using tickets.