Show graph of visits/pages over the last 30 days, for a specific row (keyword, country, etc.)

[matt]

The plugin would add, in all data tables, an icon next to the keyword/ country / page view (or on hover), that would ajax load an evolution graph of the number of visits for this keyword, country or page, over the last 30 days/weeks/month. The graph would have the 'data export' links.

UI

We should find out how to provide this UI feature without bloating the UI with icons and new functionnality. Note that some rows in Piwik are already clickable (eg. to get the list of keyword by search engine).

One way of doing it would be to show a semi transparent icon when hovering over a row. The icon would be shown at the end of the row.

The new UI would also be used to report the new reports from #756

Report The report would contain, for a specific keyword or search engine or country or page, the following information:

• visits in date range + sparkline
• unique visitors in date range + sparkline
• bounce rate + sparkline
• (for pages) time per page, exit rate, etc. + sparkline

Also, we should report Goals conversions, maybe in a different report, available by a click on a tab, or some other mechanism. This report would show, for this keyword/country/page, the following info:

• overall conversions / conversion rate / revenue per visit
• for each goal...
  • conversions / conversion rate/ revenue per visit.

Performance note

Piwik stores data on a "per day" basis. When plotting number of page views for the url ' http://example.org/category/photo' for the last 30 days, it will require selecting data for the 30 days, and navigate through the tree to get the child node data (/photo) - this is all done automatically via filters. In this example, piwik would select 90 blobs of data, and unserialize, load in memory, grep, etc. This is a severe requirement and feasability should be studied before starting the implementation.

When plotting evolution of keyword "piwik" over the last 30 days, this would require 30 blobs of data to be selected/unserialize/loaded/etc.

Note

In the first version, only one data would be plottable. In future versions, we could think about plotting 3 or 4 keywords/countris on the same graph. How would this be available in the UI? Clicking on the icon could "enable" the line to be displayed on the graph, and a "reset" button, or a click on the icon, would disable this line. Requires #397.

[matt]

 Nice!

[matt]

This would make Piwik a much stronger Analytics tool.

[DaSch]

to stay on track from #508

[matt]

For reports that have a limited number of rows, for example: #2031, #583, #536, #584

It would be very nice to easily be able to splot the trends of each of these values over time, without having to click on each value separately to see the graph.

The graph plotting history could be shown for all these values at once, maybe with multiple lines on a line plot?

[EZdesign]

Page URL, page title, country and keyword are the labels of the DataTables. We could add a parameter "label" to the API. If set, the API plugin filters the table by label. This way, the other plugins don't event need to know about it.

This filtering funcionality has to be available for other plugins as well, eg for Actions.getPageUrl. Maybe it can be implemented in the DataTable class itself. DataTable_Array has to implement a similar function which passes the request to the other tables.

The filtering should provide only one record. If a match is found, we can stop searching. This is especially important for nested DataTables.

If performance is poor (which I suspect), we could add some sort of index to the DataTables. It doesn't have to be super fast so we can pick a lightweight one.

[matt]

The datatable related code must definitely be automatic for all reports, and work consistently across them without need for custom hacks for some reports.

The big challenge of this ticket is the UI of drawing a graph for 1 row, or several rows at once eg.

• compare performance of 2 key pages over the last 12 months
• compare performance of the current top 10 keywords over the last 12 months (10 lines on a line plot, confusing but can be useful!)

The UI should make it easy to plot one row, but also plot multiple. How could it look like? how would we "reset" to the previous table view? Where would the graph be displayed? What would be the possible interactions with the graph?

We could ask Marcin our designer to prepare pixel perfect mockup but before would need to come up with wireframe level ideas of such complicated-to-integrate-nicely UI element. Any idea please suggest :)

[EZdesign]

I've started implementing the filters for the API using the indexes that are already there on DataTables. The API gets a new parameter "label" and filters the result, if it's a DataTable or DataTable_Array. I also updated the recursive search code of the Actions plugin - works fine.

As for the UI, we could open a properties dialog (similar to right-click in windows) when a row is clicked. There, we can show the available actions like "open url", "show recent development" etc. This is more flexible than icons and clutters the UI less.

[matt]

Proposal for below the graph mini dashboard

= Keyword "Analytics free" =

$SPARKLINE Visits: between 134 - 670 visits per day, +14% over the period
$SPARKLINE Ecommerce orders: between 13 - 44 Ecommerce orders per day, -22% over the period

etc. for all metrics, below, ordered by metrics usefulness

• Like visible on the screenshot, text for other metrics faded out for other metrics to highlight current one
• The first number shows the range of values over the period.
• The second number +14% shows the difference between the first and last day of period. It is green or red based on performance (reusable css styles from All websites dashboard)
• it will process the % evolution using last period, which can be partial (only first half of current month)... so % will go down and be RED often... maybe we could base it on the last finished period?
• % between -5 and +5 could be left black instead of colored red/green?
  • Not a big fan of right click for UI, since right click is very hard to access and most people don't expect to use it in web app.
  • Maybe we could display a graph icon on hover on each row, for example  this one which is great. I suppose it is not that frequent that users hover on the rows for no reasons.

See attached proposal mockup, workflow could be:

• show icon + text "Plot" on hover on each row, should be big enough to be easily clickable, but should not take whole width since the row itself is sometimes clickable (eg. to show Custom Variables values for a given row of Custom Variable name)
• hover on the selector would make it light yellow or something,
• a click would open the new UI element with graphs etc.
• A click on the grey outside or pressing ESC closes the popover.

Thoughts? very excited by this feature :)

[matt]

Mockup to show click to launch popover

[matt]

In the popover, when showing

• stats for browsers, OS, countries, search engines, etc. the logo (found in the api result) could be displayed
  • in the title next to the $label
  • in the graph in the legend - but maybe icons are too wide for the graph legend?
• when showing search engines keywords, or page urls evolution, the $label could be linked to the actual search engine result page or page URL (as found in API response)

thoughts?

[EZdesign]

(In [5722]) refs #534, #2742 metadata api extensions: more/better translations, actionToLoadSubTables

[EZdesign]

(In [5723]) refs #534, #2742 integration tests for previous commit

[matt]

(In [5737]) Remove exit_rate from common metrics Refs #534

[matt]

(In [5739]) * Refs #534 Removing exit_rate from metadata API, it was added in the standard processed metrics causing other reports to display it with 0% values

• Refs #1465 Leave underscores in test names just to keep tests more readable (even though it is not a best practise to have _ in class prefixes!!)

[EZdesign]

(In [5758]) refs #534 recursive label filter for the api

[EZdesign]

(In [5759]) refs #534 integration tests for label filter

[EZdesign]

The previous commits introduce a label filter. It works for all API methods and filters out a single row - even from a subtable.

This filter will be used to get statistics for a single row in row evolution. In JS, the label (or the parts of the label in a nested table) will be retrived from the data table dom-nodes and an appropriate API request will be made.

Please review this part.

[EZdesign]

(In [5760]) refs #534 label filter integration tests for range (=> DataTable_Array)

[EZdesign]

(In [5761]) refs #534 removed originalLabel (left over from an earlier version of row evolution, not needed anymore)

[EZdesign]

(In [5762]) refs #534 svn properties set on new file DataTableLabelFilter.php

[EZdesign]

(In [5767]) refs #534: introducing row actions for data tables

[EZdesign]

The previous commit does not change the UI yet becuase I didn't commit the row actions' HTML in the template. I'll do that as soon as the other row evolution code is ready (because the committed code triggers row evolution).

For now, we can discuss the way row actions work. I didn't add them to datatable.js because this code is already pretty long and complex. Instead, datatable.js contains code that loads row action objects based on DOM elements in the table. This way, we can add more actions to the HTML and the existing code will load new actions, which we then have to add to datatable_rowactions.js.

Sounds good?

[matt]

@EZdesign Kuddos!! Review:

• Great that you put the datatable_rowactions.js in a separate file. As you said the datatable.js is already quite bloated, so that's very welcome..
• Do we really need to support the DataTable_Array use case? Because, if it is not going to be used by the UI, then there is no need to support the new DataTableLabelFilter (since no API consumer will need it?) If not needed by UI, we can remove the special Datatable_Array code and tests in [5760]
• in the following code

  if (!plabel) {
  	break;
  }
  labels.push(plabel);
  

  I think the if() could actually maybe be triggered, because of this frequent bug: #1024 So in the case that a "row evolution" report cannot be drawn (data not found in parent table, etc.), it would be nice to test/check that the code fails gracefully (since this edge case will happen).

I will make a few minor modifications as well in the next few hours

[EZdesign]

The DataTable_Array case is used every time row evolution is displayed. We want to get statistics for the previous days at once, so that will be an array.

[matt]

(In [5774]) Refs #534

• To ensure that the "label filtering" works on all types of labels including special characters "' <> etc. I refactored the SafeDecodeLabel filter function.
• Changed DataTableLabelFilter to use Piwik_API_ResponseBuilder directly rather than duplicating code
• Added few tests cases

[matt]

One more code review note:

• would you mind adding a test for Referers Search -> Keywords which would test the case that different APIs are called for parents VS children.

Otherwise looks good, please go ahead with the rest of commit -- I'm excited! :)