Opened 6 years ago

Closed 4 years ago

Last modified 22 months ago

#389 closed New feature (fixed)

Dashboard for multiple sites (potentially hundreds of websites)

Reported by: matt Owned by: matt
Priority: critical Milestone: Piwik 0.5.2
Component: Core Keywords: multi-sites
Cc: Sensitive: no

Description (last modified by matt)

Some Piwik users add a lot of websites in Piwik.
It is also common that users of a given Piwik installation can access more than 10 websites.
Currently they have to go to each of these websites to check their reports.

They need quick & easy way to get an overview of what's happening in all their websites at once, and a quick access to more details.

Mockup

http://dev.piwik.org/trac/attachment/ticket/389/multi%20site%20report.PNG

Overview

  • This page is space efficient but clear to display data about several websites
  • gives an overview of what happened on all your websites, so you can choose which ones need investigation.
  • display raw numbers
  • and show the evolution of the metrics over the last days/weeks/month. via sparkline + red/green indicator
    • this page links to each website report by a click on the website name
  • possible to change the period of the dashboard: Day/Week/Month/Year
  • each website is a simple row on the page
  • for each website, print the
    • website name
    • nb of unique visitors
    • nb of pages
    • (keep space for nb of goal conversions - coming)
    • a sparklines of the last 30 days/week/month/year for unique visitors
  • when hovering the website name, the "drag icon" appears on the left of the website name. This icon makes draggable the list of websites so that each user can reorder their list of websites. By default website are ordered alphabetically. The website order in the list are saved for each user in the database (like the dashboard is), except for anonymous for whom the order is only stored in session (alternatively the feature can be disabled for anonymous user). The drag n drop feature would use Jquery UI draggable that we already use to reorder widgets in the dashboard.
  • This dashboard can be accessed by a new entry "All websites" that always appears on the top of the "Website selector" on the top right of the page.

Speed
When loading 300 websites, we will lookup numerical values, for the period selected.

To show UU + visits + pages + conversions we need to load the 4 related numeric values for 300 websites. that's 1 SELECT query to return 1200 values.

to show green/red evolution indicator we need to select the compared numeric value from yesterday/last week/last month/last year. that's one SELECT to return 300 values.

loading 300 sparklines will take a loong time, as the browser will issue limited http request on the piwik domain. loading 300 sparklines will certainly take more than 20 seconds.
is that an issue?

Future releases

  • adding sorting feature so that user can sort table by unique visitors, conversions, etc.
  • adding search feature so that user can see only websites with names matching a string. the search box would have auto suggest feature.
  • add possibility to choose the default page to arrive on - either the dashboard with all websites or the first available website in the list

Estimate

between 4 and 6 days of work

Attachments (5)

multi site report.PNG (26.1 KB) - added by matt 5 years ago.
MultiSitesPlugin.zip (68.4 KB) - added by mauser 5 years ago.
MultiSitesPlugin v0.2.zip (20.2 KB) - added by mauser 5 years ago.
MultiSites.zip (12.2 KB) - added by pebosi 4 years ago.
MultiSites.2.zip (12.7 KB) - added by pebosi 4 years ago.

Download all attachments as: .zip

Change History (50)

comment:1 Changed 5 years ago by matt (mattab)

  • Description modified (diff)
  • Summary changed from Multiple site reports to Dashboard for multiple sites (potentially hundreds of websites)

comment:2 Changed 5 years ago by matt (mattab)

  • Description modified (diff)

Changed 5 years ago by matt (mattab)

comment:3 Changed 5 years ago by matt (mattab)

  • Description modified (diff)

comment:4 Changed 5 years ago by matt (mattab)

  • Description modified (diff)

comment:5 Changed 5 years ago by matt (mattab)

  • Description modified (diff)

comment:6 Changed 5 years ago by mauser (zawadzinski)

  • Owner set to mauser
  • Status changed from new to assigned

comment:7 Changed 5 years ago by mauser (zawadzinski)

Current version in the attachment.

Now working on core to optimize Archive/Array/*.php

comment:8 Changed 5 years ago by matt (mattab)

  • Priority changed from major to critical

comment:10 Changed 5 years ago by dazzlindonna

I've installed and activated this plugin but there is no new entry in the "Website selector" on the top right of the page. Only my sites are listed, but there still isn't an extra one for All websites. I installed and activated it on two different servers, and neither shows the new selection in the drop-down.

comment:11 Changed 5 years ago by dazzlindonna

Aha! I figured this one out. The documentation is incorrect. The plugin does work, just not as it states above. Rather than placing a new entry for All sites in the website selector in the top right of the page, it instead places an extra menu item in the main blue horizontal menu at the end after Dashboard, Visitors, Actions, Referrers. The new item is called Sites Summary.

comment:12 Changed 5 years ago by dazzlindonna

I originally reported this problem on the forums because trac was down for me, but now I've come here. Here's a summary of the bug below, but a larger version of the error is at the forum thread located at http://forum.piwik.org/index.php?showtopic=831. Neither this report nor the one at the forum includes the entire error because it is incredibly huge.

Since I upgraded to version 0.2.33, the multisites plugin has produced an error - the FIRST time I run it each day. After the first time, it's fine (until the next day). Here's an edited version of the error. The real version was ENORMOUS. I've also edited my domain info to keep it generic.

Notice: Undefined property: Piwik_DataTable_Simple::$rows in /home/myfolder/public_html/piwik/core/DataTable.php on line 457 Backtrace -->#0 Piwik_ErrorHandler(8, Undefined property: Piwik_DataTable_Simple::$rows, /home/myfolder/public_html/piwik/core/DataTable.php, 457, Array ()) called at home/myfolder/public_html/piwik/core/DataTable.php:457............

comment:13 Changed 5 years ago by matt (mattab)

  • Milestone changed from Stable release to DigitalVibes

comment:15 Changed 5 years ago by matt (mattab)

Please update to the new MultiSitesPlugin.tar for compatibility with 0.2.35.

comment:16 Changed 5 years ago by fuero

This pops up with piwik 0.2.35 (using the updated version above):

Fatal error: Unsupported operand types in /var/www/apps/content/piwik/plugins/MultiSitesPlugin/Controller.php on line 204

comment:17 Changed 5 years ago by mauser (zawadzinski)

Current version of MultiSitesPlugin. It still needs a lot of improvements and refactoring.

I am concerning rewriting the plugin to use directly piwik_archive_numeric_* tables in order to speed up the plugin (it is still very slow when even having "just" hundreds of sites).

If you got < 500 sites you can use this version (but remember to setup auto archiving http://piwik.org/docs/setup-auto-archiving/ - piwik won't be capable of archiving so many sites via web request!)

comment:18 Changed 5 years ago by mauser (zawadzinski)

The Problem.
There is no way to use SQL query result limiting (LIMIT) and sorting (ORDER BY) using Piwik API. This is very important for MultiSitesPlugin, because when having thousands of sites we do not want to fetch all results, and sort & limit them in PHP.

Propsed solution.
We can build MultiSitesPlugin_API that will query directly SQL fetching data from piwik_archive_numeric_* tables. We do not want to check if data for specific site is archived, so we assume that all data is archived using cron auto-archiving (users will be noticed about it when installing MultiSitesPlugin).

This solution will significantly increase MultiSitesPlugin performance.

comment:19 Changed 5 years ago by matt (mattab)

  • Type changed from Bug to New feature

comment:20 Changed 5 years ago by matt (mattab)

good stuff!
here is my feedback:

about the UI

  • the sparklines html includes could be loaded directly, no need to show the rolling image (ie. we know how the call looks like); currently it does one http request to request each image HTML but jquery could write directly the right html.
  • the numbers and text could be centered / aligned middle in the cell, and cells could be less high
  • 1 - 12 z 12 is in polish?
  • the CSS could reuse the existing data table CSS for UI consistency

thanks! it is getting real close to be included in core ;)

PS: we can include in core before the optimization happen on the php side, so please feel free to do the UI updates and then work on the php side if you wish to push this in core.

comment:21 Changed 5 years ago by TulipVorlax

Is this placement normal ?

http://img229.imageshack.us/img229/4594/20062009083816.jpg

If anyone know what to change in the main CSS file...

comment:22 Changed 5 years ago by mauser (zawadzinski)

New version in the attachment.

We use now Piwik's CSS styles, and plugin Piwik coding standards are followed.

comment:23 Changed 5 years ago by TulipVorlax

I'm sorry to say, the "button" in the menu is still "misplaced" :

(Firefox 3.0.11)

http://img526.imageshack.us/img526/3176/09072009100750.jpg

comment:24 Changed 5 years ago by mauser (zawadzinski)

Its caused by translation (that is too long and does not fit in the button), it is not this plugin related issue.

comment:25 Changed 5 years ago by TulipVorlax

Switching to english i still get this :

http://img200.imageshack.us/img200/1931/10072009151203.jpg

The submenu is always present no matter what language.

comment:26 follow-up: Changed 5 years ago by matt (mattab)

the location in the menu is temporary and will be fixed before submitting the plugin to the core - most likely it will appear as "All Websites" next to Dashboard, top left of screen.

comment:27 Changed 5 years ago by vipsoft (robocoder)

  • For consistency, I'd suggest dropping the "Plugin" suffix from the plugin name.
  • Given the autoloader, the require_once's can be removed from Controller.php and MultiSitesPlugin.php.

comment:28 Changed 5 years ago by kolchak

A few issues I have on this plugin. Firstly, it forces the php process to use 100% of the CPU for a long period of time (around 60 seconds - which would match the max process time in php.ini). It also throws a javascript error:

Webpage error details

User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 4.0.20506; Tablet PC 2.0;)
Timestamp: Wed, 15 Jul 2009 00:33:50 UTC

Message: Unknown runtime error
Line: 124348672
Char: 3
Code: 0
URI: http://example.com/piwik/index.php?module=CoreHome&action=index&idSite=2&period=day&date=2009-07-15

Message: Unknown runtime error
Line: 124348672
Char: 3
Code: 0
URI: http://example.com/piwik/index.php?module=CoreHome&action=index&idSite=2&period=day&date=2009-07-15

comment:29 Changed 5 years ago by frankvictor

this plugin can't work properly on my IE browser(can't list any website),on chrome and firefox it works well,is it a bug or I've done something wrong?

comment:30 in reply to: ↑ 26 Changed 5 years ago by albass

comment:31 Changed 5 years ago by matt (mattab)

  • Milestone changed from 2- DigitalVibes to 2 - Piwik 0.5
  • Sensitive unset

comment:32 Changed 5 years ago by spomoni

comment:33 Changed 5 years ago by vipsoft (robocoder)

  • Status changed from assigned to new

comment:34 Changed 4 years ago by matt (mattab)

  • Milestone changed from 1 - Piwik 0.5 to 1 - Piwik 0.5.1

Once the few bugs listed here are fixed, we would like to commit it to trunk asap after Piwik 0.6. Maciej, would you mind taking a look at the reported issues? thanks!

Changed 4 years ago by pebosi

comment:35 Changed 4 years ago by pebosi

removed the suffix and the require_once calls. added a link to topbar (see included patch).

changed some stuff in index.tpl, but i was just testing, so there's a problem when changing date or period.

regards pebosi

comment:36 Changed 4 years ago by matt (mattab)

Pebosi, can you please clarify what the issues are? What is missing for the plugin to be included in 0.5.1 in a stable state?
Thank you!

comment:37 Changed 4 years ago by pebosi

When changing the date the parameter "date" in url is not replace but it's appended to the url without an ampersand. so there's an error...
i tried using no ajax so i think there's a problem with the broadcast.

regards

Changed 4 years ago by pebosi

comment:38 Changed 4 years ago by vipsoft (robocoder)

In [1678], refs #389, commit MultiSites plugin

comment:39 Changed 4 years ago by pebosi

Hi,

bugs in the last commit [1678]:

  • Site selector is not hidden
  • date change is not affected

regards pebosi

comment:40 Changed 4 years ago by matt (mattab)

  • Owner changed from mauser to matt

comment:40 Changed 4 years ago by matt (mattab)

(In [1692]) refs #389 Fixing date selection, fixing column sorting, few styles modifications

comment:41 Changed 4 years ago by matt (mattab)

(In [1696]) MultiSites plugin enabled by default refs #389

comment:42 Changed 4 years ago by matt (mattab)

(In [1701]) refs #389 using the jquery magic so that IE works OK

comment:43 Changed 4 years ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from new to closed

comment:44 Changed 4 years ago by matt (mattab)

Included and enabled by default in 0.5.3

Next steps are performance improvements, see #1077

comment:45 Changed 22 months ago by JulienM (JulienMoumne)

(In [6478]) fixes #2708

refs #71

  • PDFReports major refactoring. Any plugin can now add new kinds of reports. Required for #2708 and #3118.
  • test report functionality ($idReport == 0) dropped in Piwik_PDFReports_API->generateReport()
  • All Websites report shows 3 more metrics: Goal Conversions, eCommerce Conversions and eCommerce Revenue. Can be removed if asked to.
  • Piwik_PDFReports_API->sendEmailReport renamed to sendReport
  • All Piwik_PDFReports_API method signatures updated to support generic report parameters

refs #389

  • new API method to retrieve only one Piwik site : Piwik_MultiSites_API->getOne()
  • per #2708 description, Piwik_MultiSites_API methods now support a new parameter named enhanced. When activated, Goal Conversions, eCommerce Conversions and eCommerce Revenue along with their evolution will be included in the API output.
  • API metrics refactored in (@ignored)Piwik_MultiSites_API->getApiMetrics()
  • Metadata now returns 12 metrics : nb_visits, visits_evolution, nb_actions, actions_evolution, revenu, revenue_evolution, nb_conversions, nb_conversions_evolution, orders, orders_evolution, ecommerce_revenue, ecommerce_revenue_evolution

refs #3118

  • ReportPublisher plugin could now easily be implemented

commits merged

TODO

  • the MobileMessaging settings page may need some embellishment
  • @review annotations need some attention
  • test if the MultiSites API evolutions have some impact on Piwik Mobile and other client code
Note: See TracTickets for help on using tickets.