Opened 2 years ago

Closed 10 months ago

Last modified 10 months ago

#3124 closed Bug (fixed)

Make "All Websites Dashboard" usable with 20,000+ websites

Reported by: capedfuzz Owned by: capedfuzz
Priority: critical Milestone: 2.x - The Great Piwik 2.x Backlog
Component: Performance Keywords:
Cc: Sensitive: no

Description

Currently the All Websites dashboard will work well when there are hundreds or even thousands of websites, but after a point it becomes unusable. It should, however, be able to scale to any number of websites.

There are currently many barriers to scaling this area, including:

  • Lack of server-side pagination means all site data must be queried at once.
  • The design of the Array Archive types will result in unnecessary object creation (in this case several objects per website).
  • There's no way to archive metrics for all websites, so summary & evolution data has to be calculated when queried.

These issues will be addressed in this ticket.

Change History (12)

comment:1 Changed 2 years ago by matt (mattab)

  • Milestone set to 1.8 Piwik 1.8
  • Priority changed from normal to major

comment:2 Changed 2 years ago by matt (mattab)

  • Milestone changed from 1.9 Piwik 1.9 to 1.8.x - Piwik 1.8.x

comment:3 Changed 23 months ago by matt (mattab)

  • Priority changed from major to critical

comment:4 Changed 21 months ago by matt (mattab)

  • Owner set to capedfuzz

What is the status of this ticket, and how much work do you estimate is left to finish it?

comment:5 Changed 21 months ago by capedfuzz (diosmosis)

Still quite a bit of work left. I have to finish the refactoring of the Archive types, then commit the changes, piece by piece. One of the other two changes may not be as hard to implement. I'll send you a patch of what I've done so far so you can take a look (w/o having to review it).

comment:6 Changed 18 months ago by matt (mattab)

  • Component changed from Core to Performance

comment:7 Changed 14 months ago by matt (mattab)

  • Milestone changed from 1.12.x - Piwik 1.12.x to 2.x - Piwik 2.x

Please contact us if you are interested in this fix!

comment:8 follow-up: Changed 11 months ago by pebosi

When using 100-200 sites and changing period to month it's also unusable slow, even with sparklines off. Are there any options to optimize on server-side or in piwik it self?

comment:9 in reply to: ↑ 8 Changed 11 months ago by capedfuzz (diosmosis)

Replying to pebosi:

When using 100-200 sites and changing period to month it's also unusable slow, even with sparklines off. Are there any options to optimize on server-side or in piwik it self?

Hi @pebosi, the dashboard should be usable for ~1000 websites on any period except range, but ONLY if browser-initiated archiving is disabled (controlled by the first setting in the General Settings admin page). Is this disabled for you?

comment:10 Changed 10 months ago by capedfuzz (diosmosis)

  • Resolution set to fixed
  • Status changed from new to closed

Fixed by Archive.php refactor. When archiving is disabled and there are 20,000 sites each w/ visits, MultiSites.getAll takes 12-14s to return the result on my laptop. Memory usage is at 118 MB.

Last edited 10 months ago by capedfuzz (previous) (diff)

comment:11 Changed 10 months ago by capedfuzz (diosmosis)

In 3a3517e48823a4d7dc521e5a754c51ddbcf5d210:

Refs #3124, add benchmark for MultiSites.getAll and benchmark fixture that adds 20,000 sites to the DB then tracks one pageview per site.

comment:12 Changed 10 months ago by matt (mattab)

Awesome Benaka, truly beautiful!

Note: See TracTickets for help on using tickets.