Dupe php, mysql, massive data set
|Reported by:||matt||Owned by:|
|Component:||Core||Keywords:||performance fast scalability high traffic|
Description (last modified by matt)
SEE NEW TICKET AT: #1999
This one was closed
This ticket is a place holder for all performance related notes/thoughts/feedback.
There are already a number of interesting performance improvements tickets
Here is the list:
- Piwik creates one cookie per install which is not scalable, we should build a server side DB based cookiestore: #409
- Piwik logs should be rotated into a yearly table: #5
- Bulk load Piwik logs (with a documented API) which would improve tracking performance, and make it easy to do performance testing #134
- Fix memory leak error during Piwik archiving task for larger piwik setup #766
- All Websites dashboard should work when Piwik has thousands of registered websites (currently only scales to a few hundreds websites) #1077
Building a regression and performance testing environment for Piwik
Partly described in #134.
The objective of this project is to build a automatic and reusable performance testing infrastructure, that would make it easy to generate lots of hits and big data sets
- Setup performance testing server (called "preprod") with monitoring
- Replaying logs from previous days (see #134)
- Use preprod to have a precise idea of load that Piwik can handle (pages per day, per month), the size of data in mysql
- Use preprod to run Archiving and determine where Archiving is slow, or fails (via profiling).
- Use preprod to optimize the tracking mechanism (denormalize tables, review index strategy)
- Implement quick wins and plan for bigger changes 1 day
Once we have a system to assess performance, we could answer in a specific documentation page a few of the more common questions
- Does it use a lot of bandwith on your site?
- Does it take a lot of space/memory in the MySQL database?
- Does it make your site load (and work) slower?
- Expected DB size for a website with 1k, 10k, 100k visits and 1M pages. List server configuration required to run Piwik: on a shared server / on a dedicated server. List archiving processing time examples for large traffic.
- Suggest best setting configuration for small & medium (default config) and higher traffic installs
Interesting read on scalability & performance
- Monitoring tools: nagios, mrtg, puppet, munin, ganglia.sourceforge.net (graph across multiple servers)
Change History (47)
comment:16 Changed 5 years ago by vipsoft (robocoder)
- Milestone changed from DigitalVibes to Stable release
comment:17 Changed 5 years ago by matt (mattab)
- Description modified (diff)
- Summary changed from Performance improvement to Scaling and Performance improvement - php, mysql, massive data set
comment:19 Changed 5 years ago by matt (mattab)
- Summary changed from Scaling and Performance improvement - php, mysql, massive data set to Scaling Piwik - Performance improvement - php, mysql, massive data set
comment:44 Changed 4 years ago by matt (mattab)
- Milestone changed from 4 - Piwik 1.0 - Stable release to Features requests - after Piwik 1.0
comment:46 Changed 3 years ago by matt (mattab)
- Resolution set to invalid
- Status changed from new to closed
- Summary changed from Scaling Piwik - Performance improvement - php, mysql, massive data set to Dupe php, mysql, massive data set