Opened 19 months ago

Closed 6 months ago

Last modified 2 weeks ago

#3405 closed Bug (fixed)

when archive.php is run multiple times, each concurrent run should archive different websites

Reported by: matt Owned by: mattab
Priority: critical Milestone: 2.0 - Piwik 2.0
Component: Core Keywords:
Cc: Sensitive: no

Description

Reported in forums

The goal is to make archiving faster for high traffic piwik server, especially those with thousands of websites. They can exceute php /path/to/cron/archive.php multiple times in parallel, and each instance will archive different websites. Overall it should be much faster than archiving one after the other.

NOte: it used to work as I tested this case when building the feature, but was broken at some point.

Change History (11)

comment:1 Changed 19 months ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from new to closed

(In [7042]) Fixes #3405 Skipping websites which have been processed by another concurrent archive.php

comment:2 Changed 19 months ago by pmontana

Will it work if archive.php is launched on different servers but for the "same" database ?

comment:3 Changed 19 months ago by matt (mattab)

It should work yes!

comment:4 Changed 19 months ago by mstenz

This is a great thing :)

comment:5 Changed 6 months ago by matt (mattab)

  • Resolution fixed deleted
  • Status changed from closed to reopened

It's not working in 2.0

comment:6 Changed 6 months ago by matt (mattab)

  • Milestone changed from 1.9 -- Piwik 1.9 to 2.0 - Piwik 2.0
  • Priority changed from normal to critical

comment:7 Changed 6 months ago by matt (mattab)

In 0ea3b0e69954b7a4c57c43da0e72d70ba0678f43:

Refs #3405 Refactoring archive.php so I can understand it well

comment:8 Changed 6 months ago by matt (mattab)

  • Owner set to mattab
  • Resolution set to fixed
  • Status changed from reopened to closed

In 5db45de92582cfd5cef4a0c9ff6622b9f41d37b3:

Fixes #3405 Clearning up the code and fixing the logic so archive.php will process websites properly when executed concurrently!

comment:10 Changed 5 months ago by matt (mattab)

In ac3b5bf1ab198cd30b361da4e2c849b6dc5f3149:

Fixes #4309 adding --force-idsites=1,2,n parameter
Refs #3405 fixing regression in concurrent runs

comment:11 Changed 2 weeks ago by matt (mattab)

See follow up #4903 Add possibility to run multiple archiver in parallel

Note: See TracTickets for help on using tickets.