Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#1227 closed Bug (fixed)

archive.sh script should loop over all websites and trigger archiving in a separate request for each

Reported by: matt Owned by:
Priority: critical Milestone: Piwik 0.6
Component: Core Keywords:
Cc: Sensitive: no

Description

Currently, archive.sh simply loops over the periods and triggers archiving for all websites: http://dev.piwik.org/trac/browser/trunk/misc/cron/archive.sh

because there are memory leaks issues during archiving, see #766, when a Piwik has hundreds or thousands of websites in the system, the memory usage peaks and can lead to ridiculous values.

Until #766 is addressed and as a useful safe guard, we should looop over each website and trigger archiving separately for each. This is a quick fix that will help many users get past the memory usage issues.

Change History (6)

comment:1 Changed 4 years ago by ts77

Subscribing to this ticket as I would really appreciate that method. Let me know if I can / should test something :)

comment:2 Changed 4 years ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from new to closed

(In [2025]) Fixes #1227
the archive script now loops over all websites and triggers a different request for each archive, helping out with the archive memory exhausted issue (refs #766)

comment:3 Changed 4 years ago by ts77

Thanks for your work on this. Overall I would say its better. Its still aching slow running over the sites and my most active site (15k visits / 20k actions per day, out of around 150k per day) is taking 2 GB to process the yearly stats.

comment:4 Changed 4 years ago by alx

I tried the new archive.sh script, but it didn't work for me, because the list of site id's seems to be utf encoded it doesn't pass the isnumeric test. The convertToUnicode=0 param to the api call doesn't seem to do anything here.

comment:5 Changed 4 years ago by matt (mattab)

alx, did you apply the full patch? only applying a subset will obviously fail. does it work for you?

comment:6 Changed 4 years ago by alx

Oops, my bad. I didn't apply the full patch, sorry 'bout that.

Note: See TracTickets for help on using tickets.