Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

archive.sh script should loop over all websites and trigger archiving in a separate request for each #1227

Closed
mattab opened this issue Mar 21, 2010 · 6 comments
Labels
Bug For errors / faults / flaws / inconsistencies etc. Critical Indicates the severity of an issue is very critical and the issue has a very high priority.
Milestone

Comments

@mattab
Copy link
Member

mattab commented Mar 21, 2010

Currently, archive.sh simply loops over the periods and triggers archiving for all websites: https://github.com/piwik/piwik/blob/master/misc/cron/archive.sh

because there are memory leaks issues during archiving, see #766, when a Piwik has hundreds or thousands of websites in the system, the memory usage peaks and can lead to ridiculous values.

Until #766 is addressed and as a useful safe guard, we should looop over each website and trigger archiving separately for each. This is a quick fix that will help many users get past the memory usage issues.

@anonymous-matomo-user
Copy link

Subscribing to this ticket as I would really appreciate that method. Let me know if I can / should test something :)

@mattab
Copy link
Member Author

mattab commented Mar 31, 2010

(In [2025]) Fixes #1227
the archive script now loops over all websites and triggers a different request for each archive, helping out with the archive memory exhausted issue (refs #766)

@anonymous-matomo-user
Copy link

Thanks for your work on this. Overall I would say its better. Its still aching slow running over the sites and my most active site (15k visits / 20k actions per day, out of around 150k per day) is taking 2 GB to process the yearly stats.

@anonymous-matomo-user
Copy link

I tried the new archive.sh script, but it didn't work for me, because the list of site id's seems to be utf encoded it doesn't pass the isnumeric test. The convertToUnicode=0 param to the api call doesn't seem to do anything here.

@mattab
Copy link
Member Author

mattab commented Apr 8, 2010

alx, did you apply the full patch? only applying a subset will obviously fail. does it work for you?

@anonymous-matomo-user
Copy link

Oops, my bad. I didn't apply the full patch, sorry 'bout that.

@mattab mattab added this to the Piwik 0.6 milestone Jul 8, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc. Critical Indicates the severity of an issue is very critical and the issue has a very high priority.
Projects
None yet
Development

No branches or pull requests

2 participants