Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance: Fix memory leak during Archiving process #5794

Closed
mattab opened this issue Oct 8, 2008 · 20 comments
Closed

Performance: Fix memory leak during Archiving process #5794

mattab opened this issue Oct 8, 2008 · 20 comments
Labels
Bug For errors / faults / flaws / inconsistencies etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.
Milestone

Comments

@mattab
Copy link
Member

mattab commented Oct 8, 2008

The following php memory error is showing up when there is too much data.

``

`
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 35 bytes) in /home/www/piwik/demo/core/DataTable.php on line 807

There are improvements to be made in the archiving process to make it more memory / computation efficient.
``````

@anonymous-matomo-user
Copy link

[this topic](watching)
I have the same problem with month and year view, because my shared hosting allows only 32MB Php-Memory (was ticket:371 )

@mattab
Copy link
Member Author

mattab commented Nov 28, 2008

fixed in 0.2.26

@anonymous-matomo-user
Copy link

I am sorry, but I have still some problems. Generating grafics for Month doesn’t work. Generating grafics and calculation numbers for Year doesn’t work. And in visitors>overview I get:

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 71 bytes) in /var/www/web5/html/piwik/core/DataTable.php on line 822

@robocoder
Copy link
Contributor

Is this still occuring in 0.2.28?

@anonymous-matomo-user
Copy link

Glad you ask. I have no errors within Month and year views anymore. (But I also have 64MB php memory now)

However, Piwik is still not able to display the sparklines and the grafics “last visitors” and “returnig visitors”. I think it could be some kind of timeout.

@robocoder
Copy link
Contributor

Firefox’s Error Console reports:

Warning: 09 is not a legal ECMA-262 octal constant
Source File: http://servername/index.php?module=CoreHome&action=index&idSite=1&period=month&date=yesterday
Line: 40, Column: 132
Source Code:
piwik.idSite = “1”; piwik.period = “month”; piwik.currentDateString = “2009-01-17”; piwik.minDateYear = 2008; piwik.minDateMonth = 09; piwik.minDateDay = 11;

@robocoder
Copy link
Contributor

In plugins/CoreHome/templates/js_global_variables.tpl, change these lines:

```
{if isset($minDateMonth)}piwik.minDateMonth = {$minDateMonth};{/if}
{if isset($minDateDay)}piwik.minDateDay = {$minDateDay};{/if}
```

to:

```
{if isset($minDateMonth)}piwik.minDateMonth = parseInt({$minDateMonth}, 10);{/if}
{if isset($minDateDay)}piwik.minDateDay = parseInt({$minDateDay}, 10);{/if}
```

@anonymous-matomo-user
Copy link

Thanks for the help!!! I changed the line, but the error was still there. Then I changed it to:

```
{if isset($minDateMonth)}piwik.minDateMonth = parseInt(“{$minDateMonth}”, 10);{/if}
{if isset($minDateDay)}piwik.minDateDay = parseInt(“{$minDateDay}”, 10);{/if}
```

Now the error is gone, but I have still no flash chart with the last visitors :-(

Anyway it looks like this is problem is not connected with the memory leak?

@robocoder
Copy link
Contributor

This is the error from the data URL from your Last visits graph and Visitors Evolution graph (day=yesterday, period=month):

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 72 bytes) in /var/www/web5/html/piwik/core/DataTable.php on line 836

Matt?

@mattab
Copy link
Member Author

mattab commented Jan 19, 2009

vipsoft, what happens is that, for a given website, we store all data in memory (in ArchiveProcessing_Record_*), and at the end of processing all the processed statistics for a given website, we store all this data in DB.

What we could do is:
- process each piece of data (ie. visits by keywords)
- directly save it in DB
- flush it from memory

Downside is that we would not have this data anymore, if other plugins want to access data that has just been processed by other plugins, to enrich/change it, for example. However this use case is not yet found in Piwik code, so we could assume it is a change we can make.

What would be changed in Piwik_ArchiveProcessing_Record_Manager:
- get rid of Piwik_ArchiveProcessing_Record_Manager class,
- after each creating each Piwik_ArchiveProcessing_Record ( new Piwik_ArchiveProcessing_Record_*) put $archiveProcessing→insertRecord()

@mattab
Copy link
Member Author

mattab commented Feb 11, 2009

konus, vipsoft, committed octal-javascript bug in 898

@anonymous-matomo-user
Copy link

Hello,

I’ve also got this problem, and I had to re-install Piwik. I do not think that is a good option, I do think I will get the same problem again when the archive is big enough.

Is there any work-around for this issue?

@mattab
Copy link
Member Author

mattab commented Mar 17, 2009

(In 987) – refs #5794 test post commit

@mattab
Copy link
Member Author

mattab commented Mar 27, 2009

(In 1035) – memory and speed optimizations of archiving, refs #5794 (hopefully fixing it but not sure yet)
- added support for profiling memory & time in Piwik via the events mechanism
- small other changes

@mattab
Copy link
Member Author

mattab commented Mar 27, 2009

fixed!
the culprit was PHP bug http://bugs.php.net/bug.php?id=33595
to get around I had to follow recommendations on http://paul-m-jones.com/?p=262

apparently bug has been fixed in php 5.3 so we shall see if php is dealing better with deleting old objects with cross references.

the memory leak problem is GONE! we were able to process stats of piwik.org for the last few months and year without any problem for 100k+ unique visitors

@anonymous-matomo-user
Copy link

Thank you very much! I am happy to report, that piwik is now reporting within month and year timeframe.

@anonymous-matomo-user
Copy link

Hi guys,
I have the same problem. My piwik has only a domain with 11.000 unique visitors / avg a day.
I dont know if this issue will fix in the next release or not but I need a workaround and I think wait for 5.3 php release is not a good idea.

any help please?

Cheers.

@robocoder
Copy link
Contributor

macada: there is a fix in Piwik that avoids this particular memory leak situation, and does not require php 5.3. If you are still encountering a problem with archiving, please open a new ticket and provide detailed information.

@anonymous-matomo-user
Copy link

The new ticket is #685

@mattab
Copy link
Member Author

mattab commented Oct 23, 2009

This ticket was reopened in #766

@mattab mattab added this to the RobotRock milestone Jul 8, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.
Projects
None yet
Development

No branches or pull requests

3 participants