Performance: Fix memory leak during Archiving process

[matt]

The following php memory error is showing up when there is too much data.

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 35 bytes) in /home/www/piwik/demo/core/DataTable.php on line 807

There are improvements to be made in the archiving process to make it more memory / computation efficient.

[konus]

[watching this topic] I have the same problem with month and year view, because my shared hosting allows only 32MB Php-Memory (was ticket:371 )

[matt]

fixed in 0.2.26

[konus]

I am sorry, but I have still some problems. Generating grafics for Month doesn't work. Generating grafics and calculation numbers for Year doesn't work. And in visitors>overview I get:

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 71 bytes) in /var/www/web5/html/piwik/core/DataTable.php on line 822

[vipsoft]

Is this still occuring in 0.2.28?

[konus]

Glad you ask. I have no errors within Month and year views anymore. (But I also have 64MB php memory now)

However, Piwik is still not able to display the sparklines and the grafics "last visitors" and "returnig visitors". I think it could be some kind of timeout.

[vipsoft]

Firefox's Error Console reports:

Warning: 09 is not a legal ECMA-262 octal constant Source File:  http://servername/index.php?module=CoreHome&action=index&idSite=1&period=month&date=yesterday Line: 40, Column: 132 Source Code:

piwik.idSite = "1"; piwik.period = "month"; piwik.currentDateString = "2009-01-17"; piwik.minDateYear = 2008; piwik.minDateMonth = 09; piwik.minDateDay = 11;

[vipsoft]

In plugins/CoreHome/templates/js_global_variables.tpl, change these lines:

        {if isset($minDateMonth)}piwik.minDateMonth = {$minDateMonth};{/if}
        {if isset($minDateDay)}piwik.minDateDay = {$minDateDay};{/if}

to:

        {if isset($minDateMonth)}piwik.minDateMonth = parseInt({$minDateMonth}, 10);{/if}
        {if isset($minDateDay)}piwik.minDateDay = parseInt({$minDateDay}, 10);{/if}

[konus]

Thanks for the help!!! I changed the line, but the error was still there. Then I changed it to:

{if isset($minDateMonth)}piwik.minDateMonth = parseInt("{$minDateMonth}", 10);{/if}
{if isset($minDateDay)}piwik.minDateDay = parseInt("{$minDateDay}", 10);{/if}

Now the error is gone, but I have still no flash chart with the last visitors :-(

Anyway it looks like this is problem is not connected with the memory leak?

[vipsoft]

This is the error from the data URL from your Last visits graph and Visitors Evolution graph (day=yesterday, period=month):

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 72 bytes) in /var/www/web5/html/piwik/core/DataTable.php on line 836

Matt?

[matt]

vipsoft, what happens is that, for a given website, we store all data in memory (in ArchiveProcessing_Record_*), and at the end of processing all the processed statistics for a given website, we store all this data in DB.

What we could do is: * process each piece of data (ie. visits by keywords) * directly save it in DB * flush it from memory

Downside is that we would not have this data anymore, if other plugins want to access data that has just been processed by other plugins, to enrich/change it, for example. However this use case is not yet found in Piwik code, so we could assume it is a change we can make.

What would be changed in Piwik_ArchiveProcessing_Record_Manager:

• get rid of Piwik_ArchiveProcessing_Record_Manager class,
• after each creating each Piwik_ArchiveProcessing_Record ( new Piwik_ArchiveProcessing_Record_*) put $archiveProcessing->insertRecord()

[matt]

konus, vipsoft, committed octal-javascript bug in [898]

[jebbis]

Hello,

I've also got this problem, and I had to re-install Piwik. I do not think that is a good option, I do think I will get the same problem again when the archive is big enough.

Is there any work-around for this issue?

[matt]

(In [987]) - refs #374 test post commit

[matt]

(In [1035]) - memory and speed optimizations of archiving, refs #374 (hopefully fixing it but not sure yet) - added support for profiling memory & time in Piwik via the events mechanism - small other changes

[matt]

fixed! the culprit was PHP bug  http://bugs.php.net/bug.php?id=33595 to get around I had to follow recommendations on  http://paul-m-jones.com/?p=262

apparently bug has been fixed in php 5.3 so we shall see if php is dealing better with deleting old objects with cross references.

the memory leak problem is GONE! we were able to process stats of piwik.org for the last few months and year without any problem for 100k+ unique visitors

[konus]

Thank you very much! I am happy to report, that piwik is now reporting within month and year timeframe.

[macada]

Hi guys, I have the same problem. My piwik has only a domain with 11.000 unique visitors / avg a day. I dont know if this issue will fix in the next release or not but I need a workaround and I think wait for 5.3 php release is not a good idea.

any help please?

Cheers.

[vipsoft]

macada: there is a fix in Piwik that avoids this particular memory leak situation, and does not require php 5.3. If you are still encountering a problem with archiving, please open a new ticket and provide detailed information.

[konus]

The new ticket is #685

[matt]

This ticket was reopened in #766