Opened 6 years ago

Closed 5 years ago

Last modified 3 years ago

#374 closed Bug (fixed)

Performance: Fix memory leak during Archiving process

Reported by: matt Owned by:
Priority: major Milestone: RobotRock
Component: Core Keywords:
Cc: Sensitive: no

Description

The following php memory error is showing up when there is too much data.

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 35 bytes) in /home/www/piwik/demo/core/DataTable.php on line 807

There are improvements to be made in the archiving process to make it more memory / computation efficient.

Change History (26)

comment:1 Changed 6 years ago by matt (mattab)

  • Milestone changed from RobotRock to DigitalVibes
  • Summary changed from Fix memory leak during Archiving process to Performance: Fix memory leak during Archiving process

comment:2 Changed 5 years ago by konus

[watching this topic]
I have the same problem with month and year view, because my shared hosting allows only 32MB Php-Memory (was ticket:371 )

comment:3 Changed 5 years ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from new to closed

fixed in 0.2.26

comment:4 Changed 5 years ago by konus

  • Resolution fixed deleted
  • Status changed from closed to reopened

I am sorry, but I have still some problems. Generating grafics for Month doesn't work. Generating grafics and calculation numbers for Year doesn't work. And in visitors>overview I get:

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 71 bytes) in /var/www/web5/html/piwik/core/DataTable.php on line 822

comment:5 Changed 5 years ago by vipsoft (robocoder)

Is this still occuring in 0.2.28?

comment:6 Changed 5 years ago by konus

Glad you ask. I have no errors within Month and year views anymore. (But I also have 64MB php memory now)

However, Piwik is still not able to display the sparklines and the grafics "last visitors" and "returnig visitors". I think it could be some kind of timeout.

comment:7 Changed 5 years ago by vipsoft (robocoder)

Firefox's Error Console reports:

Warning: 09 is not a legal ECMA-262 octal constant
Source File: http://servername/index.php?module=CoreHome&action=index&idSite=1&period=month&date=yesterday
Line: 40, Column: 132
Source Code:

piwik.idSite = "1"; piwik.period = "month"; piwik.currentDateString = "2009-01-17"; piwik.minDateYear = 2008; piwik.minDateMonth = 09; piwik.minDateDay = 11;

comment:8 Changed 5 years ago by vipsoft (robocoder)

In plugins/CoreHome/templates/js_global_variables.tpl, change these lines:

        {if isset($minDateMonth)}piwik.minDateMonth = {$minDateMonth};{/if}
        {if isset($minDateDay)}piwik.minDateDay = {$minDateDay};{/if}

to:

        {if isset($minDateMonth)}piwik.minDateMonth = parseInt({$minDateMonth}, 10);{/if}
        {if isset($minDateDay)}piwik.minDateDay = parseInt({$minDateDay}, 10);{/if}

comment:9 Changed 5 years ago by konus

Thanks for the help!!! I changed the line, but the error was still there. Then I changed it to:

{if isset($minDateMonth)}piwik.minDateMonth = parseInt("{$minDateMonth}", 10);{/if}
{if isset($minDateDay)}piwik.minDateDay = parseInt("{$minDateDay}", 10);{/if}

Now the error is gone, but I have still no flash chart with the last visitors :-(

Anyway it looks like this is problem is not connected with the memory leak?

comment:10 Changed 5 years ago by vipsoft (robocoder)

This is the error from the data URL from your Last visits graph and Visitors Evolution graph (day=yesterday, period=month):

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 72 bytes) in /var/www/web5/html/piwik/core/DataTable.php on line 836

Matt?

comment:11 Changed 5 years ago by matt (mattab)

vipsoft, what happens is that, for a given website, we store all data in memory (in ArchiveProcessing_Record_*), and at the end of processing all the processed statistics for a given website, we store all this data in DB.

What we could do is:

  • process each piece of data (ie. visits by keywords)
  • directly save it in DB
  • flush it from memory

Downside is that we would not have this data anymore, if other plugins want to access data that has just been processed by other plugins, to enrich/change it, for example. However this use case is not yet found in Piwik code, so we could assume it is a change we can make.

What would be changed in Piwik_ArchiveProcessing_Record_Manager:

  • get rid of Piwik_ArchiveProcessing_Record_Manager class,
  • after each creating each Piwik_ArchiveProcessing_Record ( new Piwik_ArchiveProcessing_Record_*) put $archiveProcessing->insertRecord()

comment:12 Changed 5 years ago by matt (mattab)

konus, vipsoft, committed octal-javascript bug in [898]

comment:13 in reply to: ↑ description Changed 5 years ago by jebbis

Hello,

I've also got this problem, and I had to re-install Piwik. I do not think that is a good option, I do think I will get the same problem again when the archive is big enough.

Is there any work-around for this issue?

comment:14 Changed 5 years ago by matt (mattab)

  • Milestone changed from DigitalVibes to RobotRock

comment:15 Changed 5 years ago by matt (mattab)

(In [987]) - refs #374 test post commit

comment:16 Changed 5 years ago by matt (mattab)

(In [1035]) - memory and speed optimizations of archiving, refs #374 (hopefully fixing it but not sure yet)

  • added support for profiling memory & time in Piwik via the events mechanism
  • small other changes

comment:17 Changed 5 years ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from reopened to closed

fixed!
the culprit was PHP bug http://bugs.php.net/bug.php?id=33595
to get around I had to follow recommendations on http://paul-m-jones.com/?p=262

apparently bug has been fixed in php 5.3 so we shall see if php is dealing better with deleting old objects with cross references.

the memory leak problem is GONE! we were able to process stats of piwik.org for the last few months and year without any problem for 100k+ unique visitors

comment:18 Changed 5 years ago by konus

Thank you very much! I am happy to report, that piwik is now reporting within month and year timeframe.

comment:19 Changed 5 years ago by macada

  • Resolution fixed deleted
  • Status changed from closed to reopened

Hi guys,
I have the same problem. My piwik has only a domain with 11.000 unique visitors / avg a day.
I dont know if this issue will fix in the next release or not but I need a workaround and I think wait for 5.3 php release is not a good idea.

any help please?

Cheers.

comment:20 Changed 5 years ago by vipsoft (robocoder)

  • Resolution set to fixed
  • Status changed from reopened to closed

macada: there is a fix in Piwik that avoids this particular memory leak situation, and does not require php 5.3. If you are still encountering a problem with archiving, please open a new ticket and provide detailed information.

comment:21 Changed 5 years ago by konus

The new ticket is #685

comment:22 Changed 5 years ago by alivenk

comment:23 Changed 5 years ago by spomoni

comment:24 Changed 5 years ago by spomoni

comment:25 Changed 4 years ago by matt (mattab)

  • Sensitive unset

This ticket was reopened in #766

comment:26 Changed 3 years ago by konus

Note: See TracTickets for help on using tickets.