#3438 closed Bug (worksforme)

Stats randomly revert to zero

Reported by: Mooash Owned by:
Priority: normal Milestone: 1.12 - The Great 1.x Backlog
Component: Core Keywords:
Cc: Sensitive: no

Description

PHP Version: 5.2.17

I've noticed whilst running Piwik 1.9-b10 that after running a cronjob the current week/month for websites that haven't received a visit in sometime revert to zero.

Once the website gets a new visit they repopulate to what they should be, IE the stats for the week/month are no longer zero and populate to what they were before.

I've attached images explaining what I mean

Attachments (3)

before visit.PNG (18.3 KB) - added by Mooash 18 months ago.
Before visits
after 1 visit.PNG (19.8 KB) - added by Mooash 18 months ago.
After 1 visit and the cronjob has been run again
piwik-archive.log (69.8 KB) - added by phoenix 17 months ago.
Piwik Archive Log

Download all attachments as: .zip

Change History (34)

Changed 18 months ago by Mooash

Before visits

Changed 18 months ago by Mooash

After 1 visit and the cronjob has been run again

comment:1 Changed 18 months ago by matt (mattab)

  • Resolution set to worksforme
  • Status changed from new to closed

Do you still experience the issue? what value do you use in Settings> general settings > timeout ? try 3600 ? marking as works-for-me but I can reopen if you think there is a bug, because there might be.

comment:2 Changed 18 months ago by phoenix

  • Resolution worksforme deleted
  • Status changed from closed to reopened

Hi,

I have the same problem, and i have the last version of piwik.

For site with regular visit, i have statistic on the month. But for site with fewer visit (one visit by week by example), the statistic is reset to zero for the month or the week. If i look at day level, or at year level, i can see visit, but at month or week level, visit will be set to zero.

If i run archive.php with --force-all-websites and --force-all-periods, visit are correctly refresh.

Timeout is set to 3600.

How can i resolv this problem.

Thanks

comment:3 Changed 18 months ago by phoenix

Finally, years are set to zero too. So for site with fewer visit, i can look only day stat.

comment:4 Changed 18 months ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from reopened to closed

please try with latest version from: http://builds.piwik.org/?C=N;O=D

this bug should be fixed. reopen if you tsill experience issue and you are using 1.9.2-b3 or greater

comment:5 Changed 17 months ago by phoenix

Hi,

I have the last version of piwik 1.9.2.

I always have this problem for fewer visit site. What information can i give you to find the problem.

comment:6 Changed 17 months ago by phoenix

  • Resolution fixed deleted
  • Status changed from closed to reopened

comment:7 Changed 17 months ago by Mooash

Just upgraded to the latest beta build. I'll be testing it over the next week or so.

comment:8 Changed 17 months ago by Mooash

Yeah I'm still having this issue. A website with only 5 vists this week keeps reverting back to zero until I run a --force-all-websites and --force-all-periods archive.

I'm on 1.9.3-b4

comment:9 Changed 17 months ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from reopened to closed

(In [7481]) I think this should fix #3438 because indeed there was a bug in the code.
Please try with the new archive.php from svn and reopen ticket if you still have this problem after one day running the new archive.php.

comment:10 Changed 17 months ago by phoenix

Sorry,

I'm with the last 1.9.3-b4, with the last version of archive.php from the svn trunk, but i still have the problem.

Site with no visit in the day, revert to zero at the night.

comment:11 Changed 17 months ago by phoenix

  • Resolution fixed deleted
  • Status changed from closed to reopened

comment:12 Changed 17 months ago by Mooash

I can confirm that I'm still having this issue as well. I'm using the svn archive.php with 1.9.3-b4 everything else

If you want access to my Piwik install then just give me a heads up and I'll give you access

comment:13 Changed 17 months ago by matt (mattab)

It didnt show up here but I committed a fix in: [7481]

Please try to grab the latest archive.php from SVN: http://dev.piwik.org/svn/trunk/misc/cron/archive.php

Then wait 24 hours and it should work OK!

I leave the bug opened, but please close it if you find it's working for you since I think it is now fixed! thanks for your help testing

comment:14 Changed 17 months ago by matt (mattab)

  • Milestone changed from 1.9.2 - Piwik 1.9.2 to 1.9.3 - Piwik 1.9.3

comment:15 Changed 17 months ago by phoenix

Sorry, i already have the last file.
The last time, i have load in one hand the beta zip 1.9.3-b4 and in an another hand the archive.php from http://dev.piwik.org/svn/trunk/misc/cron/archive.php

If i redownload the file and make a diff

mv archive.php archive.php.old
wget http://dev.piwik.org/svn/trunk/misc/cron/archive.php
diff archive.php archive.php.old

There is no difference with the version i have already downloaded and the current version.

And unfortunately i still have the problem. But if i can give you something to help (sql dump, zip, log), ask me.

Last edited 17 months ago by phoenix (previous) (diff)

comment:16 Changed 17 months ago by matt (mattab)

What would be very help ful would be to store all archive.php output logs appended after another in the log file. To do so replace < with << in the cron line.

Then, can you send the lines from the archive.php run that show that the website that had visits was still not re-processed when it should have been?

Hope it's not too complicated. I would really like to understand and fix this bug!

comment:17 Changed 17 months ago by Mooash

I'll change this in my cronjob later tonight.

Changed 17 months ago by phoenix

Piwik Archive Log

comment:18 Changed 17 months ago by phoenix

I attach the file. At 23:45, site have good stats, not this morning.

I think the problem can from

[2012-11-21 00:05:05] [6dc9850c] task,output Piwik_CoreAdminHome.purgeOutdatedArchives,Time elapsed: 0.830s Piwik_PDFReports.dailySchedule,Time elapsed: 0.036s Piwik_CoreAdminHome.optimizeArchiveTable,Time elapsed: 1.709s

but i'm not sure.

If you want make a more verbose version to test, i can test it.

comment:19 follow-up: Changed 17 months ago by matt (mattab)

@phoenix thanks that is very interesting. What timezones are your websites set to in piwik ?

also can you check that your server is at the right datetime ? also check that the mysql server is at the same datetime as php server ?

If all is good on your side i'll investigate further for sure, I'm really curious what the bug could be.

comment:20 in reply to: ↑ 19 Changed 17 months ago by phoenix

Replying to matt:

@phoenix thanks that is very interesting. What timezones are your websites set to in piwik ?

I have only one server that contains all my website, piwik, and mysql database. This server is at UTC+0100 (Europe/Paris).

also can you check that your server is at the right datetime ? also check that the mysql server is at the same datetime as php server ?

The Linux command date give me the right date and hour.

If all is good on your side i'll investigate further for sure, I'm really curious what the bug could be.

comment:21 Changed 17 months ago by Mooash

I've been logging the archive output, it doesn't show any errors. However I'm still needing to run the --force-all-websites --force-all-periods every now and then. It doesn't look like I'll be much help.

I'm still in the same boat, a website will revert to zero but once it gets a visit it returns to what it should.

comment:22 Changed 17 months ago by Mooash

I've found an error! See here!

My stats have since been reverted to zero for all websites but one. I assume this error caused an issue?

comment:23 Changed 17 months ago by Mooash

Also sorry, I got this as well but I assume thats meant to be there. This code I'm adding now did not appear when the code I posted above appeared. They happened on different archive attempts.

[2012-11-24 00:03:33] [48abce63] task,output Piwik_CoreAdminHome.purgeOutdatedArchives,Time elapsed: 1.076s Piwik_PDFReports.dailySchedule,Time elapsed: 1.807s Piwik_PrivacyManager.deleteReportData,Time elapsed: 0.002s Piwik_PrivacyManager.deleteLogData,Time elapsed: 0.000s Piwik_CoreAdminHome.optimizeArchiveTable,Time elapsed: 4.151s

Its worth noting that I have Piwik set to NOT delete old visitor data or anything. Piwik doesn't delete anything from my database.

comment:24 Changed 17 months ago by Mooash

Still having this issue. See here and then here

The second image is after I ran my cronjob with --force-all-websites --force-all-periods. I've been checking my cronjob and nothing weird happens - theres no errors or anything out of the ordinary.

comment:25 Changed 16 months ago by phoenix

Sorry, the problem come when process all web site at 23:05 in the log. As index.php contains @date_default_timezone_set('UTC'); we are in UTC time zone in the log, and not in the server time zone. So at 23:05 we are in france à 0:05.

Unfortunately I do not know enough about the internal of piwik to debug the problem by myself.

The part that reset visit, in the log :

[2012-11-20 23:05:02] [50d2df27] ---------------------------
[2012-11-20 23:05:02] [50d2df27] INIT
[2012-11-20 23:05:02] [50d2df27] Querying Piwik API at: xxxxxxxxxxxxxxxxxxxxxxx
[2012-11-20 23:05:02] [50d2df27] Running as Super User: xxxxxxx
[2012-11-20 23:05:02] [50d2df27] Notes
[2012-11-20 23:05:02] [50d2df27] - Reports for today will be processed at most every 3600 seconds. You can change this value in Piwik UI > Settings > General Settings.
[2012-11-20 23:05:02] [50d2df27] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
[2012-11-20 23:05:02] [50d2df27] - Archiving was last executed without error 59 min 14s ago
[2012-11-20 23:05:03] [50d2df27] Will process 1 websites with new visits since 59 min 15s , IDs: 10
[2012-11-20 23:05:03] [50d2df27] Will process 9 other websites because the last time they were archived was on a different day (in the website's timezone) , IDs: 1, 3, 7, 8, 9, 11, 15, 17, 18
[2012-11-20 23:05:03] [50d2df27] ---------------------------
[2012-11-20 23:05:03] [50d2df27] START
[2012-11-20 23:05:03] [50d2df27] Starting Piwik reports archiving...
[2012-11-20 23:05:03] [50d2df27] Skipped website id 10, already processed today's report in recent run, 59 min 40s ago, Time elapsed: 0.002s
[2012-11-20 23:05:03] [50d2df27] Archived website id = 1, period = day, Time elapsed: 0.918s
[2012-11-20 23:05:04] [50d2df27] Archived website id = 1, period = week, 133 visits, Time elapsed: 0.338s
[2012-11-20 23:05:04] [50d2df27] Archived website id = 1, period = month, 686 visits, Time elapsed: 0.387s
[2012-11-20 23:05:05] [50d2df27] Archived website id = 1, period = year, 9232 visits, Time elapsed: 0.468s
[2012-11-20 23:05:05] [50d2df27] Archived website id = 1, today = 0 visits, 4 API requests, Time elapsed: 2.113s [2/10 done]
[2012-11-20 23:05:05] [50d2df27] Archived website id = 3, period = day, Time elapsed: 0.738s
[2012-11-20 23:05:06] [50d2df27] Archived website id = 3, period = week, 65 visits, Time elapsed: 0.316s
[2012-11-20 23:05:06] [50d2df27] Archived website id = 3, period = month, 353 visits, Time elapsed: 0.357s
[2012-11-20 23:05:06] [50d2df27] Archived website id = 3, period = year, 3411 visits, Time elapsed: 0.387s
[2012-11-20 23:05:06] [50d2df27] Archived website id = 3, today = 0 visits, 4 API requests, Time elapsed: 1.801s [3/10 done]
[2012-11-20 23:05:07] [50d2df27] Archived website id = 7, period = day, Time elapsed: 0.815s
[2012-11-20 23:05:08] [50d2df27] Archived website id = 7, period = week, 3 visits, Time elapsed: 0.962s
[2012-11-20 23:05:11] [50d2df27] Archived website id = 7, period = month, 16 visits, Time elapsed: 2.811s
[2012-11-20 23:05:16] [50d2df27] Archived website id = 7, period = year, 703 visits, Time elapsed: 4.778s
[2012-11-20 23:05:16] [50d2df27] Archived website id = 7, today = 0 visits, 4 API requests, Time elapsed: 9.369s [4/10 done]
[2012-11-20 23:05:16] [50d2df27] Archived website id = 8, period = day, Time elapsed: 0.350s
[2012-11-20 23:05:17] [50d2df27] Archived website id = 8, period = week, 4 visits, Time elapsed: 0.398s
[2012-11-20 23:05:19] [50d2df27] Archived website id = 8, period = month, 15 visits, Time elapsed: 2.621s
[2012-11-20 23:05:23] [50d2df27] Archived website id = 8, period = year, 165 visits, Time elapsed: 3.648s
[2012-11-20 23:05:23] [50d2df27] Archived website id = 8, today = 0 visits, 4 API requests, Time elapsed: 7.020s [5/10 done]
[2012-11-20 23:05:23] [50d2df27] Archived website id = 9, period = day, Time elapsed: 0.404s
[2012-11-20 23:05:24] [50d2df27] Archived website id = 9, period = week, 1 visits, Time elapsed: 0.399s
[2012-11-20 23:05:26] [50d2df27] Archived website id = 9, period = month, 3 visits, Time elapsed: 1.901s
[2012-11-20 23:05:29] [50d2df27] Archived website id = 9, period = year, 824 visits, Time elapsed: 3.254s
[2012-11-20 23:05:29] [50d2df27] Archived website id = 9, today = 0 visits, 4 API requests, Time elapsed: 5.960s [6/10 done]
[2012-11-20 23:05:29] [50d2df27] Archived website id = 11, period = day, Time elapsed: 0.376s
[2012-11-20 23:05:30] [50d2df27] Archived website id = 11, period = week, 0 visits, Time elapsed: 0.398s
[2012-11-20 23:05:31] [50d2df27] Archived website id = 11, period = month, 6 visits, Time elapsed: 1.425s
[2012-11-20 23:05:33] [50d2df27] Archived website id = 11, period = year, 50 visits, Time elapsed: 1.550s
[2012-11-20 23:05:33] [50d2df27] Archived website id = 11, today = 0 visits, 4 API requests, Time elapsed: 3.751s [7/10 done]
[2012-11-20 23:05:33] [50d2df27] Archived website id = 15, period = day, Time elapsed: 0.205s
[2012-11-20 23:05:33] [50d2df27] Archived website id = 15, period = week, 0 visits, Time elapsed: 0.219s
[2012-11-20 23:05:33] [50d2df27] Archived website id = 15, period = month, 0 visits, Time elapsed: 0.364s
[2012-11-20 23:05:34] [50d2df27] Archived website id = 15, period = year, 0 visits, Time elapsed: 0.342s
[2012-11-20 23:05:34] [50d2df27] Archived website id = 15, today = 0 visits, 4 API requests, Time elapsed: 1.132s [8/10 done]
[2012-11-20 23:05:34] [50d2df27] Archived website id = 17, period = day, Time elapsed: 0.204s
[2012-11-20 23:05:34] [50d2df27] Archived website id = 17, period = week, 0 visits, Time elapsed: 0.219s
[2012-11-20 23:05:34] [50d2df27] Archived website id = 17, period = month, 0 visits, Time elapsed: 0.375s
[2012-11-20 23:05:35] [50d2df27] Archived website id = 17, period = year, 2 visits, Time elapsed: 0.788s
[2012-11-20 23:05:35] [50d2df27] Archived website id = 17, today = 0 visits, 4 API requests, Time elapsed: 1.588s [9/10 done]
[2012-11-20 23:05:35] [50d2df27] Archived website id = 18, period = day, Time elapsed: 0.194s
[2012-11-20 23:05:36] [50d2df27] Archived website id = 18, period = week, 0 visits, Time elapsed: 0.215s
[2012-11-20 23:05:36] [50d2df27] Archived website id = 18, period = month, 1 visits, Time elapsed: 0.375s
[2012-11-20 23:05:37] [50d2df27] Archived website id = 18, period = year, 5 visits, Time elapsed: 0.894s
[2012-11-20 23:05:37] [50d2df27] Archived website id = 18, today = 0 visits, 4 API requests, Time elapsed: 1.680s [10/10 done]
[2012-11-20 23:05:37] [50d2df27] Done archiving!
[2012-11-20 23:05:37] [50d2df27] ---------------------------
[2012-11-20 23:05:37] [50d2df27] SUMMARY
[2012-11-20 23:05:37] [50d2df27] Total daily visits archived: 0
[2012-11-20 23:05:37] [50d2df27] Archived today's reports for 9 websites
[2012-11-20 23:05:37] [50d2df27] Archived week/month/year for 9 websites. 
[2012-11-20 23:05:37] [50d2df27] Skipped 1 websites: no new visit since the last script execution
[2012-11-20 23:05:37] [50d2df27] Skipped 1 websites day archiving: existing daily reports are less than 3600 seconds old
[2012-11-20 23:05:37] [50d2df27] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
[2012-11-20 23:05:37] [50d2df27] Total API requests: 36
[2012-11-20 23:05:37] [50d2df27] done: 9/10 90%, 0 v, 9 wtoday, 9 wperiods, 36 req, 34423 ms, no error
[2012-11-20 23:05:37] [50d2df27] Time elapsed: 34.423s
[2012-11-20 23:05:37] [50d2df27] ---------------------------
[2012-11-20 23:05:37] [50d2df27] SCHEDULED TASKS
[2012-11-20 23:05:37] [50d2df27] Starting Scheduled tasks... 
[2012-11-20 23:05:37] [50d2df27]  No task to run
[2012-11-20 23:05:37] [50d2df27] done

comment:26 Changed 16 months ago by matt (mattab)

I think it might be the scheduled task Piwik_CoreAdminHome.purgeOutdatedArchives that would purge too much data, but I'm not sure. I haven't yet been able to reproduce the issue myself. Keep posting more info if you have!

comment:27 Changed 16 months ago by Mooash

Thats weird, because I don't have purge old log data on. So it shouldn't actually be purging anything...

comment:28 Changed 15 months ago by Mooash

I think I may have found what could be causing the problem. I tracked a few proxies using Piwik that generated a whole new URL each time they visited a different page so I was literally tracking thousands of different pages a day.

I've since fixed this issue but my stats are still reverting, is there a way I can go through a change all the proxies's pages that start in browse.php to JUST browse.php so that there is alot less pages in Piwik?

comment:29 Changed 15 months ago by matt (mattab)

  • Resolution set to answered
  • Status changed from reopened to closed

you could write a SQL query that will look for all pages in log_action containing browse.php? and delete these IDs from log_link_visit_action (delete these page views). sorry wont write SQL & test, but if you need help: http://piwik.org/consulting/

comment:30 Changed 15 months ago by Mooash

  • Resolution answered deleted
  • Status changed from closed to reopened

Sorry, but this issue appears to still be present.

I've noticed the issue happens when it does its daily tasks which are as follows:

[2013-01-20 01:00:09] [7bfe734c] task,output Piwik_CoreAdminHome.purgeOutdatedArchives,Time elapsed: 3.076s Piwik_PDFReports.dailySchedule,Time elapsed: 2.372s Piwik_PrivacyManager.deleteReportData,Time elapsed: 0.002s Piwik_PrivacyManager.deleteLogData,Time elapsed: 0.000s Piwik_CoreAdminHome.optimizeArchiveTable,Time elapsed: 1.671s

Could it be that it doesn't have sufficient MySQL privileges or there is something wrong with my MySQL installation? I'm running it on a cPanel server so it could be that cPanel is locking it down to much.

Whats also interesting is that I've got Piwik set to delete nothing, I retain all data, but it still purges outdated archives? I assume this just means that it updates them or does it delete them?

comment:31 Changed 13 months ago by matt (mattab)

  • Resolution set to worksforme
  • Status changed from reopened to closed

purgeOutdatedArchives should only remove the "partial" reports which were processed for "unfinished days". do you still experience the issue using Piwik 1.12-beta5 here: http://builds.piwik.org/piwik-1.12-b5.zip

If so please re-open and thanks for your patience

Note: See TracTickets for help on using tickets.