Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stats randomly revert to zero #3438

Closed
jloh opened this issue Oct 14, 2012 · 31 comments
Closed

Stats randomly revert to zero #3438

jloh opened this issue Oct 14, 2012 · 31 comments
Labels
Bug For errors / faults / flaws / inconsistencies etc. worksforme The issue cannot be reproduced and things work as intended.
Milestone

Comments

@jloh
Copy link

jloh commented Oct 14, 2012

PHP Version: 5.2.17

I've noticed whilst running Piwik 1.9-b10 that after running a cronjob the current week/month for websites that haven't received a visit in sometime revert to zero.

Once the website gets a new visit they repopulate to what they should be, IE the stats for the week/month are no longer zero and populate to what they were before.

I've attached images explaining what I mean

@jloh
Copy link
Author

jloh commented Oct 14, 2012

Attachment: Before visits
before visit.PNG

@jloh
Copy link
Author

jloh commented Oct 14, 2012

Attachment: After 1 visit and the cronjob has been run again
after 1 visit.PNG

@mattab
Copy link
Member

mattab commented Oct 19, 2012

Do you still experience the issue? what value do you use in Settings> general settings > timeout ? try 3600 ? marking as works-for-me but I can reopen if you think there is a bug, because there might be.

@anonymous-matomo-user
Copy link

Hi,

I have the same problem, and i have the last version of piwik.

For site with regular visit, i have statistic on the month. But for site with fewer visit (one visit by week by example), the statistic is reset to zero for the month or the week. If i look at day level, or at year level, i can see visit, but at month or week level, visit will be set to zero.

If i run archive.php with --force-all-websites and --force-all-periods, visit are correctly refresh.

Timeout is set to 3600.

How can i resolv this problem.

Thanks

@anonymous-matomo-user
Copy link

Finally, years are set to zero too. So for site with fewer visit, i can look only day stat.

@mattab
Copy link
Member

mattab commented Nov 9, 2012

please try with latest version from: http://builds.piwik.org/?C=N;O=D

this bug should be fixed. reopen if you tsill experience issue and you are using 1.9.2-b3 or greater

@anonymous-matomo-user
Copy link

Hi,

I have the last version of piwik 1.9.2.

I always have this problem for fewer visit site. What information can i give you to find the problem.

@jloh
Copy link
Author

jloh commented Nov 14, 2012

Just upgraded to the latest beta build. I'll be testing it over the next week or so.

@jloh
Copy link
Author

jloh commented Nov 15, 2012

Yeah I'm still having this issue. A website with only 5 vists this week keeps reverting back to zero until I run a --force-all-websites and --force-all-periods archive.

I'm on 1.9.3-b4

@mattab
Copy link
Member

mattab commented Nov 16, 2012

(In [7481]) I think this should fix #3438 because indeed there was a bug in the code.
Please try with the new archive.php from svn and reopen ticket if you still have this problem after one day running the new archive.php.

@anonymous-matomo-user
Copy link

Sorry,

I'm with the last 1.9.3-b4, with the last version of archive.php from the svn trunk, but i still have the problem.

Site with no visit in the day, revert to zero at the night.

@jloh
Copy link
Author

jloh commented Nov 18, 2012

I can confirm that I'm still having this issue as well. I'm using the svn archive.php with 1.9.3-b4 everything else

If you want access to my Piwik install then just give me a heads up and I'll give you access

@mattab
Copy link
Member

mattab commented Nov 20, 2012

It didnt show up here but I committed a fix in: [7481]

Please try to grab the latest archive.php from SVN: https://github.com/piwik/piwik/blob/master/misc/cron/archive.php

Then wait 24 hours and it should work OK!

I leave the bug opened, but please close it if you find it's working for you since I think it is now fixed! thanks for your help testing

@anonymous-matomo-user
Copy link

Sorry, i already have the last file.
The last time, i have load in one hand the beta zip 1.9.3-b4 and in an another hand the archive.php from https://github.com/piwik/piwik/blob/master/misc/cron/archive.php

If i redownload the file and make a diff

#!sh
mv archive.php archive.php.old
wget https://github.com/piwik/piwik/blob/master/misc/cron/archive.php
diff archive.php archive.php.old

There is no difference with the version i have already downloaded and the current version.

And unfortunately i still have the problem. But if i can give you something to help (sql dump, zip, log), ask me.

@mattab
Copy link
Member

mattab commented Nov 20, 2012

What would be very help ful would be to store all archive.php output logs appended after another in the log file. To do so replace < with << in the cron line.

Then, can you send the lines from the archive.php run that show that the website that had visits was still not re-processed when it should have been?

Hope it's not too complicated. I would really like to understand and fix this bug!

@jloh
Copy link
Author

jloh commented Nov 20, 2012

I'll change this in my cronjob later tonight.

@anonymous-matomo-user
Copy link

Attachment: Piwik Archive Log
piwik-archive.log

@anonymous-matomo-user
Copy link

I attach the file. At 23:45, site have good stats, not this morning.

I think the problem can from

[2012-11-21 00:05:05] [6dc9850c] task,output Piwik_CoreAdminHome.purgeOutdatedArchives,Time elapsed: 0.830s Piwik_PDFReports.dailySchedule,Time elapsed: 0.036s Piwik_CoreAdminHome.optimizeArchiveTable,Time elapsed: 1.709s

but i'm not sure.

If you want make a more verbose version to test, i can test it.

@mattab
Copy link
Member

mattab commented Nov 22, 2012

@phoenix thanks that is very interesting. What timezones are your websites set to in piwik ?

also can you check that your server is at the right datetime ? also check that the mysql server is at the same datetime as php server ?

If all is good on your side i'll investigate further for sure, I'm really curious what the bug could be.

@anonymous-matomo-user
Copy link

Replying to matt:

@phoenix thanks that is very interesting. What timezones are your websites set to in piwik ?

I have only one server that contains all my website, piwik, and mysql database. This server is at UTC+0100 (Europe/Paris).

also can you check that your server is at the right datetime ? also check that the mysql server is at the same datetime as php server ?

The Linux command date give me the right date and hour.

If all is good on your side i'll investigate further for sure, I'm really curious what the bug could be.

@jloh
Copy link
Author

jloh commented Nov 23, 2012

I've been logging the archive output, it doesn't show any errors. However I'm still needing to run the --force-all-websites --force-all-periods every now and then. It doesn't look like I'll be much help.

I'm still in the same boat, a website will revert to zero but once it gets a visit it returns to what it should.

@jloh
Copy link
Author

jloh commented Nov 24, 2012

I've found an error! See here!

My stats have since been reverted to zero for all websites but one. I assume this error caused an issue?

@jloh
Copy link
Author

jloh commented Nov 24, 2012

Also sorry, I got this as well but I assume thats meant to be there. This code I'm adding now did not appear when the code I posted above appeared. They happened on different archive attempts.

[2012-11-24 00:03:33] [48abce63] task,output Piwik_CoreAdminHome.purgeOutdatedArchives,Time elapsed: 1.076s Piwik_PDFReports.dailySchedule,Time elapsed: 1.807s Piwik_PrivacyManager.deleteReportData,Time elapsed: 0.002s Piwik_PrivacyManager.deleteLogData,Time elapsed: 0.000s Piwik_CoreAdminHome.optimizeArchiveTable,Time elapsed: 4.151s

Its worth noting that I have Piwik set to NOT delete old visitor data or anything. Piwik doesn't delete anything from my database.

@jloh
Copy link
Author

jloh commented Dec 2, 2012

Still having this issue. See here and then here

The second image is after I ran my cronjob with --force-all-websites --force-all-periods. I've been checking my cronjob and nothing weird happens - theres no errors or anything out of the ordinary.

@anonymous-matomo-user
Copy link

Sorry, the problem come when process all web site at 23:05 in the log. As index.php contains @date_default_timezone_set('UTC'); we are in UTC time zone in the log, and not in the server time zone. So at 23:05 we are in france 0:05.

Unfortunately I do not know enough about the internal of piwik to debug the problem by myself.

The part that reset visit, in the log :

[2012-11-20 23:05:02] [50d2df27] ---------------------------
[2012-11-20 23:05:02] [50d2df27] INIT
[2012-11-20 23:05:02] [50d2df27] Querying Piwik API at: xxxxxxxxxxxxxxxxxxxxxxx
[2012-11-20 23:05:02] [50d2df27] Running as Super User: xxxxxxx
[2012-11-20 23:05:02] [50d2df27] Notes
[2012-11-20 23:05:02] [50d2df27] - Reports for today will be processed at most every 3600 seconds. You can change this value in Piwik UI > Settings > General Settings.
[2012-11-20 23:05:02] [50d2df27] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
[2012-11-20 23:05:02] [50d2df27] - Archiving was last executed without error 59 min 14s ago
[2012-11-20 23:05:03] [50d2df27] Will process 1 websites with new visits since 59 min 15s , IDs: 10
[2012-11-20 23:05:03] [50d2df27] Will process 9 other websites because the last time they were archived was on a different day (in the website's timezone) , IDs: 1, 3, 7, 8, 9, 11, 15, 17, 18
[2012-11-20 23:05:03] [50d2df27] ---------------------------
[2012-11-20 23:05:03] [50d2df27] START
[2012-11-20 23:05:03] [50d2df27] Starting Piwik reports archiving...
[2012-11-20 23:05:03] [50d2df27] Skipped website id 10, already processed today's report in recent run, 59 min 40s ago, Time elapsed: 0.002s
[2012-11-20 23:05:03] [50d2df27] Archived website id = 1, period = day, Time elapsed: 0.918s
[2012-11-20 23:05:04] [50d2df27] Archived website id = 1, period = week, 133 visits, Time elapsed: 0.338s
[2012-11-20 23:05:04] [50d2df27] Archived website id = 1, period = month, 686 visits, Time elapsed: 0.387s
[2012-11-20 23:05:05] [50d2df27] Archived website id = 1, period = year, 9232 visits, Time elapsed: 0.468s
[2012-11-20 23:05:05] [50d2df27] Archived website id = 1, today = 0 visits, 4 API requests, Time elapsed: 2.113s [2/10 done]
[2012-11-20 23:05:05] [50d2df27] Archived website id = 3, period = day, Time elapsed: 0.738s
[2012-11-20 23:05:06] [50d2df27] Archived website id = 3, period = week, 65 visits, Time elapsed: 0.316s
[2012-11-20 23:05:06] [50d2df27] Archived website id = 3, period = month, 353 visits, Time elapsed: 0.357s
[2012-11-20 23:05:06] [50d2df27] Archived website id = 3, period = year, 3411 visits, Time elapsed: 0.387s
[2012-11-20 23:05:06] [50d2df27] Archived website id = 3, today = 0 visits, 4 API requests, Time elapsed: 1.801s [3/10 done]
[2012-11-20 23:05:07] [50d2df27] Archived website id = 7, period = day, Time elapsed: 0.815s
[2012-11-20 23:05:08] [50d2df27] Archived website id = 7, period = week, 3 visits, Time elapsed: 0.962s
[2012-11-20 23:05:11] [50d2df27] Archived website id = 7, period = month, 16 visits, Time elapsed: 2.811s
[2012-11-20 23:05:16] [50d2df27] Archived website id = 7, period = year, 703 visits, Time elapsed: 4.778s
[2012-11-20 23:05:16] [50d2df27] Archived website id = 7, today = 0 visits, 4 API requests, Time elapsed: 9.369s [4/10 done]
[2012-11-20 23:05:16] [50d2df27] Archived website id = 8, period = day, Time elapsed: 0.350s
[2012-11-20 23:05:17] [50d2df27] Archived website id = 8, period = week, 4 visits, Time elapsed: 0.398s
[2012-11-20 23:05:19] [50d2df27] Archived website id = 8, period = month, 15 visits, Time elapsed: 2.621s
[2012-11-20 23:05:23] [50d2df27] Archived website id = 8, period = year, 165 visits, Time elapsed: 3.648s
[2012-11-20 23:05:23] [50d2df27] Archived website id = 8, today = 0 visits, 4 API requests, Time elapsed: 7.020s [5/10 done]
[2012-11-20 23:05:23] [50d2df27] Archived website id = 9, period = day, Time elapsed: 0.404s
[2012-11-20 23:05:24] [50d2df27] Archived website id = 9, period = week, 1 visits, Time elapsed: 0.399s
[2012-11-20 23:05:26] [50d2df27] Archived website id = 9, period = month, 3 visits, Time elapsed: 1.901s
[2012-11-20 23:05:29] [50d2df27] Archived website id = 9, period = year, 824 visits, Time elapsed: 3.254s
[2012-11-20 23:05:29] [50d2df27] Archived website id = 9, today = 0 visits, 4 API requests, Time elapsed: 5.960s [6/10 done]
[2012-11-20 23:05:29] [50d2df27] Archived website id = 11, period = day, Time elapsed: 0.376s
[2012-11-20 23:05:30] [50d2df27] Archived website id = 11, period = week, 0 visits, Time elapsed: 0.398s
[2012-11-20 23:05:31] [50d2df27] Archived website id = 11, period = month, 6 visits, Time elapsed: 1.425s
[2012-11-20 23:05:33] [50d2df27] Archived website id = 11, period = year, 50 visits, Time elapsed: 1.550s
[2012-11-20 23:05:33] [50d2df27] Archived website id = 11, today = 0 visits, 4 API requests, Time elapsed: 3.751s [7/10 done]
[2012-11-20 23:05:33] [50d2df27] Archived website id = 15, period = day, Time elapsed: 0.205s
[2012-11-20 23:05:33] [50d2df27] Archived website id = 15, period = week, 0 visits, Time elapsed: 0.219s
[2012-11-20 23:05:33] [50d2df27] Archived website id = 15, period = month, 0 visits, Time elapsed: 0.364s
[2012-11-20 23:05:34] [50d2df27] Archived website id = 15, period = year, 0 visits, Time elapsed: 0.342s
[2012-11-20 23:05:34] [50d2df27] Archived website id = 15, today = 0 visits, 4 API requests, Time elapsed: 1.132s [8/10 done]
[2012-11-20 23:05:34] [50d2df27] Archived website id = 17, period = day, Time elapsed: 0.204s
[2012-11-20 23:05:34] [50d2df27] Archived website id = 17, period = week, 0 visits, Time elapsed: 0.219s
[2012-11-20 23:05:34] [50d2df27] Archived website id = 17, period = month, 0 visits, Time elapsed: 0.375s
[2012-11-20 23:05:35] [50d2df27] Archived website id = 17, period = year, 2 visits, Time elapsed: 0.788s
[2012-11-20 23:05:35] [50d2df27] Archived website id = 17, today = 0 visits, 4 API requests, Time elapsed: 1.588s [9/10 done]
[2012-11-20 23:05:35] [50d2df27] Archived website id = 18, period = day, Time elapsed: 0.194s
[2012-11-20 23:05:36] [50d2df27] Archived website id = 18, period = week, 0 visits, Time elapsed: 0.215s
[2012-11-20 23:05:36] [50d2df27] Archived website id = 18, period = month, 1 visits, Time elapsed: 0.375s
[2012-11-20 23:05:37] [50d2df27] Archived website id = 18, period = year, 5 visits, Time elapsed: 0.894s
[2012-11-20 23:05:37] [50d2df27] Archived website id = 18, today = 0 visits, 4 API requests, Time elapsed: 1.680s [10/10 done]
[2012-11-20 23:05:37] [50d2df27] Done archiving!
[2012-11-20 23:05:37] [50d2df27] ---------------------------
[2012-11-20 23:05:37] [50d2df27] SUMMARY
[2012-11-20 23:05:37] [50d2df27] Total daily visits archived: 0
[2012-11-20 23:05:37] [50d2df27] Archived today's reports for 9 websites
[2012-11-20 23:05:37] [50d2df27] Archived week/month/year for 9 websites. 
[2012-11-20 23:05:37] [50d2df27] Skipped 1 websites: no new visit since the last script execution
[2012-11-20 23:05:37] [50d2df27] Skipped 1 websites day archiving: existing daily reports are less than 3600 seconds old
[2012-11-20 23:05:37] [50d2df27] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
[2012-11-20 23:05:37] [50d2df27] Total API requests: 36
[2012-11-20 23:05:37] [50d2df27] done: 9/10 90%, 0 v, 9 wtoday, 9 wperiods, 36 req, 34423 ms, no error
[2012-11-20 23:05:37] [50d2df27] Time elapsed: 34.423s
[2012-11-20 23:05:37] [50d2df27] ---------------------------
[2012-11-20 23:05:37] [50d2df27] SCHEDULED TASKS
[2012-11-20 23:05:37] [50d2df27] Starting Scheduled tasks... 
[2012-11-20 23:05:37] [50d2df27]  No task to run
[2012-11-20 23:05:37] [50d2df27] done

@mattab
Copy link
Member

mattab commented Dec 13, 2012

I think it might be the scheduled task Piwik_CoreAdminHome.purgeOutdatedArchives that would purge too much data, but I'm not sure. I haven't yet been able to reproduce the issue myself. Keep posting more info if you have!

@jloh
Copy link
Author

jloh commented Dec 13, 2012

Thats weird, because I don't have purge old log data on. So it shouldn't actually be purging anything...

@jloh
Copy link
Author

jloh commented Jan 14, 2013

I think I may have found what could be causing the problem. I tracked a few proxies using Piwik that generated a whole new URL each time they visited a different page so I was literally tracking thousands of different pages a day.

I've since fixed this issue but my stats are still reverting, is there a way I can go through a change all the proxies's pages that start in browse.php to JUST browse.php so that there is alot less pages in Piwik?

@mattab
Copy link
Member

mattab commented Jan 18, 2013

you could write a SQL query that will look for all pages in log_action containing browse.php? and delete these IDs from log_link_visit_action (delete these page views). sorry wont write SQL & test, but if you need help: http://piwik.org/consulting/

@jloh
Copy link
Author

jloh commented Jan 20, 2013

Sorry, but this issue appears to still be present.

I've noticed the issue happens when it does its daily tasks which are as follows:

[2013-01-20 01:00:09] [7bfe734c] task,output Piwik_CoreAdminHome.purgeOutdatedArchives,Time elapsed: 3.076s Piwik_PDFReports.dailySchedule,Time elapsed: 2.372s Piwik_PrivacyManager.deleteReportData,Time elapsed: 0.002s Piwik_PrivacyManager.deleteLogData,Time elapsed: 0.000s Piwik_CoreAdminHome.optimizeArchiveTable,Time elapsed: 1.671s

Could it be that it doesn't have sufficient MySQL privileges or there is something wrong with my MySQL installation? I'm running it on a cPanel server so it could be that cPanel is locking it down to much.

Whats also interesting is that I've got Piwik set to delete nothing, I retain all data, but it still purges outdated archives? I assume this just means that it updates them or does it delete them?

@mattab
Copy link
Member

mattab commented Apr 4, 2013

purgeOutdatedArchives should only remove the "partial" reports which were processed for "unfinished days". do you still experience the issue using Piwik 1.12-beta5 here: http://builds.piwik.org/piwik-1.12-b5.zip

If so please re-open and thanks for your patience

@jloh jloh added this to the 1.12 - The Great 1.x Backlog milestone Jul 8, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc. worksforme The issue cannot be reproduced and things work as intended.
Projects
None yet
Development

No branches or pull requests

3 participants