Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#1097 closed New feature (fixed)

Installer to check that all files are uploaded correctly during install / upgrades

Reported by: vipsoft Owned by: matt
Priority: normal Milestone: Piwik 0.5.5
Component: Core Keywords:
Cc: Sensitive: no

Description

Support issue: users who upload (eg recursive ftp put) an already extracted Piwik folder. Files missing or corrupt.

Enhancement:

  • build script: generate file manifest with filesizes (and optional checksum)
  • installation/upgrade: iterate thru manifest and verify integrity of installation

Attachments (2)

testintegrity.php (431 bytes) - added by vipsoft 4 years ago.
Standalone integrity test script
manifest.inc.php (151.8 KB) - added by vipsoft 4 years ago.
Sample (generated) manifest.inc.php (using piwik 0.5.4)

Download all attachments as: .zip

Change History (24)

comment:1 Changed 4 years ago by matt (mattab)

  • Milestone changed from 1 - Piwik 0.5.5 to Features requests - after Piwik 1.0

Marking it as feature request. This is an interesting improvement (I've seen it in Gallery installer before), but can take many seconds to execute, and adds a bit of overhead on the release process. If someone wants to grab the task and build it, please let us know.

comment:2 Changed 4 years ago by matt (mattab)

  • Summary changed from manifest to Installer to check that all files are uploaded correctly during install / upgrades

comment:3 Changed 4 years ago by matt (mattab)

  • Type changed from Bug to New feature

comment:4 Changed 4 years ago by vipsoft (robocoder)

Change to build script:

  cp piwik/misc/How\ to\ install\ Piwik.html .
  cp piwik/misc/WebAppGallery/*.xml .

+ find . -printf '%s ' -exec md5sum {} \; >config/manifest
+ 
  zip -r piwik-$VERSION.zip piwik How\ to\ install\ Piwik.html  > /dev/null 2> /dev/null

comment:6 Changed 4 years ago by vipsoft (robocoder)

Revised manifest generator for build script:

find piwik -type f -printf '%s ' -exec md5sum {} \; | fgrep -v 'manifest.inc.php' | sed '1,$ s/\([0-9]*\) \([a-z0-9]*\) *piwik\/\(.*\)/\t\t"\3" => array("\1", "\2"),/; 1 s/^/<?php\nclass Manifest {\n\tstatic $files=array(\n/; $ s/$/\n\t);\n}/' > piwik/config/manifest.inc.php

Timings from an (ancient) Athlon 1.4GHz test box:

  • Generating the manifest.inc.php took 11.117 seconds on a clean (svn export) folder.
  • Loading manifest.inc.php, iterating through the files, and running file_exists/file_size/md5_file took 0.541 seconds.

comment:7 Changed 4 years ago by matt (mattab)

For the UI part, I suggest that if the test fails, a red icon appears in the system checklist page. If it passes, it is just one more green item in the list (with required functions and librairies).

comment:8 Changed 4 years ago by vipsoft (robocoder)

(In [1756]) refs #1097 - file integrity check (optional) added to Installation. A warning is given if manifest.inc.php or md5_file() don't exist.

Matt: please update the build script with the 'find' in comment:6 and then mark this ticket as fixed.

Changed 4 years ago by vipsoft (robocoder)

Standalone integrity test script

Changed 4 years ago by vipsoft (robocoder)

Sample (generated) manifest.inc.php (using piwik 0.5.4)

comment:9 Changed 4 years ago by vipsoft (robocoder)

(In [1757]) refs #1097 - exclude development environments; translate exception messages

more php docs

comment:10 Changed 4 years ago by vipsoft (robocoder)

(In [1758]) refs #1097 - reorganize code to better accommodate an integrity check after an upgrade (not yet implemented in CoreUpdater)

comment:11 Changed 4 years ago by matt (mattab)

before marking as fixed, integrity check must happen during upgrades too. I will update the release script once this is code complete, nice work on this, this was a very frequent support issue.

comment:12 Changed 4 years ago by vipsoft (robocoder)

  • Milestone changed from Features requests - after Piwik 1.0 to 1 - Piwik 0.5.5
  • Owner set to matt

comment:13 Changed 4 years ago by vipsoft (robocoder)

(In [1764]) refs #1097 - check file integrity after updates; simplify some messages (e.g., md5 sum)

comment:14 Changed 4 years ago by vipsoft (robocoder)

(In [1765]) refs #1097 - refactoring

comment:15 Changed 4 years ago by TulipVorlax

I think this will be a really great feature.
Thanks a real lot !

comment:16 Changed 4 years ago by matt (mattab)

Anthon, is the piwik code ready for this new feature? is the only thing left the one line update to the build script?

comment:17 Changed 4 years ago by vipsoft (robocoder)

Matt: yes, it's ready.

While you're at it, please also add a line to the build script to remove unit test directories under the plugins folder.

  rm -rf piwik/tests/*
+ find piwik/plugins -name tests -type d -exec rm -rf {} \;
  mv README.txt piwik/tests/

comment:18 Changed 4 years ago by matt (mattab)

Anthon,

I added the find to delete the tests plugins folders and I get:

find: piwik/plugins/UsersManager/tests: No such file or directory
find: piwik/plugins/UserSettings/tests: No such file or directory
find: piwik/plugins/Actions/tests: No such file or directory
find: piwik/plugins/LanguagesManager/tests: No such file or directory
find: piwik/plugins/UserCountry/tests: No such file or directory
find: piwik/plugins/AnonymizeIP/tests: No such file or directory
find: piwik/plugins/SitesManager/tests: No such file or directory

is that expected?

comment:19 Changed 4 years ago by matt (mattab)

good stuff Anthon! closing

comment:20 Changed 4 years ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from new to closed

comment:21 Changed 4 years ago by vipsoft (robocoder)

Yes, expected because 'find' is diving through folders while we remove it. We can suppress the messages using redirection, e.g, 2> /dev/null, or use xargs (if you don't mind the extra process being spawned), e.g.,

find piwik/plugins -name tests -type d -exec rm -rf {} \; 2>/dev/null

find piwik/plugins -name tests -type d -print0 | xargs -0 /bin/rm -rf

comment:22 Changed 4 years ago by matt (mattab)

(In [1944]) Refs #1097
adding explanation when failure occurs, inviting user to reupload the files

Note: See TracTickets for help on using tickets.