"File size mismatch" warning with a modified robots.txt

[larsen]

Piwik´s own robots.txt is empty, but as the reporter from ticket #431 mentioned a global disallowed is the desired behaviour in most cases.

Starting with the file integrity check in Piwik 0.5.5 you get a warning when you have changed the robots.txt what I guess several people will have done.

To prevent this, robots.txt should not be checked.

[vipsoft]

Vote to simply remove robots.txt from the distro because:

• it is not checked by crawlers if Piwik is installed in a subfolder of the web document root
• it is overwritten by the auto-update
• it's empty!

[vipsoft]

Part 2:

• enhance installer to create robots.txt if Piwik is installed in the web document root, otherwise warn user and provide sample exclusion

[matt]

We initially put the robots.txt because otherwise, when Piwik is installed in a subdomain, it shows up as 404 in the error logs, which is a nuisance.

Maybe we can

• not overwrite it during automatic upgrades
• not check it in the file integrity check

which would give us the best of both worlds?

[vipsoft]

I'll modify the installer to simply create an empty robots.txt in the piwik root if it doesn't already exist, rather than creating exceptions in the updater and integrity check.

[matt]

I think this will be an issue as it would require write permissions on the root folder, which are currently not required..

[vipsoft]

On review, why not re-open #431 and implement the original recommendation?

[vipsoft]

(In [1957]) fixes #1215 - protect locally modified 'robots.txt'; skip file integrity check and don't overwrite when updating