Opened 4 years ago

Closed 3 years ago

#1460 closed Bug (fixed)

Workaround broken mod_security rules causing Piwik to not track visits

Reported by: halfdan Owned by: vipsoft
Priority: low Milestone: Piwik 1.2
Component: Core Keywords:
Cc: Sensitive: no

Description

Ticket for http://forum.piwik.org/index.php?showtopic=11321

I haven't had the time to investigate this any deeper, but the number of people reporting that Piwik is not tracking any visits is increasing and we should take a look at it.

Quote from mutor:
I am suspecting that the problem is that the tracking URL structure (query string after piwik.php) has changed slightly with the new version (0.6.3) and many hosts don't allow to have an URL as query string variable because of security reasons (injecting malicious code from external websites).

Change History (21)

comment:1 Changed 4 years ago by vipsoft (robocoder)

One of the posts referenced HostGator ... sure enough, I verified they have mod_security and are filtering any URL that contains "http:/" or "https:/" (and escaped variants; note: one forward slash) Reiterating/paraphrasing #564, mod_security users don't do the due diligence to inspect the rules before installing them -- some rules are absolute sh*t. The following return a 403 Forbidden om HostGator:

Vote wontfix.

comment:2 Changed 4 years ago by matt (mattab)

I think we did something earlier, strip http:// or https:// (maybe mailto: as well?) in front of the URLs, and put them back in the tracker.

This could easily be done in trunk, probably would have to wait for a next release coming in maybe 2 weeks?

comment:3 Changed 4 years ago by vipsoft (robocoder)

  • Summary changed from Piwik not tracking visits on some hosts to Workaround broken mod_security rules causing Piwik to not track visits

We used to strip the scheme/protocol on the server-side.

If we simply strip the protocol on the client-side, we'd lose outlinks that begin with ftp:// (also blocked by HostGator), svn://, mailto:, etc.

To reassemble this on the server implies the workaround would be to separate the schema from the rest of the URL, e.g.,

  • http;example.com/piwik.php?url=site1.com%2fpath&proto=http&urlref=site2.com%path&protoref=http

Note: this should be affecting the installer (first website setup) and the sitemanager -- as it would explain http://forum.piwik.org/index.php?showtopic=11291

comment:4 Changed 4 years ago by vipsoft (robocoder)

HostGator and users in the forum confirm whitelisting the mod_security rule as the workaround.

comment:5 Changed 4 years ago by vipsoft (robocoder)

At the moment, the HG mod_security rule (purportedly as a security measure against "XSS accounts") affects only $_GET parameters.

comment:6 Changed 4 years ago by matt (mattab)

  • Milestone changed from 0 - Piwik 0.6.4 to Features requests - after Piwik 1.0

maybe it is possible to white list mod_security in .htaccess?

Otherwise, is it wontfix? if we update piwik.js to not pass http:// , we also have to update the PiwikTracker client.

comment:7 Changed 4 years ago by vipsoft (robocoder)

I proposed that we would still send the protocol (ftp or http) but as separate parameters. The parameters (containing the protocol) can be optional. If specified, the server concatenates it with the url. This allow for backward compatibility and cases where the browser has cached piwik.js.

Newer versions of mod_security (since 2008?) can no longer be disabled via .htaccess (eg "SecFilterEngine off" causes error code 500 Internal Server Error).

comment:8 Changed 4 years ago by matt (mattab)

  • Priority changed from critical to low

comment:9 Changed 4 years ago by vipsoft (robocoder)

I'm leaning towards making this an installation systemcheck since the mod_security rules broadly impact Piwik, eg tracker, REST API functions that accept URLs, customData that might contain URLs, etc.

comment:10 Changed 4 years ago by vipsoft (robocoder)

  • Milestone changed from Features requests 1.x or 2.x to 1.1 - Piwik 1.1
  • Owner set to vipsoft

comment:11 Changed 4 years ago by vipsoft (robocoder)

The installation system check will test an actual URL to see if it fails. If so, it's an error that can't be skipped.

(An alternative would be to query a remote server to get this server's IP address, then reverse whois to see if the IP range is assigned to HostGator or The Planet. One disadvantage of this approach is that it is host provider specific...not that they should be singled out, but they are the leading source for support requests.)

These workarounds are ugly, but do-able.

  • piwik.js: strip http:// and https:// from url, urlref, link, download, and redirect parameters; server-side: prepend urls with http:// if protocol is missing (to maintain backwards compatibility); make a similar change to PiwikTracker for consistency; similar changes to SitesManager (API.php and SitesManager.js) required
    • doesn't help with customdata that might contain user-specified URLs
  • piwik.js: add option to POST the request to the server; add a similar option to PiwikTracker; in both cases, the default is GET; note: subject to same-origin policy
    • some headers may be suppressed, so Cookie and UserAgent must be sent in the payload
    • some browsers provide limited information, e.g., no version number in Safari's user agent; Accept-Language not accessible via JavaScript -- might be able to fake it with navigator.language, navigator.browserLanguage, navigator.systemLanguage, and/or navigator.userLanguage

References:

comment:12 Changed 4 years ago by vipsoft (robocoder)

(In [3231]) refs #1460 - add .setRequestMethod("POST") to piwik.js

Use cases:

  • workaround GET request length limit
  • workaround mod_security or .htaccess rules that block request containing "ht$

Notes:

  • subject to same origin policy

comment:13 Changed 3 years ago by vipsoft (robocoder)

  • Resolution set to fixed
  • Status changed from new to closed

(In [3301]) fixes #1460 - SitesManager - add/update site uses POST

comment:14 Changed 3 years ago by matt (mattab)

  • Resolution fixed deleted
  • Status changed from closed to reopened

Thanks vipsoft, great work on this one. Reopening to remember to document the procedure in a FAQ answer, and also on http://piwik.org/docs/javascript-tracking/

comment:15 Changed 3 years ago by vipsoft (robocoder)

I already documented .setRequestMethod().

I just updated the FAQ http://piwik.org/faq/troubleshooting/#faq_58.

comment:16 Changed 3 years ago by vipsoft (robocoder)

  • Resolution set to fixed
  • Status changed from reopened to closed

(In [3304]) fixes #1460 - add Installation check for the tracker (GET piwik.php?url=http://example.com); fix the tracker proxy when using POST

comment:17 Changed 3 years ago by vipsoft (robocoder)

(In [3305]) refs #1460 - comment out checkForErrors on Installation welcome screen as it gets a false positive from the inline JavaScript source

comment:18 Changed 3 years ago by vipsoft (robocoder)

(In [3306]) refs #1460 - add retry loop and pause for tracking test to complete

comment:19 Changed 3 years ago by vipsoft (robocoder)

(In [3307]) refs #1460 - typo

comment:20 Changed 3 years ago by matt (mattab)

  • Resolution fixed deleted
  • Status changed from closed to reopened

I'll update the warning message to a new FAQ answer about mod_security support

  • disable mod_security
  • or set POST method

comment:21 Changed 3 years ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from reopened to closed

Anthon's message is fine

Note: See TracTickets for help on using tickets.