Opened 3 years ago

Closed 17 months ago

Last modified 16 months ago

#2375 closed New feature (fixed)

Exclude Visitors matching User Agent across all websites

Reported by: noreiller Owned by: capedfuzz
Priority: major Milestone: 1.10 - Piwik 1.10
Component: Core Keywords:
Cc: Sensitive: no

Description

Hi, I use Yottaa to monitor my websites, but their bots appear in the statistics.
While they don't provide a list of ip's ("because our bots come from all over the world and there's no set block of addresses you could use" ), they suggest to exclude the bits from their user-agent "YottaaMonitor".
It would be nice to have a feature to exclude those bots.

Thanks,
AM

Attachments (1)

2375.diff.tar.gz (7.0 KB) - added by capedfuzz 17 months ago.
Patch for this issue.

Download all attachments as: .zip

Change History (17)

comment:1 Changed 3 years ago by vipsoft (robocoder)

I've quick-fixed the Yottaa filter in #2376.

comment:2 Changed 3 years ago by noreiller

So I close this ticket, thanks !

comment:3 Changed 3 years ago by vipsoft (robocoder)

  • Keywords user-agent exclusion removed

Well, I'll keep this ticket open. I still like the idea of having this as a general feature since the image tracker has more exposure to bot traffic.

comment:4 Changed 3 years ago by rafal

I politely request this feature, too, as any web site using a monitoring systems (WatchMouse, Pingdom) that emulate a real browser can have their statistics skewed. Those systems tend to hit the site once every minute, from different IPs and different parts of the world.

It would be great if there was an option/button on the Visitor: Log, Providers, and Settings pages to exclude something based on the visitor's or providers IP/host, and, most importantly, the User Agent.

Many thanks,
RL

comment:5 Changed 3 years ago by matt (mattab)

  • Milestone changed from Feature requests to 1.6 Piwik 1.6
  • Let's add a new field in Websites > General settings to exclude a visit based on user agent, across all websites.
  • Note: cache the user agents to exclude in the cache/tracker/ file via the hook Common.fetchWebsiteAttributes
  • Visit should be excluded as long as it "contains" the user agent (not exact match)

comment:6 Changed 3 years ago by matt (mattab)

  • Milestone changed from 1.6 Piwik 1.6 to 1.x - Piwik 1.x

comment:7 Changed 2 years ago by shought

I would like to add my name to the list here, this would be a really useful feature as currently I can't really enable monitoring tools on our websites.

Obviously excluding the IPs by hand is possible, but not really desirable as the IPs are subject to change when servers are added/removed/moved.

comment:8 Changed 21 months ago by matt (mattab)

  • Milestone changed from 1.x - Piwik 1.x to 1.8.x - Piwik 1.8.x
  • Owner set to capedfuzz
  • Summary changed from Exclude visitors from their user agent to Exclude Visitors matching User Agent across all websites

comment:9 Changed 18 months ago by rowdy

I would like to add my vote to this as well.
We're also using a monitoring service with a certain User Agent. Since monitoring is important to know if the site is running and statistics (Piwik) are important to know who's visiting, it seems to me this is an important request.
Without this option, the statistics are completely useless when also using a monitoring service (which has many IP address which also change regularly).

Thanks.

comment:10 Changed 18 months ago by matt (mattab)

  • Milestone changed from 1.9.x - Piwik 1.9.x to 1.9.2 - Piwik 1.9.2

Let's work on this feature which will be an important improvement :)

See spec in: http://dev.piwik.org/trac/ticket/2375#comment:5

Thanks!

comment:11 Changed 18 months ago by rowdy

Thanks, matt! This will be really useful.

comment:12 Changed 18 months ago by rowdy

Hello Matt,

I don't want to sound annoying or anything, but could you please move this ticket to (for example) release 1.10? I see it's not yet included in 1.9.2 and I don't see it in the list for 1.10.

Thanks,
Rowdy

Changed 17 months ago by capedfuzz (diosmosis)

Patch for this issue.

comment:13 Changed 17 months ago by capedfuzz (diosmosis)

Hey matt, I attached a patch that fixes this issue. Let me know if it's good to commit.

comment:14 Changed 17 months ago by matt (mattab)

@capedfuzz Very nice!

Code review:

  • NoVisitTest.php should not change position of the updateSite setting the excluded IP or the test is for user agent exclude is not executed
  • we are already at b7 so please set to b8 or b9
  • setGlobalExcludedUserAgents() should require super user: otherwise any admin user in a shared piwik server, could set to exclude "mozilla,msie,chrome,opera" and cause some trouble
  • getExcludedUserAgentsGlobal() should require at least some admin access.. but even
  • UX KISS: 99% of users will never specify a "per website user agent". In fact it is not in the specification for this ticket ;) I thought that only "global user agent exclude" would be enough... but since you've built it, and that I can think of some use cases where this is useful, then
    • However since most people wil never use the feature, we should hide the column on the webvsite listing by default. Can you please, add a checkbox in the "super user User Agents" section, that reads "Enable custom User Agents to exclude for each website" (or similar). the box is unchecked by default (and the column "User agents to exclude" hidden from the table). When users click and press save, the page reloads and the columns is displayed.
    • NOTE: when the preference is set from "Enable custom agents for each site" from checked to unchecked, then all excluded_user_agents columns should be deleted to later avoid hard to find tracking bugs.
    • implementation; to keep code simple, I would recommend to simply hide the column using jquery, this way the code is unchanged appart from getting/setting this new user preference?

comment:15 Changed 17 months ago by capedfuzz (diosmosis)

  • Resolution set to fixed
  • Status changed from new to closed

(In [7535]) Fixes #2375, added ability to exclude visits from tracking based on global & site-specific blacklists user agent substrings. Also fixed bug in website management page that broke website editing/creation.

comment:16 Changed 16 months ago by matt (mattab)

  • Priority changed from normal to major
Note: See TracTickets for help on using tickets.