To me this is a major issue, as non-javascript users still give us valuable information. I would love to see this implemented before 1.0

Suggestion 1

Couldn't the http:BL by Project Honeypot be used to filter out any bots? They offer an API to identify Search Engines, Spammers and other bots by IP address.

Piwik could work like this:

• Javascript enabled:
  • count users as usual
• Javascript disabled:
  • Discard all users that are known search engine bots (by User-Agent)
  • Check IP of all remaining users against http:BL, discard if known bot, count otherwise.

This way traffic for the blacklist server would be kept low. I still think every Piwik installation would need their own API key, though.

Suggestion 2

Piwik should include its own tiny honeypot. The <noscript> tag should include a link that is invisible to the user and that has rel=nofollow.

<a href="http://domain/piwik/honeypot.php" rel="nofollow">&nbsp;</a>

Only malicious crawlers will follow this link, so Piwik can exclude their IPs from tracking. Known, well-behaving search bots can still be identified by User-Agent. This way, most bots will probably get identified.

Replying to matt:
I wan't this feature too. Not only users are interessting. I want see which bots crawl my site.

Can I add my vote for this as well? We're missing out on visits from many mobile phone users and disabled people using screen readers, for example, because they don't have javascript. And there are legitimate reasons for disabling javascript in a normal browser as well. I agree we need to separate out the bots somehow for the statistics, but really that's a separate issue. I'd like the option of counting all visitors, even if that includes bots.

The code

<a href="http://domain/piwik/honeypot.php" rel="nofollow">&nbsp;</a>

will be visible to blind persons using screen-reader software. It would be better to code this as

<a href="http://domain/piwik/honeypot.php" rel="nofollow" style="display:none;">&nbsp;</a>

which will also hide it from the screen readers.

Hope this helps,
Charles Belov
SFMTA Webmaster
www.sfmta.com/webmaster

Charles: that's not our tracking code. Piwik's tracking code doesn't contain an anchor link (honeypot or otherwise).

re: comment:3 - The idea behind the noscript tag is to track Javascript-disabled visitors. We'll provide a hook here so third-party plugins can implement suggestion 2.

In order to report search engine bot activity, we could reuse some of the GPL code from http://www.crawltrack.net/ which is a php bot tracker tool. The logic could sit in a Piwik plugin. There could be a new sub tab, that would report bot activity for each bot that was seen during the selected date range.

Bots would be identified by user agents and / or IPs, see eg. the list at crawltrack: http://www.crawltrack.net/crawlerlist.php

Additional features could include:

• give ratio of bots VS human activity on the website (what percentage of traffic comes from bots VS humans)
• for a given bot on a given day, list all pages crawled
• list bot crawling frequency in a new column (next to Visits, Page views, etc.). eg. google can crawl one page every 10s, other bots would crawl one page every 1 min, etc.

So i think it would be interesting to track also robots f.e. for big sites.
With this feature you can see how many bots a scraping your site. But it make sense to see Googlebot, Msnbot and maybe Slurp (Yahoo)

But this should track in a seperatet table with a special plugin - like Live Bots ;-)

In my tool http://www.spider-trap.de/en_index.html i ban a lot of bad bots. Maybe Piwik can report the webmaster if an bot is crawling.

The Tracking API has been released, which can help track visitors without Javascript, or even track visits Mobile apps, desktop apps and more.

http://piwik.org/docs/tracking-api/