Opened 16 months ago

Closed 11 months ago

Last modified 2 months ago

#3612 closed New feature (fixed)

Better Device detection (mobile, TVs, consoles, etc.): New "Devices" report?

Reported by: matt Owned by: mattab
Priority: major Milestone: 2.x - The Great Piwik 2.x Backlog
Component: Core Keywords:
Cc: mauser Sensitive: no

Description (last modified by matt)

Piwik should provide a more clear set of reports regarding Devices, Operating systems, browsers, significantly enhancing the analytics insights on the visitors browsing experience.

This ticket is about adding a couple new report and fixing related issues at the same time. Proposal:

  • Add New Device type report comparing generic families of devices: Computer, Mobile, Tablet, TV, Console, Portable console.
    • It would be displayed between "resolutions" and "mobile vs dekstop" report.
    • This would be a superset of the existing reports "Operating systems" which does already a simple "device" aggregation (especially reports mobile devices separately since they run a different OS by design). I propose to add a "related report" link that will not load the report in ajax, but will instead link to #devices hash tag in the page below to load the 'Device type report'
  • Add new Manufacturer" report that would grep the user agent for known brands: Samsung, Apple, HTC, Motorola, Amazon, etc.
    • we need to find the full list somewhere
    • and their favicons.
  • Add new Manufacturer+ Model report reporting eg. Nokia N80
  • Modify Browser version report to segment by Opera Mobile VS Opera #2807
  • Modify Operating systems report
    • to segment Wii VS WiiU #3599
    • to segment Chrome OS VS other Linux (follow up of #3213)
    • detect Ubuntu VS other Linux

Also see a feature request: Detecting TVs #3505

While we work on this, please let's try to implement this as well: #3157 Add a new visitor segment attribute "isMobile" (including tablet)
Open for discussion...

Change History (20)

comment:1 Changed 16 months ago by matt (mattab)

  • Description modified (diff)

comment:2 Changed 16 months ago by matt (mattab)

Note: this will require some changes in the Schema of log_visit and log_conversion so this is definitely not trivial change. Before starting work on this one please get in touch to discuss specs.

comment:3 Changed 16 months ago by Lesjokolat2

I would like to recommend a separate tab between Referers and Goals called "Mobile". The mobile universe is such a critical part of digital online strategies that it should be separate in this way. Most companies like to segment their mobile data so capturing it and presenting this way will feel very natural. Once there a sub set of tabs would then list the Following in Order of importance:

Device (Specific Device name, (Note would be great to perform regex like filtering to allow grouping of devices by families)
Device Type (generic device family type as noted earlier(Computer, Mobile, Tablet, TV, Console, Portable console), excellent for marketing analysis)
Manufacturer
Operating System
Browser Version
Screen Size (HxW)
Video Support (Various types of video formats if available)
Net Protocal

Standard Metrics PVs UVs and Geo Location (using lat and long)

The report "Mobile VS Desktop" Devices, Device type and manufacturer could be the initial Widgets under the New Mobile tab. If only one report is allowed then use Mobile VS Desktop.

Sidenote: Aps a component of the Mobile universe should really be treated as their own site from the master site pop up. Companies like to compare and/or total their desktop sites to mobile apps so it would be great if the all sites report allowed the basic PVs and UVs to display together.

comment:4 Changed 16 months ago by old_bill

One thing that could help to speed up the range of devices/OS/browsers that Piwik can recognize as specific ones, is some kind of "User Agent Logger" that could be turned on/off in the configuration tab of every Piwik installation.

It would save all the User Agent Data from unknown User Agents as a list plus an optional text field where the Administrator could add some notes per User Agent (e.g. a suggestion of the kind of device this agent should berecognized).

As a manual task, the Admin could "upload" this list from time to time to some kind of master list here, which would be used continuously to maintain the existing device/OS/Browser database. The upload itself could be done automatically by pressing a button, but the upload task should not be automated to avoid giving Piwik an image of automatically uploading user data somewhere.

comment:5 Changed 16 months ago by matt (mattab)

@old_bill we won't do anything like that, but instead will use a library that mostly works and that we can update later on when new devices are released. Updating the list will happen when you update Piwik ~ every month or two.

If you're interested in this ticket and can sponsor please get in touch NOW with us at: http://piwik.org/consulting/#contact-consultant

comment:6 Changed 16 months ago by mahdi

It would be great if you could adjust current browser/OS reports to look like those in this plugin (I know it's flash based, so it can't be just re-used)

http://dev.piwik.org/trac/ticket/1558
http://dev.piwik.org/trac/attachment/ticket/1558/screenshot.png

I find this much more informational than current pie charts.

comment:7 Changed 15 months ago by matt (mattab)

  • Description modified (diff)

comment:8 Changed 15 months ago by matt (mattab)

  • Description modified (diff)

comment:9 Changed 13 months ago by matt (mattab)

  • Milestone changed from 1.x - Piwik 1.x to 1.12 - Piwik 1.12
  • Priority changed from normal to major

comment:10 Changed 13 months ago by matt (mattab)

  • Cc mauser added

comment:11 Changed 13 months ago by matt (mattab)

  • Milestone changed from 1.12 - The Great 1.x Backlog to 2.x - Piwik 2.x

comment:12 Changed 12 months ago by saxon

Hi, does this mean there's no easy way to work out what % of our traffic is mobile/tablet?

We're considering creating more responsive designs and we're trying to justify the outlay.

comment:13 Changed 11 months ago by matt (mattab)

In 83904c1de7bb5b6efe20bbb73f56e67f2aeb291a:

Adding user agent to the newVisitorInformation tracker hook
refs #3612

comment:14 Changed 11 months ago by matt (mattab)

In 7a3e6f4bb793c387b5bf6b599e1bd3b968163024:

Refs #3612 Adding new icons for browsers/os detected by upcoming advanced device detection

comment:15 Changed 11 months ago by matt (mattab)

  • Owner set to mattab
  • Resolution set to fixed
  • Status changed from new to closed

In 6804733ddcb9fcf72b2df3f796a0d5b228bb0ab2:

Adding new DevicesDetection plugin, developed by Clearcode and sponsored by a client of Piwik Professional Services. A beautiful work of engineering, all released under GNU/GPL license!

Fixes #3612

  • when enabled, the plugin will add a new submenu: Visitors> Devices
  • the new devices report contains NEW reports:
    • Much improved Device type (tracking 'car browser', 'console', 'desktop', 'feature phone', 'smartphone', 'tablet', 'tv')
    • Device brand (tracking more than 100 brands such as Nokia, Nintendo, Lenovo or Apple.
    • Device model (tracking hundreds of phone/console models)
    • Operating System versions (tracking 70 variations of operating systems including Ubuntu vs Kubuntu vs Debian vs Lubuntu vs Xubuntu)
    • Operating System families (Android vs Google TV vs Windows vs Windows mobile vs Mobile gaming consoles)
    • Browser versions
    • Browser families

Refs #3505 There is some basic TV detection included and maybe you can help contribute better detections (see the .yml data files containing the regular expressions in YAML format)

HOW DOES IT WORK

This is quite beautiful system. It is a plugin that disabled by default. when enabled, it will create additional columns in the DB. Also at tracking, it will look at the user agent, and try to match it against one browser we know in the databases. The databases of user agent matching are composed by 3 YML files, parsed by spyc.php into php array.

These 3 YML took dozens of hours of work and testing with dozens of mobile phones and devices for accuracy. We are happy with the result as they should cover > 80% of the devices commonly used. We hope the community will help us build up these YML files and make them better, so we can track accurately 90% or 95% of requests.

The performance overhead is pretty small, but parsing the YML files + running dozens of regex will add some overhead. This is why it is still disabled by default. We will think about how to integrate it in core, in the next few months.

Please let me know if you find any problem with this new awesome code!

comment:16 Changed 11 months ago by matt (mattab)

In 4f4d377876adcf698c00ca2141f9cff26bf75a83:

Refs #3612

  • making some tests pass == fixing bugs

comment:17 Changed 11 months ago by matt (mattab)

In 58c88c1b2980ee1c02a47cb3d35de1480402e6ac:

Refs #3612

  • making tests pass!
  • enabling DevicesDetection tests, even though DevicesDetection plugin is disabled by default.

comment:18 Changed 11 months ago by matt (mattab)

In 771239a7dafff862f25be8abf1d4d68bacd91f03:

Adding Chrome mobile icon (ideally we need a unit test to check we have icons for all browsers/OS referenced in the UserAgentParserEnhanced)
Refs #3612

comment:19 Changed 11 months ago by matt (mattab)

See also: #3961 New segment: deviceType to let you segment visitors by: desktop, smartphone, tablet, feature phone, console, tv, car browser

comment:20 Changed 2 months ago by matt (mattab)

See follow up ticket: #3962

Note: See TracTickets for help on using tickets.