Opened 3 years ago

Closed 11 months ago

Last modified 11 months ago

#2135 closed New feature (fixed)

Segmentation: implement UI for adding/editing segments, and switching segments

Reported by: matt Owned by: matt
Priority: critical Milestone: 1.12 - The Great 1.x Backlog
Component: Core Keywords:
Cc: Sensitive: no

Description (last modified by matt)

Piwik now support full segmentation via the API only: http://piwik.org/docs/analytics-api/segmentation/

We want to be able to

  • Create segments
    • define the conditions AND / OR
      • pick metric
      • pick match (equals, contains, etc.)
      • pick value
    • give the segment a name
    • Segment is created inside the "report" UI like Goals
  • Test Segment in real time (fetch number of visits for it). Just calls the API with segment= and fetches number of visits.
  • Edit segment: UI page resets to current segment definition
  • Build the feature to autocomplete when filling a value for a given dimension/metric, for efficiency and ease of use. Reuse the code in getLabelsFromTable()
  • For metrics, when the value in the filter is not numeric, throw an error

Also, when creating a segment, you can add check 2 boxes

  • Add to my segment list (checked by default - available only to user who creates it)
  • Add to this website segment list (viewable by all & editable by all users with admin on this site)
  • For super user, "Add this to all websites segment list".

Segment definitions are stored in a new 'segment' table, which contains the creator user login, idsites the segment appears in.
In the super user case, the segment will have a flag "idsites" = "all".

  • Archive.php: update to pre-process segments created for a particular website. call API to get per-site segments. Let user disable this with config setting.

In the Reporting UI, there would be a new box probaby similar to current Calendar / Info box that would list the current Segment applied to the data.
It would open up (similarly to the calendar) to show the list of pre-created segments, or the "Add a new segment" link.

User can then click on the Segment link to ajax reload the report with the segment applied to all visible reports.

The list of segment will lists all Websites segment first, then all User segments.

Note: some of the UI logic to define segment can be similar to the one used to define custom alerts (#1486), in particular the "value" ajax suggestion would reuse the same code.

GA supports visitsToTransaction and daysToTransaction - is it equivalent to "visitConverted==1;daysSinceFirstVisit==X" ? maybe not

Summary: the UI will allow to create/edit/delete segments, we will add a new box next to the calendar to select currently applied segments.

BUGS to fix
At the same time, it would be nice:

  • to understand & fix this bug as well: #3144
  • Also fix this: #2981

Later version

In the V1, we will not compare the Segment data with the "All visits" data or allow to compare segments together. This feature would be very nice and is important to have, see also ticket to compare several date ranges #291

Requirements

I will ask Marcin to provide some mockups for the screens of this feature. Once we have them we can implement this new UI element.

Please post here any idea/feedback!

Change History (51)

comment:2 Changed 3 years ago by matt (mattab)

  • Description modified (diff)

comment:3 Changed 3 years ago by msteffeck

I'm wondering what that status of this feature is. It looks like you folks have been recently active on it.

The reason I ask is because I have need of segmentation in a 3rd party plugin I'm using/developing. I was just going to hack something together, but if you are far along I'd rather do it correctly.

comment:4 Changed 3 years ago by matt (mattab)

No work has been done on the "UI to create Segment picker" but the backend of analytics segmentation works

See also: http://help.yandex.com/metrika/?id=1122004

comment:5 Changed 3 years ago by msteffeck

I noticed in the Archive.php docs that you can set a segment. Is the segment archiving functional? It would be great to be able to use that

http://qa.piwik.org:8080/phpdocs/Piwik/Piwik_Archive/Piwik_Archive.html#setSegment

comment:7 Changed 3 years ago by msteffeck

comment:8 Changed 3 years ago by matt (mattab)

As per report in: http://forum.piwik.org/read.php?2,80293

there is a few issues when a segment is passed to the request with regards to graphs.

comment:9 Changed 3 years ago by matt (mattab)

Once this is implemented, a typical workflow could be:

  • Go to Visitors > Custom Variables report
  • create/select Segment: visits who looked at [this page]
  • Click "apply"
  • Custom variables report now reports only for users who have seen a given page

comment:10 Changed 3 years ago by matt (mattab)

  • Milestone changed from 1.x - Piwik 1.x to 1.7 Piwik 1.7

I really want to work on this feature so increasing priority as personal reminder :)

comment:11 Changed 3 years ago by matt (mattab)

When implementing Segmentation UI, let's think about Visitor Log. A Frequently requested feature is to be able to "filter" the visitor log to specific countries, pages, actions, IPs, hostnames, etc. See #1839

Maybe this UI can be reused also as a real time filter UI for the Visitor Log. The backend is the same code anyway: creating a segment then applied to the Live! APIs.

See also this post

This would be great to implement both features at once!

comment:12 Changed 2 years ago by matt (mattab)

  • add support for Segmentation for Scheduled reports. For example, a new line in the Report editor would allow to generate and email the scheduled report (pdf or html) for a given segment, including all selected reports.
  • Segment creator / edit
    • Sharing / reusing segments
      • "( ) Share this segment with all users." - Only available to super user.
      • "( ) Share this segment with specific users
    • When a segment is applied to the report, the Segment edit would have a checkbox "By default, load Piwik reports filtered to '$NAME' instead of 'All Visits'".
  • because segment name is going to be used a lot in reports, UI, etc. we should let users know what type of name is expected. It would be used also in report footer see #2786
  • The segment selector could display first
    • Segments last used for this website
    • Most used across websites
    • top 10 common pre-selected by Piwik
    • Inline search box

Some ideas for pre-created segments:

  • Customers

A segment that includes only visits from existing customers — even if they do not purchase from your site when they visit. This is why this segment is so valuable — these visitor have bought from you before!

  • Engagers

After customers, Engagers are your next most valuable group of visitors. They have shown engagement, i.e., made an effort, perhaps even given you their contact details. Examples include registrations, contact requests (form submissions), file downloads, ranking products and content, comments and feedback, video views, clicks on social love buttons etc.

  • Mobilers

A segment that includes only visits from mobile devices. Mobile is a very different experience to browsing on a bigger screen. Lots of implication for your web site structure/design if this segment is significant to you.

  • Non-Bouncers

A segment that excludes all bounced visits i.e., removes single page visits that have no engagement. The theory is that these visitors are clearly not interested in your website, so remove them from your analysis.

See cmswire.com/cms/customer-experience/interview-brian-clifton-on-cutting-through-data-noise-and-the-future-of-analytics-017510.php

See also support.google.com/analytics/bin/answer.py?hl=en&answer=1011783

http://searchenginewatch.com/article/2268458/16-Secret-Google-Analytics-Advanced-Segments-Worth-Their-Weight-in-Gold

Last edited 11 months ago by matt (previous) (diff)

comment:13 Changed 2 years ago by matt (mattab)

  • Priority changed from normal to critical

comment:14 Changed 18 months ago by matt (mattab)

As part of #3088 we should allow Scheduled reports to be sent with a specific segment applied. The API should accept either False, a segment ID, or a new segment string specific to this report. Reported in forums

comment:15 Changed 18 months ago by matt (mattab)

As reported in #3507 Visitors Log under Visitors tab - it would be nice if we could filter
by:

IP address
Goal (show only visits that achieved a specific goal)
Referrer URL
Keyword
Campaign (Show only visitors from a specific Campaign)
visited page (URL)
Action String (search the "action" column for specific string, p.e. "*product-name*")

This is the link to the forum request:

http://forum.piwik.org/read.php?3,88704

comment:16 follow-up: Changed 18 months ago by nathanzachary

It would also be nice if the visitor log could be filtered by 'Provider.' At current, I have to grep through my logs for the provider name, and then compare it to the Apache logs to see what pages the visitors viewed.

Cheers,
Nathan Zachary

comment:17 follow-up: Changed 17 months ago by matt (mattab)

We are officially calling for sponsors for funding for this feature! We would love to implement this feature in Piwik. If you can help sponsor even just a few hours of work, we hope to crowdfund and find many more. Contact us now to get started and help make this feature possible!

See also #3095

Last edited 16 months ago by matt (previous) (diff)

comment:18 in reply to: ↑ 17 Changed 17 months ago by lining

Replying to matt:

We are officially calling for sponsors for funding for this feature! We would love to implement this feature in Piwik. If you can help sponsor even just a few hours of work, we hope to crowdfund and find many more. Contact us now to get started and help make this feature possible!

I am very interested in this feature, and also like to participate implementation.

comment:19 Changed 15 months ago by matt (mattab)

  • Description modified (diff)

comment:20 Changed 13 months ago by matt (mattab)

  • Description modified (diff)

comment:21 Changed 13 months ago by matt (mattab)

  • Milestone changed from 1.12.x - Piwik 1.12.x to 1.12 - Piwik 1.12

comment:22 Changed 13 months ago by matt (mattab)

  • Owner set to matt

we reached the crowdfunding goal, awesome!! http://crowdfunding.piwik.org/custom-segments-editor/

Note, to fix at the same time: #3639

  • each segment can be
    • "Use only today" (default)
    • "Saved to use later (and pre-process every night)"

When "Use only today" the segmented data will be deleted after 1 day.

See also:

  • Process segmented reports in parallel #3658
Last edited 13 months ago by matt (previous) (diff)

comment:23 Changed 12 months ago by matt (mattab)

In 7933de115b3fc7e5dc9bc58b7e979cdefb89d16d:

Refs #2135

  • BETA release of Segment editor.
  • API to add/update/delete/get recorded segments
  • cron script will pre-process segments marked as such
  • UI to create, update, apply segments

TODOs @mattab

  • test with NON super user (anon/view/admin)
  • testing (and more testing)-- try to break it (it's not hard...)
  • UI: Saving the segment for reuse / pre-processing
  • Loading message: "This can take a while based on the amount of data that needs to be pre-archived"
  • i18n
  • Save & Apply this segment ==> "Test this segment" which would call VisitsSummary.get for early feedback ?

+ hoping for Piotr superJS Ninjas skills with some jquery/js that I can't figure out

comment:24 Changed 12 months ago by matt (mattab)

In b5067e0f1c106f6d6cae2415eaf67052c96d5ace:

Refs #2135
Reverting some reformat code that went wrong (as Jquery 1.9 expects HTML strings to start with < rather than a space followed by <

comment:25 Changed 12 months ago by matt (mattab)

In 0f3d97694d69df6f06ca1e5194e14e58d6da9b28:

Refs #2135 Few JS fixes for a cleaner UI/UX

comment:26 Changed 12 months ago by matt (mattab)

In 304a03e2a5d542a62c66a54b53d35614a9623f2f:

Refs #2135 forgot to to check these in

comment:27 Changed 12 months ago by matt (mattab)

In 85a30e37744a27941d2d96e51ce9e9d2da5dfd0b:

Refs #2135

  • Adding auto suggest listing
  • Fixing remaining JS bugs

Kuddos Michal & Piotr for your help!

comment:28 Changed 12 months ago by matt (mattab)

In 3212ef63391c874937fb3eb641e925ade4bc37a9:

Refs #2135

  • Adding latest version of mousewheel
  • fixing line endings (why do I always have to do this, somehow it reverts itself...)
  • Fixing css bugs (they required the DOCTYPE to be changed, MAYBE THIS WILL BREAK SOMETHING? I wonder why we haven't changed the DOCTYPE before?)

OMG this is starting to be awesome!

comment:29 Changed 12 months ago by matt (mattab)

In b4c56616539be02b3c29ca0727baee287e04e424:

Showing suggested value on click on the INPUT field Refs #2135

comment:30 follow-ups: Changed 12 months ago by peterb (peterbo)

Here is a little feedback on a short testing period of this fantastic feature:

  • Segments not created by the superuser trigger a login screen in the dashboard widgets, saying that you need at least superuser rights to access this resource (This is listed in your TODO, so I think you're aware of this one)
  • as superuser, when you are viewing a segment and change to another segment, the first selected segment stays applied (even when you switch back to "all visits (default)"). This is not the case when you view a segment in admin or user permission mode.
  • when you edit an existing segment, blanks and other special chanracters are urlencoded (e.g. "%20")
  • The "Delete" a segment function does not seem to work. There seems to be an error while triggering the "are you sure" modal dialog

Features

  • Would it make sense to include the segments selector in the "all websites" overview? A user could be able to apply a segment which was created for "all websites".
  • A warning or a notice that the processing of a new / not yet archived segment can be slow.
  • A user should be able to only create and "save" a segment and not apply it yet to the reports. If you want to create a bunch of segments, "save & apply" could be really slow and disturb the workflow.
  • Later: a feature for "cloning" a segment could be nice. If you have a complex segment and you only want to change a few details, this feature would be quite handy.
Last edited 12 months ago by peterb (previous) (diff)

comment:31 Changed 11 months ago by matt (mattab)

In 028342ad359aa86515c7d404c76b780ab3addcc4:

Refs #2135

  • Fixing some encoding issues and many small bugs
  • When user is anonymous, show message inviting user to login to try the feature

comment:32 Changed 11 months ago by matt (mattab)

In 263892f231da5f6e998342aef02f33aa7edf8622:

Refs #2135

  • NAILED the encoding! wow that was tough for my brain, but finally got around it.

Some findings:
_GET is automatically URL decoded
QUERY_STRING contains raw URL so we'll use this one instead
I REALLY want to simplify the whole "_GET" variables within Piwik so that all code uses a static method to get input variables. Refs #3931

comment:33 Changed 11 months ago by matt (mattab)

In d89a08b8b27ef9a7293e9f8cf351bedbd838e2cb:

Fixes #3932

  • you can now write browserCode==ff;referrerKeyword!= to select all visitors using firefox and that have a keyword set
  • or you can write referrerKeyword==;browserCode==ff to select all visitors using firefox and that did not have any keyword set

Also fixes #3933

Refs #2135

  • fixing last bugs with segment selector encoding (working on chrome + FF + opera) - I 'hope' it will work on iE...

comment:34 in reply to: ↑ 30 Changed 11 months ago by matt (mattab)

Replying to peterb:

Here is a little feedback on a short testing period of this fantastic feature:

  • Segments not created by the superuser trigger a login screen in the dashboard widgets, saying that you need at least superuser rights to access this resource (This is listed in your TODO, so I think you're aware of this one)
  • as superuser, when you are viewing a segment and change to another segment, the first selected segment stays applied (even when you switch back to "all visits (default)"). This is not the case when you view a segment in admin or user permission mode.
  • when you edit an existing segment, blanks and other special chanracters are urlencoded (e.g. "%20")
  • The "Delete" a segment function does not seem to work. There seems to be an error while triggering the "are you sure" modal dialog

Hey Peter, thanks for testing! Could you check again, these should now be fixed!

comment:35 in reply to: ↑ 30 Changed 11 months ago by matt (mattab)

Replying to peterb:

Here is a little feedback on a short testing period of this fantastic feature:

Thanks for suggestions. I will implement the "Warning/notice message" but I moved the rest to a "feature request" ticket: #3934

comment:37 Changed 11 months ago by matt (mattab)

In 25272167844ed0cd2c24121d0600edf1e2f53ebe:

Refs #2135 i18n + improvements (custom loading message when segment applied)

comment:38 Changed 11 months ago by matt (mattab)

In 36fc4ca1db9840fe25bd0f6ac6f8e51ac5bd9a06:

Refs #2135

  • when changing website in the selector, the segment parameter will be reset (to no segment).

comment:39 Changed 11 months ago by matt (mattab)

In 028acc4e42af442c3600d4ebabed70f1ec8c9dcb:

Refs #2135 Enabling plugin by default for all updates

comment:40 Changed 11 months ago by peterb (peterbo)

Hi Matt, great work and very much fun to use so far! - Feedback:

  • setting the visibility of a segment to "All Users" is always reset to "me", same with "this website only" and "all websites"
  • The segment editor should be visible within the PDFReports plugin to have segmented PDF / HTML Reports available (perhaps it would make sense to include the segments name within the Report, as well)
  • The input field for the segment's name disappers on focus loss - if you save the segment then, you get an error message and you have to click "edit" again to enter the segment's name. This could be a bit confusing; The behaviour should be, that the segment's name input field should be always visible when you are setting up a new segment. Does this make sense?
  • (minor flaw) the segment param is kept in the url when you jump back to the "All websites" dashboard
  • (probably segment related) the segmentation for providers doesn't seem to work correctly (no visitors found, even if there are visitors for the selected provider)

This is all I found at testing so far!

Last edited 11 months ago by peterb (previous) (diff)

comment:41 Changed 11 months ago by peterb (peterbo)

Another Question: How to enable auto_archiving for a specific segment? There is a column for that in the segments table.

comment:42 Changed 11 months ago by matt (mattab)

Will not add segment editor to email reports in v1 but we will do soon, I am sure (hopefully can get sponsoring for it).

Thanks for the bugs, will fix them asap. Also will add the selector to set a segment as "Auto archive" (this is last item of my todo list!).

please keep feedback/bug report coming

comment:43 Changed 11 months ago by SteveG (sgiehl)

In 7c58869c60a2b6b3cdf68dca4d0a4046449942aa:

refs #2135 always show autocomplete when clicking on the input element

comment:44 Changed 11 months ago by matt (mattab)

In 229a3f213f1102c197549e282d6324416d9412cf:

Refs #2135

  • Fixing select boxes logic/API
  • better design for anonymous user. He can apply the segments that are shared
  • better message in case anonymous tries delete/edit a custom segment
  • Adding ability to set a segment as "PRE-PROCESS" or "PROCESSED ON DEMAND (Real time, default)"

comment:45 Changed 11 months ago by matt (mattab)

In 1f87f85c1a20557f34b08cdde6ea9bc137a43822:

Refs #2135

  • Fixes "The input field for the segment's name disappers on focus loss"

comment:46 Changed 11 months ago by matt (mattab)

In 047412dee8725f6f69b2c8f6bfe04d905f180673:

Refs #2135

  • Not forwarding segment parameter to the top menu links

Note: ideally we should display the Segment Editor on the WIDGET and API pages refs #3934

comment:47 Changed 11 months ago by matt (mattab)

In 13ba840fcb226a8d2f6217c28d7f71e54b6ab360:

Fixing non working footer link refs #2135

comment:48 Changed 11 months ago by matt (mattab)

I think the code is now complete. Please post here any suggestion or bugs you find. I will also add user documentation separately. Thanks for testing & all the best enjoying this feature and finding new insights!

comment:49 Changed 11 months ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from new to closed

comment:50 Changed 11 months ago by matt (mattab)

In 21877807842483cc4a3c1b3d472ddc549bb4dee7:

Refs #2135 fixing CSS/image which wasnt big enough for German button label

comment:51 in reply to: ↑ 16 Changed 11 months ago by matt (mattab)

Replying to nathanzachary:

It would also be nice if the visitor log could be filtered by 'Provider.' At current, I have to grep through my logs for the provider name, and then compare it to the Apache logs to see what pages the visitors viewed.

Cheers,
Nathan Zachary

Btw this is now possible since Benaka fixed Provider support in: https://github.com/piwik/piwik/commit/9824c498d7c180a17a562872b171f31c61da13f5

Grab yourself the 1.12-b22 and enjoy!

Note: See TracTickets for help on using tickets.