Opened 21 months ago

Closed 19 months ago

Last modified 13 months ago

#3332 closed New feature (fixed)

Transitions: reports about previous and following actions for each page URL

Reported by: EZdesign Owned by:
Priority: critical Milestone: 1.9 -- Piwik 1.9
Component: Core Keywords:
Cc: Sensitive: no

Description (last modified by matt)

Awesome feature: Analytics Previous / next pages

For each page URL, this plugin reports the top X

  • previous pages
  • following pages
  • outlinks
  • downloads
  • campaigns
  • search engine keywords
  • external websites

The information is given in a popover which can be launched via a table action on the pages report.

Attachments (12)

Transitions forum page.png (158.2 KB) - added by matt 20 months ago.
Example Transition Aug 22nd
Transitions open.png (62.3 KB) - added by matt 20 months ago.
Icon on hover could have colors and show a flow ? or something else, like a button and a text label ?
firebug error.png (19.7 KB) - added by matt 19 months ago.
merged JS file showing errors
too much data, direct entry is partly drawn.png (96.3 KB) - added by matt 19 months ago.
Example blog post page with all previous-next types set.png (133.3 KB) - added by matt 19 months ago.
Missing incoming traffic, does not sum to 61155 pageviews.png (77.5 KB) - added by matt 19 months ago.
No data for URL.png (93.7 KB) - added by matt 19 months ago.
Web Analytics Reports - piwik.org - Over 1 year - took 20s to generate.png (144.4 KB) - added by matt 19 months ago.
missing direct entries.png (97.7 KB) - added by matt 19 months ago.
03.png (28.7 KB) - added by matt 19 months ago.
unicode not displayed.png (47.4 KB) - added by matt 19 months ago.
unicode URL displayed correctly in report.png (20.4 KB) - added by matt 19 months ago.

Download all attachments as: .zip

Change History (107)

comment:1 Changed 21 months ago by EZdesign (BeezyT)

The plugin fakes an instance of Piwik_ArchiveProcessing_Day in order to use archiving code for live querying (later pre-archiving might be added). The plugin uses Piwik_ArchiveProcessing_Day::query*ByDimension. These methods have been extended in #3330.

From the returned data, only nb_actions is used in Transitions. In particular, count(distinct idvisit) and count(distinct idvisitor) is processed in MySQL and not used in the report. Count distinct takes significantly more time and memory than other aggregation functions. Should we extend the query*ByDimension methods to limit the columns in the result? The signatures are already rather long after #3330. I'm not sure what's best...

comment:2 Changed 21 months ago by EZdesign (BeezyT)

(In [6811]) refs #3332: Preparations for Transitions

  • advanced period factory
  • fake archive processing for live querying

comment:3 Changed 21 months ago by EZdesign (BeezyT)

(In [6812]) refs #3332 Transitions

  • Archiving code that doesn't hook into the actual archiving
  • API code that fakes an archive processing instance and calls archiving
  • PHPUnit test

comment:4 Changed 21 months ago by EZdesign (BeezyT)

(In [6813]) refs #3332: properties for previous commit

comment:5 Changed 21 months ago by EZdesign (BeezyT)

In case you're wondering: I can't commit properties from git svn. I have to commit with git svn, then update, propset & commit with svn. That's why I always set properties in a separate commit.

comment:6 Changed 20 months ago by EZdesign (BeezyT)

(In [6843]) refs #3332 row actions registry: preparations for Transitions UI

  • row actions are no longer in the HTML templates
  • they are generated in JS
  • plugins can register their custom actions in a registry
  • actions can be available on a subset of tables and rows

comment:7 Changed 20 months ago by EZdesign (BeezyT)

(In [6844]) refs #3332: Transitions UI (row action for pages report + popover)

comment:8 Changed 20 months ago by EZdesign (BeezyT)

(In [6845]) refs #3332 properties

comment:9 Changed 20 months ago by EZdesign (BeezyT)

So here's the current state of the Transitions UI. The plugin adds a new row action to the pages report which opens the Transitions popover.

It would be great if some of you could give it a try and report back to me. I'm especially interested in performance. How long does it take to load the popover on how many pageviews per day?

Please don't take too much time writing suggestions for the UI itself. We already have lots of changes planned. Once that's done and you still have suggestions, please let me know.

The icon that launches the popover is just a placeholder. I'd like to have an icon that looks a bit like the popover with curvy bars going in and out of a box. Unfortunately, everything I came up with didn't look good. Do we have someone who could design the icon?

comment:10 Changed 20 months ago by EZdesign (BeezyT)

(In [6846]) refs #3332 more work on Transitions

  • highlight the internal traffic as a group (light blue background + headline)
  • close popover on overlay click
  • slight color changes
  • refactoring to remove duplication in JS
  • "Others" not clickable anymore
  • center box contains more info and is a little higher

comment:11 Changed 20 months ago by EZdesign (BeezyT)

Even though we did URL normalization recently, we still have some problems with non-normalized URLs in this plugin. URLs with different domains or different strings after the hash have different idactions but end up in the same row in the pages report. This causes inconsistency because the numbers from the pages report and the ones derived on the basis of the idaction don't match. Since the Transitions report mixes both, that's bad.

One possible solution would be to add two options to the configuration:

  • One to replace the tracked domain with the main domain as provided in the website configuration. I'm not sure whether we should replace all domains or just the ones configured as alternative domains.
  • Another option would normalize a little more, removing the hash part and other stuff that causes two idactions to end up in the same row in the actions report.

When both these options are enabled, the problem should be gone. When enabling one of the options would break the links for a user, he can disable them. Honestly, I think most users wouldn't care very much about broken links. But with the options, it's up to them.

An alternative solution would be this: in order to avoid conflicting numbers from the pages report and the Transitions plugin's archiving, we don't reuse the pages report and have to get the main metrics from the logs. In particular, we're talking about #hits, #entries, #exits. it should be possible to get #hits by summing up the other numbers.

So what do you think? Which option's better? Are there other alternatives?

comment:12 Changed 20 months ago by EZdesign (BeezyT)

(In [6853]) refs #3332 Transitions

  • other groups (websites, search engines, campaigns) can be opened
  • refactored some redundant JS code
  • translations

comment:13 Changed 20 months ago by EZdesign (BeezyT)

(In [6854]) refs #3332 updated expected test output for Transitions

comment:14 Changed 20 months ago by EZdesign (BeezyT)

(In [6855]) refs #3332

  • reset open group when closing the popover or changing the url
  • fixed some lint

comment:15 follow-ups: Changed 20 months ago by matt (mattab)

From the returned data, only nb_actions is used in Transitions. In particular, count(distinct idvisit) and count(distinct idvisitor) is processed in MySQL and not used in the report. Count distinct takes significantly more time and memory than other aggregation functions. Should we extend the query*ByDimension methods to limit the columns in the result? The signatures are already rather long after #3330. I'm not sure what's best...

If it's one more parameter and not too tricky, I'd say it's better to put it in. It might earn a few milliseconds here and there, which would tremendously help for pre-archiving (where milliseconds quickly make minutes ;))

So please add the parameter, and unit test, and all good. It's complicated but it has to be.

Even though we did URL normalization recently, we still have some problems with non-normalized URLs in this plugin. URLs with different domains or different strings after the hash have different idactions but end up in the same row in the pages report.

Hash tags: good point, I put a feature request in: http://dev.piwik.org/trac/ticket/3232#comment:3 - comment there if you have to add info.

Different domains: I think we can leave with this limitation for now. Users can use setCustomUrl in javascript to correct input URL if necessary...
Later we could also implement a filter mechanism to modify input URLs and define rules in the piwik UI, but seems too edge case for now - UI and usability is IMHO more useful than perfect data!

Changed 20 months ago by matt (mattab)

Example Transition Aug 22nd

Changed 20 months ago by matt (mattab)

Icon on hover could have colors and show a flow ? or something else, like a button and a text label ?

comment:16 in reply to: ↑ 15 Changed 20 months ago by EZdesign (BeezyT)

Hash tags: good point, I put a feature request in: http://dev.piwik.org/trac/ticket/3232#comment:3 - comment there if you have to add info.

Not recording the hash part in the database by default (as you suggested in #3232) would help lots.

Thanks for creating such a lovely icon by the way - lol! Seriously, there must be someone who is good at this kind of thing and can design an icon for us.

Also, why are my commits not showing up in the ticket anymore? I made four today referring this ticket and none are shown here.

comment:17 in reply to: ↑ 15 Changed 20 months ago by EZdesign (BeezyT)

Replying to matts E-Mail:

Popover Title: could have a one line title such as "Transitions for /blog/hello-world... - April 15th 2012" indicating 2 important elements: date, and the page we are looking at.. would add real value without using more pixels, always appreciated.

The popover already conatins a lot of text and the title would make it look really packed. Also the title is way too big compared to the other text in the popover. Imho it looks quite ugly. Besides, the page we are looking at is already indicated in the center box.

So I'd rather not set the title. OK?

comment:18 Changed 20 months ago by EZdesign (BeezyT)

(In []) refs #3332

  • aligning outgoing traffic right, pageviews centered in the main box
  • fixing last imprefect translation
  • removing some duplication from JS

comment:19 Changed 20 months ago by SteveG (sgiehl)

(In [6868]) refs #3332, refs #3227 fixed possible fatal if no row is available for page metrics; do not run transition tests by default; ensure that _all_ plugins are loaded before integration tests

comment:20 Changed 20 months ago by matt (mattab)

  • Milestone changed from 1.8.x - Piwik 1.8.x to 1.8.4 - Piwik 1.8.4

Timo, WOOW! Excellent feature, awesome potential and I really like the first implementation you made!

Here is my feedback regarding the User Interface.

It took 3-4 seconds max (often 1s) to load for the homepage with 1500 pageviews. It is definitely super responsive ons mall websites, very nice speed! Benefits of RankingQuery and URL normalization are huge.

Very well done.. :-) impressed by your work & the performance side of it.

Here is full review of using the feature review & making it more accessible:

  • Popover Title: could have a one line title such as "Transitions for /blog/hello-world... - April 15th 2012" indicating 2 important elements: date, and the page we are looking at.. would add real value without using more pixels, always appreciated.
  • Loading: on click to load a new page, would be nice to show a white background with opacity, and a loading message in the middle, as to avoid the 'flash' that occurs
    • Ideally the loading would say "Loading Transitions for $url_truncated_with_tooltip" as to make waiting user fully aware of what's happening during these long seconds ;-) Hopefully only for high traffic piwik. So basically add "Loading Transitions" above the loading spin?
  • About the Central Panel
    • Important UX/data improvement: on hover on the central panel, the absolute numbers could be displayed with opacity, on left and right outside to the central div, in their respective flow? It would give a nice flashy/halo effect and would also display the numbers on the bar itself, avoiding user/brain to read & parse the text in the middle. I think this would also look quite awesome, especially if the border to the central DIV would slightly illuminate or something like that?
    • Could "Outgoing traffic" text section block be aligned on the right as to visually show it refers to data on the right?
    • Could Exit bar be orangy/grey so it doesn't give bad feeling, but green is too positive for exits?
    • The number of page views for zoomedin page should be written much bigger, font 16pt.
    • Below the big page view metric should be "This page is N% of all pageviews" in grey, as to give context of this page within the whole.
    • Could the H2 have a font-size: 15pt or so? It looks nice.
  • A few times clicking on the labels, I got a "The label '/feed.php' could not be found in Actions.getPageUrls". In this case, could you instead leave the previous view displayed, and simply write a message inline below the clicked label, saying "No data found for $label_clicked", instead of popup?
  • When clicking from one Flow group to another (from Websites to Search Engines) would it be possible to see the transition animation of the groups in 500ms in the UI, so to show users how the ui works? I guess it's non trivial. ;)
  • When a "group" is hovered, it would be nice that all groups on the same side, would display the number of clicks in the bar
    • Note: whenever we draw numbers in bars, we should only draw for numbers > X% so that there is enough space for the text + some white space to look nice. Small occurence referers groups will not have the metric displayed.

Other review:

  • It would be great to have URL with hash tags to open the Transitions view (idem for Row Evolution)
  • Transitions & Row Evolution feature access:
    • I like the idea of icons, even Row Evolution and Transition icons could be colored on hover.
      • On hover on the Transitions icon, I have an idea: the current grey T letter, could on hover show the next/previous flow on left / right of image? I guess we'd have to design an icon for it to look ince, I crafted a preview (see screenshot) but it's not ideal is it ;)
      • Also we could extend the 2 icons as buttons, and write "[ See Evolution ] [ See Transitions! ]" or similar. On hover, the icons could also show a Label a make like a little button?

These icons+labels in a button would represented for what they really are: "Actions" for a given row.

More reviews to come later

comment:21 Changed 20 months ago by matt (mattab)

  • Priority changed from normal to critical

comment:22 Changed 20 months ago by matt (mattab)

(In [6987]) Removing temporarily Transitions plugin from release Refs #3332

on the demo, when enabling the plugin, it was creating a JS error in the merged JS file

comment:23 Changed 20 months ago by matt (mattab)

(In [6988]) Removing temporarily Transitions plugin from release Refs #3332

comment:24 Changed 19 months ago by EZdesign (BeezyT)

Some general comments about the reviews (also the ones received via e-mail)

  • Thanks for all the suggestions, they were very useful!
  • I'll start with what I won't do: we can't use larger fonts because urls can get very lengthy and it's good to have as much of them as possible. Also, I won't implement an animation when opening groups because it would be very hard to do and result in even more complex code (which is quite complex already).
  • The conntection between the flows and the metrics in the center were hard to see. Many of your comments resulted from this. In the upcoming commit, I have implemented something to make it easier.
  • We need some general popover code: loading messages, changing between popovers and using the history. Many of the comments related to that were almost identical to comments about RowEvolution.
  • Outlinks and page URLs should be clickable, I agree. I think we should use a redirect proxy to hide the current URL from the HTTP referrer. As far as I can see, that doesn't exist so far. Right?

comment:25 Changed 19 months ago by EZdesign (BeezyT)

(In [7032]) refs #3332 adding Transitions again

comment:26 Changed 19 months ago by EZdesign (BeezyT)

(In [7033]) refs #3332 highlighting groups on hover in Transitions

comment:27 Changed 19 months ago by EZdesign (BeezyT)

After the last commit, when you hover a goup title, it highlights the metric in the center. Also, the group is highlighted when you hover the metric in the center box. This makes the mapping between the groups and the center box easier.

comment:28 Changed 19 months ago by EZdesign (BeezyT)

(In [7035]) refs #3332 nicer loading message for transitions, introducing a general popover helper

comment:29 Changed 19 months ago by matt (mattab)

(In [7037]) will release 1.9 B to test transitions on demo and bring back Row evolution on demo Refs #3332

comment:30 Changed 19 months ago by matt (mattab)

When Transitions is enabled, the Merged Javascript file returns the error:

SyntaxError: missing } after property list
index.php?module=Proxy&action=getJs&cb=0703c62885101c6797656014692f23aa
Line 1122

See attached screenshot showing error in firebug.

Changed 19 months ago by matt (mattab)

merged JS file showing errors

comment:31 follow-ups: Changed 19 months ago by matt (mattab)

Wow very cool changes! They make a big difference indeed :)

I think we should use a redirect proxy to hide the current URL from the HTTP referrer. As far as I can see, that doesn't exist so far. Right?

Yes it's there, you can use: href="?module=Proxy&action=redirect&url='.urlencode($URL)

After the last commit, when you hover a goup title, it highlights the metric in the center. Also, the group is highlighted when you hover the metric in the center box. This makes the mapping between the groups and the center box easier.

Very nice indeed!

Some feedback after testing this feature for 1 hour:

  • Tooltip for a given previous/next page, on hover on left/right percentages, says: "22 of 45 from search engine", could you slightly clarify and write "$URL \n 22 visits (out of 45 visits) from search engines"
    • I think that, on hover on the label, and the %, the same tooltip should be shown which would contain URL + detail. This way we don't have 2 different tooltips (one for URL, one for the %) in the same block.
  • currently the orange is displayed/highlighted when the label is hovered. Could you also color in orange, when the mouse hovers on the blue/green curbs on left/right of central box? It will really improve usability as it's natural for me to hover on the curb (ie. closer to central box) rather than the actual group's label
  • small detail: when the box is hovered orange, it would be nice to show the + icon in orange as well (rather than blue)
  • Could you add a tooltip, on the main "1834 pageviews" in the middle. The Tooltip would say "In July 2012, This page had 1834 pageviews (5% of all pageviews)"
  • when there are zero from internal page, from websites, or zero to internal page, etc. write the label in light grey to maximise focus on data points
  • Transitions works for Actions>Pages report. Could you also enable it for Actions>Dowloands and Actions>Outlinks? ie. see the transition report for a specific downloaded file, would be useful.
    • For example It would be very interesting to see for "piwik.org/latest.zip" which pages and keywords were used as previous interaction to the download!
    • Also it would _really_ interest me as I'd like to show Transitions for Internal search keywords :)
  • the center box top blue link (ie. the name of the page looked at) should link to the actual page (with target=_blank, via proxy)

Transitions is such a great feature!! :)

comment:32 Changed 19 months ago by EZdesign (BeezyT)

(In [7043]) refs #3332 fixing JS error in merged file

comment:33 Changed 19 months ago by EZdesign (BeezyT)

(In [7045]) refs #3332 Transitions remembers the popover state in a second hash part of the URL => using direct links and the browser history is possible now!

comment:34 Changed 19 months ago by EZdesign (BeezyT)

(In [7046]) refs #3332 Transitions: main url, external referrers, outlinks and downloads open in new tab on click

comment:35 in reply to: ↑ 31 Changed 19 months ago by EZdesign (BeezyT)

  • Tooltip for a given previous/next page, on hover on left/right percentages, says: "22 of 45 from search engine", could you slightly clarify and write "$URL \n 22 visits (out of 45 visits) from search engines"

I played a bit with that idea and decided to do only a minor change

  • We can't write "visits" because the base is not visits but transitions. One visit can do multiple transitions to an internal URL for example.
  • Writing the URL in the tooltip is a bit much and distracts from the actual info we want to show.
  • I like "X (out of Y)" better than "X of Y", the change will be included in the next commit.
  • currently the orange is displayed/highlighted when the label is hovered. Could you also color in orange, when the mouse hovers on the blue/green curbs on left/right of central box? It will really improve usability as it's natural for me to hover on the curb (ie. closer to central box) rather than the actual group's label

Unfortunately, that's a limitation of canvas: we can't attach events to the shapes. I know it would be nice but it's nearly impossible.

  • For example It would be very interesting to see for "piwik.org/latest.zip" which pages and keywords were used as previous interaction to the download!

It would be nice to have downloads, campaigns and external websites in the center. It's on my list but I won't do it for the first version.

  • Also it would _really_ interest me as I'd like to show Transitions for Internal search keywords :)

It is planned to have internal search keywords on the left so you can see which searches lead to the page. Having internal searches in the center is interesting but that kind of info should be included in the internal search plugin.

comment:36 in reply to: ↑ 31 Changed 19 months ago by EZdesign (BeezyT)

  • small detail: when the box is hovered orange, it would be nice to show the + icon in orange as well (rather than blue)

Actually, I think it's good that it's blue because the orange background highlights the blue icon. If we make it orange on orange background, it's easier to miss it and expanding the groups is one of the coolest parts about the feature so I'd like to make it visible.

comment:37 Changed 19 months ago by EZdesign (BeezyT)

(In [7050]) refs #3332 Transitions:

  • tooltip on title showing share of all pageviews and date range
  • metrics with value 0 are greyed out in the center box
  • adjusted dimensions for wider popover
  • re-clicking the current menu item reloads the page again
  • transitions is available on entry and exit pages as well

comment:38 Changed 19 months ago by EZdesign (BeezyT)

Matt, in your screenshot the percentage labels on the curves sometimes had two lines (becuase you don't have Arial). Can you please check whether that's fixed now?

comment:39 Changed 19 months ago by EZdesign (BeezyT)

(In [7051]) refs #3332 Transitions:

  • Showing referrals from campaigns in the center box
  • Proper error message when URL was not found or truncated from the actions report (introducing general popover error method)
  • Stylish icon for row-action
  • Minor JS and CSS fixes

comment:40 Changed 19 months ago by matt (mattab)

(In [7052]) Refs #3332 will test latest Transitions update on demo :)

comment:41 Changed 19 months ago by matt (mattab)

(In [7054]) Refs #3332 Fixing error when merged JS file, broadcast.js should be loaded before other files eg. it is used in datatable_rowactions.js which was loaded first on the demo

comment:42 Changed 19 months ago by matt (mattab)

very nice changes!

I put some screenshots to show examples of the latest of Transitions.

Feedback:

  • The tooltip on the focused page "This page had N views (M% of all visits)" Here the % displayed is not valid, it shows 0% or 0.2% when should be more. In the link above page had 3353 / 55538 = 6% but displays 0.1%
  • There is a scrolling bar on the right of popover, Cf. screenshot, which can be removed (maybe popover slightly higher?)
  • In the middle box, hovering on an incoming/outgoing traffic type will highlight the flow. Please make these labels also clickable, so they expand the flow, for added efficiency analysing one page transitions. we can then only keep the mouse in the middle block to analyse the page.

Having internal searches in the center is interesting but that kind of info should be included in the internal search plugin.

It isn't planned in SiteSearch to show the most "pages starting a search" or "Most visited pages after search". I think in fact it should use transitions for that, since it's the perfect UI for it.

It would be nice to have downloads, campaigns and external websites in the center.

It would be really great to have... Hopefully you might change your mind before V1 :)

  • I still have to review code for security & best practise, will do once I finish sitesearch.
  • The % now display correctly on one line.
  • OK with your other feedback!

Changed 19 months ago by matt (mattab)

comment:43 Changed 19 months ago by matt (mattab)

Further feedback:

  • Nice that you added Campaigns in middle box, I didn't notice it was missing
  • The Loading... and the Hash tag features are really useful & add a lot to usabilit!
  • Now that I think of it, it's interesting that it works with the double hash tag in the URLs... I even tested with IE and it works. I'm not sure if it's standard practise, did you try to have a Hash tag as follows?
#module=Actions&action=indexPageUrls&idSite=7&period=month&date=2012-08-23&RowAction=Transitions:http://forum.piwik.org/

instead of

#module=Actions&action=indexPageUrls&idSite=7&period=month&date=2012-08-23#RowAction:Transitions:http://forum.piwik.org/

didn't load neither, so not sure.

  • When keywords are long, Direct entry is partially drawn
  • On this screenshot you can see that the sum of incoming traffic+reload is not equal to the pageviews of the clicked page. 49192+5716+366+1161−61155 = -4720 pages difference. What do you think are possible explanations for such discrepency ?

============================
Timo's answer:

  • Now that I think of it, it's interesting that it works with the double hash tag in the URLs... I even tested with IE and it works. I'm not sure if it's standard practise, did you try to have a Hash tag as follows?

We need to have two separate parts because the change of the second part should not cause a change to be triggered in the first part. This means that when the popover changes the report in the background should not be reloaded. Apart from perfomance that would be really bad if a user has opened several folders on the actions report and the actions report is back to start when he closes the popover.

comment:44 Changed 19 months ago by EZdesign (BeezyT)

(In [7057]) refs #3332 Transitions:

  • if a url isn't found in the actions report, show the rest of the UI even though exits and bounces are not available (previously, there was an error message)
  • the number of pageviews is derived from the live queries - makes data more consistent
  • sanitize data taken from actions report using the live data
  • more robust second hash handling

comment:45 Changed 19 months ago by EZdesign (BeezyT)

I'm sorry, I changed you last comment (http://dev.piwik.org/trac/ticket/3332#comment:43) instead of replying to it. I still have it in my notes so none of your suggestions will be lost :)

comment:46 Changed 19 months ago by EZdesign (BeezyT)

On this screenshot you can see that the sum of incoming traffic+reload is not equal to the pageviews of the clicked page. 49192+5716+366+1161−61155 = -4720 pages difference. What do you think are possible explanations for such discrepency ?

There can be multiple reasons:

  • The page has been loaded sometimes with a hash part or different domains. In this case, the number from the actions report is too high because multiple idactions map to the same row in the data table. This problem should be gone after the update.
  • Previous downloads and outlinks are not shown in transitions. 4720 is a bit much compared to the following downloads so I guess this number will drop after the next update. Let's keep an eye on this!

comment:47 Changed 19 months ago by EZdesign (BeezyT)

(In [7058]) refs #3332 Transitions:

  • clicking the group name in the center opens the group
  • fixing percentage of all pageviews
  • when all groups on the left are available, draw them a bit more compact in order to fit into the popover
  • showing previously removed error message again in a real error case
  • removing forgotten debug output

comment:48 Changed 19 months ago by EZdesign (BeezyT)

(In [7059]) refs #3332 Transitions:

  • hover effect and tooltip for row action icons
  • more robust hash handling

comment:49 Changed 19 months ago by EZdesign (BeezyT)

Please update the demo so that we can verify that the problems you reported are fixed.

comment:50 follow-up: Changed 19 months ago by EZdesign (BeezyT)

I had a crazy idea... If we really trust the queries i wrote, we can derive the number of exits from pageviews - following pages - outlinks - downloads. If we abandon the number of bounces, we could get rid of using the actions report altogether! That should speed everything up a lot - especially for pages with many distinct URLs. Thoughts?

comment:51 in reply to: ↑ 50 Changed 19 months ago by matt (mattab)

Replying to EZdesign:

I had a crazy idea... If we really trust the queries i wrote, we can derive the number of exits from pageviews - following pages - outlinks - downloads. If we abandon the number of bounces, we could get rid of using the actions report altogether! That should speed everything up a lot - especially for pages with many distinct URLs. Thoughts?

Getting rid of bounces is not a problem, so if it makes code faster, sounds really good to me!

comment:52 Changed 19 months ago by matt (mattab)

(In [7066]) Refs #3332 Will test new Transitions on demo (Nice tooltip!!) ;-)

comment:53 Changed 19 months ago by EZdesign (BeezyT)

(In [7072]) refs #3332 Transitions: the actions blob report is not used anymore - improved performance, consistency problems are gone, bounces are not available anymore

comment:54 Changed 19 months ago by EZdesign (BeezyT)

(In [7073]) refs #3332 Transitions performance tuning

  • load total number of pageviews only once and from JS => archiving is never launched from Transitions API
  • load markup only once => when clicking a different URL, only the new data is loaded
  • load markup and data in parallel

comment:55 Changed 19 months ago by EZdesign (BeezyT)

After the last refactoring, the MySQL queries account for almost the entire processing time when switching between pages. Speeding them up will improve the UX a lot.

Here are the indexes that will do the job:

ALTER TABLE piwik_log_link_visit_action ADD INDEX transitions_url ( idaction_url , idsite , server_time );

ALTER TABLE piwik_log_link_visit_action ADD INDEX transitions_url_ref ( idaction_url_ref , idsite , server_time );

ALTER TABLE piwik_log_visit ADD INDEX transitions ( visit_entry_idaction_url , idsite , visit_last_action_time );

The order of the columns is very important. Make sure not to change it.

I'm not saying we should add them by default but maybe we can document that adding them will speep up transitions tremendously. On my Piwik, it brought processing time down from 1-2s to 100-200ms. For larger databases, the factor should be even bigger. Maybe we can try on the demo?

comment:56 Changed 19 months ago by EZdesign (BeezyT)

(In [7074]) refs #3332 Transitions:

  • segmentation works now!
  • fixing a js bug related to concurrent ajax requests
  • refactoring of the api class
  • translation update

comment:57 Changed 19 months ago by matt (mattab)

Excellent updates, faster & simpler code...

My last request as you know would be to support Transitions for Page Titles... then the feature will be fully complete IMHO!

comment:58 Changed 19 months ago by matt (mattab)

It seems the "Direct entry" metric is not processed correctly, see two attached example screens of missing direct entry.

Changed 19 months ago by matt (mattab)

comment:59 Changed 19 months ago by EZdesign (BeezyT)

(In [7091]) refs #3332 Transitions

  • support for page titles
  • language strings for error messages are loaded if not present
  • some core extensions

comment:60 Changed 19 months ago by EZdesign (BeezyT)

(In [7093]) refs #3332 Transitions:

  • center url / page title gets truncated if it's longer than four lines
  • main tooltip shows the full URL (including domain, protocol and www, untruncated)
  • truncation for boxes on the sides works again
  • tolerate url=null passed from php (happens when there are encoding errors in the db)
  • don't try to shorten page titles like urls

comment:61 Changed 19 months ago by matt (mattab)

Kuddos Timo for all these great commits....!

  • I LOVE seeing Transitions for Page Titles, this is really very useful & more readable for many websites! :) Very powerful!
  • It seems "Page title not defined" page title does not show Transitions but appears in Previous/Next. Does it sometime work? to be tested further
  • Transitions does not work for Page titles containing / which are grouped in categories and sub-cats. The icon should be hidden on these expandable groups?

I will update on demo now to test further!

Last edited 19 months ago by matt (previous) (diff)

comment:62 Changed 19 months ago by matt (mattab)

(In [7103]) Refs #3332 Summing rows now keep the latest metadata to ensure the latest format of URLs is used when summing over large date range

comment:63 follow-up: Changed 19 months ago by matt (mattab)

After using Transitions further, I have the following suggestion

  • Suggestion: Display the Transitions icon, on directories
    • Why? many times, the /sub/dir/ page will actually have transitions, since there is a page at example.org/sub/dir/. This is the case for all wordpress blogs by default and many other CMS, which serve pages at the root of each directory/category/tag.
    • How? When there is no data for this page, we could automatically run the query with the segment pageUrl=@http://example.org/sub/dir/ so Transitions would report what you'd expect: all transitions for all pages in the clicked group.
      • Blocker? I tested it seems transitions does not return data when used with the segment &segment=pageUrl=@http://piwik.org/docs/ -- to be tested further

comment:64 Changed 19 months ago by matt (mattab)

(In [7113])
Refs #3332

  • Transition works for Page titles with hierarchy (eg "$Category / $name")
  • URL metadata:
    • is only available on non Summary rows.
    • It is now set to the page with highest number of visits, when page names do overwrite.
  • However now, by default we do track keep hash tags and we do not aggregate them in one page URL.
    • Refs #3232 Will be nice to have, to allow to "not track fragment" by default. See stub processUrlFragment(). More to do: Admin UI, Tests w/ and wo/ hash tags, w/ Capital letter in hostname
    • URL being tracked, and URL read from the logs (for backward compatibility) are now cleaned: hostname is lowered, and URL Fragment is kept/removed.

comment:65 Changed 19 months ago by matt (mattab)

(In [7114])
Refs #3332
Will test latest changes on demo, to check if no more bugs

comment:66 Changed 19 months ago by matt (mattab)

  • I notice a bug: the Transitions icon is displayed on Page titles with hierarchy, could we hide it while we wait for later implementation of: #comment:63

comment:67 Changed 19 months ago by EZdesign (BeezyT)

(In [7124]) refs #3332

  • direct entries are shown last in the center (same order as on the left)
  • entries and exits in light blue
  • title of open group in orange

comment:68 in reply to: ↑ 63 Changed 19 months ago by EZdesign (BeezyT)

Replying to matt:

  • Suggestion: Display the Transitions icon on directories

If you want to see tranitions for example.org/foo/bar/, just open it on foo > bar > index. Doing it for foo/bar/* (i.e. the sum of everything in the directory) doesn't work by design because Transitions always shows stats for one idaction only. Doing something else would mean major code changes and performance issues. I'm generally not in favor for doing that but one week before the release that's definitely not possible...

comment:69 Changed 19 months ago by EZdesign (BeezyT)

(In [7125]) refs #3332

  • Transitions is not available on page title groups anymore
  • JS is slightly more efficient with row actions

comment:70 Changed 19 months ago by EZdesign (BeezyT)

(In [7126]) refs #3332

  • use action_url_category_delimiter setting for hierarchical page titles.
  • Piwik_Common::unsanitizeInputValue should not be used on page titles. otherwise titles with an ampersand don't work anymore.

comment:71 Changed 19 months ago by EZdesign (BeezyT)

It seems "Page title not defined" page title does not show Transitions but appears in Previous/Next. Does it sometime work? to be tested further

It works for me. Maybe I fixed it in [7126]? Please try again.

comment:72 Changed 19 months ago by EZdesign (BeezyT)

(In [7128]) refs #3332: edges on the sides are a bit straighter

comment:73 Changed 19 months ago by EZdesign (BeezyT)

(In [7132]) refs #3332

  • many cross browser fixes (mainly for internet explorer 7-9)
  • removing now unneeded methods Piwik_Popover.prepareContent() and Piwik_Popover.showPreparedContent()
  • using different canvases for left and right to make clearing only one half possible with excanvas
  • drawing loops on a separate canvas to make proper redrawing possible
  • canvases are cached in a similar fashion to the graph canvases in order to prevent memory leaks
  • fixing issue where lines in tooltip don't break in firefox and ie
  • applied auto code formatting (many whitespace changes)

comment:74 Changed 19 months ago by EZdesign (BeezyT)

(In [7133]) refs #3332: more cross browser compatibility and other fixes

  • the popover location is not put after a second hash anymore because that caused trouble in safari and internet explorer. instead, a popover parameter is introduced.
  • special url encoding for the popover location to make sure it's en- and decoded the same number of times.
  • minimum width for the label column in tables with row action.
  • tooltip behaves slightly nicer in ie7.
  • fixing a regression that broke transitions for subtables in the page titles report.
  • now transitions works in ie789, safari, firefox and chrome. in ie78, the css3 properties of the center box (border radius and shadow) don't work and excanvas can still be pretty slow sometimes. apart from that, everything's fine.
  • tested for memory leaks in all browsers. none found.

Apart from translation and documentation, that's it!

PLEASE TEST AS MUCH AS YOU CAN AFTER THIS COMMIT!

comment:75 Changed 19 months ago by EZdesign (BeezyT)

(In [7137]) refs #3332

  • fixing page titles containing the > character. they used to be interpreted as recursive labels, replacing the > with /. now, js puts together the label correctly. it needs to know the action_url_category_delimiter, so it's added to the global js variables.
  • fixing error for "Page URL undefined". since there is no consistent data that can be shown for this row, transitions is disabled on it.

comment:76 Changed 19 months ago by matt (mattab)

(In [7143]) Refs #3332 Fixing JS error on admin pages

FYI since Piwik_Controller is only used for reports controllers (where as Piwik_Controller_Admin is for admin controllers), the separator wasn't set on admin screens

comment:77 Changed 19 months ago by matt (mattab)

(In [7148]) Refs #3332 This will set the metadata on the URL with the most page views, for daily reports
Note to self: we could add a test case for this...

comment:78 Changed 19 months ago by matt (mattab)

(In [7160]) Refs #3332 Hopefully, fixing all problems with URL being incorrectly aggregated! Thanks Timo for the hint and bug in previous algorithm

comment:79 Changed 19 months ago by matt (mattab)

Test message notification?

comment:80 follow-up: Changed 19 months ago by matt (mattab)

It seems that all "data bugs" should now be fixed, please notify me if you find any further issue!

Timo, I noticed there is a small problem: unicode characters don't get displayed, instead the URL encoded version is displayed, in Transitions.

For example in this URL

See attached screenshot

comment:81 in reply to: ↑ 80 Changed 19 months ago by EZdesign (BeezyT)

Replying to matt:

Timo, I noticed there is a small problem: unicode characters don't get displayed, instead the URL encoded version is displayed, in Transitions.

The UTF8 chars are shown in the label of the report but the link is urlencoded. That's why it's urlencoded in transitions. I'm pretty sure the urlencoded version is also what's in the database because Transitions looks for this version and works.

So what's the best behavior? Should we urldecode once before displaying urls in Transitions? What does the actions report do with the label?

comment:82 Changed 19 months ago by EZdesign (BeezyT)

(In [7165]) refs #3332 Transitions help button linking to doc page

comment:83 Changed 19 months ago by EZdesign (BeezyT)

(In [7166]) refs #3332: url decode in transitions in order to display utf8 urls

comment:84 Changed 19 months ago by EZdesign (BeezyT)

Please test the latest commit on the demo. The UTF8 chars should work now.

I tested the code with a url that contains both utf8 chars and an html tag. UTF8 encoding and XSS protection ist working in the center box, side boxes and the tooltip. Nevertheless, please test and review closely.

comment:85 Changed 19 months ago by EZdesign (BeezyT)

OK guys, this is it.

If you can't find more problems, we can close the ticket! Yay!!!

comment:86 Changed 19 months ago by EZdesign (BeezyT)

  • Resolution set to fixed
  • Status changed from new to closed

(In [7189]) fixes #3332

  • refactoring breakpoint methods from transitions to piwikHelper
  • loading message is now displayed properly for utf8 urls

comment:87 Changed 19 months ago by EZdesign (BeezyT)

(In [7201]) refs #2992, refs #3332: integrating previous and following internal searches into transitions. the code is starting to get a little complicated ;-)

comment:88 Changed 19 months ago by EZdesign (BeezyT)

(In [7214]) refs #2992, refs #3332 fixing two ui bugs in transitions / site search integration

comment:89 Changed 19 months ago by EZdesign (BeezyT)

(In [7216]) refs #3332, refs #2992 making transitions center box higher because new lines were added for site search. fixing a js bug when truncating urls.

comment:90 Changed 19 months ago by matt (mattab)

(In [7219]) Enabling Transitions plugin by default! fixes #3332
KUDDOS TIMO!

comment:91 Changed 19 months ago by matt (mattab)

  • Description modified (diff)

comment:92 Changed 18 months ago by EZdesign (BeezyT)

(In [7227]) refs #2465, refs #3332: using Transitions for getting the following pages in Insight

comment:93 Changed 18 months ago by matt (mattab)

(In [7302]) Fixes #3453 Thanks for tip tsfoer. ANSI_SQL compatibility. Refs #3332

comment:94 Changed 18 months ago by matt (mattab)

(In [7306]) Refs #3332
Fixing JSLINT errors + packaging changes in piwik.js in the minified script

comment:95 Changed 13 months ago by EZdesign (BeezyT)

In 725ef8772cc8994ca0b405611ea383056c5fe5ac:

refs #3332 fixing regression related to using ajaxHelper in transitions: use regular transitions callback when PHP throws an exception (e.g. NoDataForAction) to let transitions handle the error and show a message in the popover.

Note: See TracTickets for help on using tickets.