Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HeatMap - Piwik plugin to track user interaction and pointer movements #1680

Closed
anonymous-matomo-user opened this issue Sep 7, 2010 · 7 comments
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone

Comments

@anonymous-matomo-user
Copy link

I am developing a native Piwik plugin to enable tracking of user interaction and pointer movements. Name and features of this plugin makes it very similar to ClickHeat (#73), but, actually there are some major differences between these two.

  • ClickHeat is a standalone script/application while HeatMap is going to be a native Piwik plugin.
  • It will use Piwik's own tracking codes/API, data structures and procedures to provide the features.
  • This plugin aims to generate visual maps for pointer movements in addition to click maps. Additional event/tracking and data representations will be implemented as per suggestions from the community.

On the way to the development of this plugin, I would appreciate a guideline from you on these areas:

  • Where should we plug in javascript (tracker) codes to capture additional data from client end?
  • Which tracker method (and parameters) should be used to record pointer movement coordinates (as serialized array)? Ideally, this should be done on the unload event. And it seems like using existing methods such as logPageView or logGoal will duplicate other metrics. How do we work around?
  • By using Piwik's own tracker code, is it possible to POST captured data instead of GET requests? There's a limit on URL length in Apache, and it seems like our captured data often hits that limit when tracked with GET requests.
  • How about archiving (blob format)? Should generated maps be saved as images? Or should we store the raw data as compressed streams? For the later case, will there be any performance hit? We have to run queries on huge amount of data. When this data is in MySQL table, it works fine. But, how do we achieve the same performance (and memory utilization) when this data is stored in blob format?
    Keywords: heatmap,clickmap,pointer,mouse,cursor
@robocoder
Copy link
Contributor

@mattab
Copy link
Member

mattab commented Sep 8, 2010

"visual maps for pointer movements": I think this is similar to clicktale 'mouse move heatmaps' ? http://www.clicktale.com/product/mouse_move_heatmaps

I'm not going to discuss here JS tracking as I am not sure what the requirements are.

Performance is critical for this type of data storing and processing. I'm not sure why you consider doing this from scratch rather than modifying (or reusing) GPL'ed Clickheat, as doing this from scratch is a major work, probably at least 3 weeks (mockups, spec, development, i18n, doc, etc) ?

Clickheat works fine currently (I believe?), appart from 2 main issues

  • lack of proper / clean integration in Piwik (which you point out)
  • lack of scalability due to the way data is stored and fetched in clickheat.

Regarding clean UI integration in Piwik, it would be best to submit mockups of your vision that we could discuss here with the community.

Regarding scalability, I think that while raw data would be stored in mysql tables (I think clickheat is using files at the moment), these data sets would need to be aggregated daily (and summed for weekly/monthly/yearly aggregates), similar to other datasets in Piwik. Aggregation would ideally reuse existing algorithms (archiveProcessing) and add required helper in the class.

You mention problem with reading an click map data set from blob. Would you store one integer (eg, intensity) for each pixel in the screen? Maybe a lower resolution would be enough, ie. if you average 8 pixels, you would have eg. 1280*1024 / 8 * 20% hovered/clicked pixels = 32768 data points. If you store a simple array this would be quite cheap and fast to read. thoughts?

Maps should always be generated from the 'aggregated' data sets. Of course, these images would be cached to ensure reload is fast. Generating + caching these images could also be done at archiving time to make UI acccess very fast.

What are the segments you want/need to process your data sets against? ie. do you have a different map for all resolutions, for browsers, countries, etc. or just one single map? Segmenting is difficult and you can quickly have a processing time multiplied by factor 10 or 100. Maybe at first no segmentation should be done (appart from DATE + URL) ?

A feature to enable/disable recording would be nice. There is only need to record one day from time to time, ensuring DB doesn't get bloated with data, and Archiving process doesn't has to process mouse move + clicks every day (this would potentially be pretty slow). This could be coupled with a simple threshold 'number of pageviews tracked for this URL', for example max 500 or 1000 sessions tracked per URL per day.

Also a similar feature to choose which URLs should be tracked would be great, ie. only track clicks/moves on homepage and product page. This could be done at the JS level or PHP level (better do it at JS level if possible).

@mattab
Copy link
Member

mattab commented Nov 16, 2010

GadgetMaster, any update regarding this ticket?

@anonymous-matomo-user
Copy link
Author

This seems to be closed, but I wanted to add this note to the ticket in case someone wanted to tackle this in the future. I came across heatmap.js from Patrick Wied which might be perfect to solve the request. Heatmap.js is an open JavaScript library that can be used to generate web heatmaps via html5canvas elements. Details :http://www.patrick-wied.at/static/heatmapjs/

@anonymous-matomo-user anonymous-matomo-user added this to the Piwik 1.0 milestone Jul 8, 2014
@mattab mattab removed the worksforme The issue cannot be reproduced and things work as intended. label Mar 24, 2017
@mattab
Copy link
Member

mattab commented Mar 24, 2017

we'll post progress in this other ticket here: #6371

@mattab
Copy link
Member

mattab commented Mar 24, 2017

Good news: we're working on Heatmap Analytics for Piwik + scrollmap + click map! It will be awesome and make sure to hear when it's released by signing up here: http://www.heatmap-analytics.com/

UPDATE: Heatmap and Session Recording has been released 🎉

you can get it here: https://plugins.piwik.org/HeatmapSessionRecording

Or learn more in the user guides:

@melroy89
Copy link

melroy89 commented Apr 5, 2018

Well too bad this is a yearly contract, which costs money, I would love to see some open-source alternative actually.

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Development

No branches or pull requests

4 participants