Opened 4 years ago

Last modified 5 weeks ago

#1486 assigned New feature

New plugin: define Custom Alerts and receive email when triggered

Reported by: matt Owned by: tsteur
Priority: critical Milestone: 2.3.0 - Piwik 2.3.0
Component: Plugins Wishlist Keywords:
Cc: Sensitive: no

Description (last modified by matt)

UPDATE January 26th: Custom Alerts plugin is now available in the Marketplace! Test it by using the latest Piwik beta version, and then installing the Custom Alerts plugin in one click: go to Piwik, click Settings then click Marketplace and click Install button.


Create custom alerts:

  • define the alert name ("Big drop in purchases")
  • select a website (or list of websites) on which the Alert is defined
  • receive alert by email (email will contain alert description + link to Piwik dashboard URL for the given website ID and period).
  • select the Alert period: should it be daily, weekly or monthly?
  • select the report (Websites, Keywords, Countries, general stats)
  • define Metrics (visits, page view, avg. visit duration, Goal 1 conversions, total goal conversions, etc.)
  • define the Alert: when Visits decrease 50%, when purchases are more than 50 per day, etc.

See attached screenshot for preview.

Note: Autocomplete the dimensions will probably not be in the V1 of the script. Autocomplete would be a nice feature as it helps users not making mistakes (eg. is it Germany or DE or germany?).

This plugin requires each Piwik plugin to define their API metadata (list of reports callable, and list of metrics + translations for each metric). Covered in #1485

Attachments (2)

Alerts_AddNewAlert.png (51.3 KB) - added by matt 4 years ago.
Alerts.zip (28.1 KB) - added by halfdan 4 years ago.
Current status - not for production

Download all attachments as: .zip

Change History (60)

comment:1 Changed 4 years ago by matt (mattab)

  • Component changed from Core to New Plugin
  • Milestone set to 1 - Piwik 0.7 - DigitalVibes
  • Type changed from Bug to New feature

Feedback

  • This applies to: group reports by category within the SELECT
  • Pre-set useful alert: Alert me when Average generation time is more than 2 seconds
  • History of triggered alerts is long list. Maybe we can use AngularJs to display this list with paging?
Last edited 5 weeks ago by matt (previous) (diff)

comment:2 Changed 4 years ago by matt (mattab)

Note: Custom Alert plugin UI html/js/css controller that allows to select one or several websites will be reused for sure in the PDF plugin.

It would be great if this bit of code was made reusable, thanks!

comment:3 Changed 4 years ago by matt (mattab)

Thought dump about unit tests

test alert on one website
test_oneWebsite_visitIncrease
test alert setup on two websites
test_severalWebsite_visitIncrease
test alert setup on two websites, but user doesn't have access to one of these anymore
test_severalWebsite_userDoesNotHaveViewAccessForOneWebsite

test all metrics matches/contains/etc. conditions
test_metrics_matchesExpression
test_metrics_containsExpression_withChineseCharacters

test different periods
test all increase/decrease/below/above conditions
test_metrics_IncreasePercent_LastDay
test_metrics_increaseValue_lastWeek
test_metrics_specificGoalConversion_isMoreThan_lastMonth

tests searched string matches several rows (ie. keywords matching 'test')
test with bounce rate: bounce rate being a processed metric, all matching rows should be summed before processing the bounce rate
test_metrics_matchSeveralRows_bounceRate
same test, but simply with Visits evolution
test_metrics_matchSeveralRows_visits

+ test Goal metrics for one website, General goal metric alert..

For each of these tests, or maybe once for all tests, you could generate tracking hits using the code from Main.test.php

For example you would generate 10 or 15 visits to try and match all your use cases, then you would run in one test method all sub tests. (otherwise, generating visits for each test case would make tests run very slow).

In UI, important to test alert name has ', " for sql, and check chinese characters.

comment:4 Changed 4 years ago by matt (mattab)

(In [2585]) * Small modifications to UI

  • created a Filter/AddColumnsProcessedMetrics.php that can be reused in the Custom Alerts plugin refs #1486 - it processes the 3 default processed variables (actions per visit, avg time on site, bounce rate) for any datatable that has a 'dimension' in the returned metadata :)

comment:5 Changed 4 years ago by matt (mattab)

(In [2594]) Refs #1485

  • Fixing wrong metadata for all plugins, cleaning up columns and ensure consistency in API (eg. VisitsSummary.get now returns conversion rate with %)
  • Adding a generic filter Truncate, updated doc at: http://dev.piwik.org/trac/wiki/API/Reference filter_truncate ; if set, will truncate the table after $filter_truncate rows. The last row will be named 'Others' (localized in the requested language) and the columns will be an aggregate of statistics of all truncated rows.
  • Adding common helpers to get a "pretty" value from a given metric and column name
  • Adding new time export format that will be used in PDF reports 12:34:56 format

Refs #1486

  • a new filter, AddColumnsProcessedMetricsGoal, can be reused to easily add processed goal metrics for a given datatable. For example:

$dataTable->filter('AddColumnsProcessedMetricsGoal', array($enable=true, Piwik_DataTable_Filter_AddColumnsProcessedMetricsGoal::GOALS_FULL_TABLE));
will process all goal metrics (and per goal conv rate, rev per visit, conversions).
When possible, only reports necessary should be loaded using GOALS_MINIMAL_REPORT or GOALS_OVERVIEW or a given idGoal

comment:6 Changed 4 years ago by matt (mattab)

(In [2701]) Refs #1486
Patch by halfdan required by Alerts plugin

  • Invert pattern match
  • Date.addPeriod and Date.subPeriod

comment:7 Changed 4 years ago by halfdan

I'm uploading a current status here - not quite working, more for code review and feedback. I'm half-way through migrating to the new UI that was introduced with PDFReports.

Integration.test is testing addAlert/editAlert (success!) and processing of alerts (little buggy).

I'll be travelling for some days and will finish Alerts when I'm back.

Changed 4 years ago by halfdan

Current status - not for production

comment:8 Changed 4 years ago by matt (mattab)

  • Milestone changed from 1 - Piwik 0.7 - DigitalVibes to Features requests - after Piwik 1.0

delaying for now, even though I hope it makes the cut :)

comment:9 Changed 3 years ago by matt (mattab)

Halfdan, do you think the plugin can be put into trunk at some point? This would be a great feature and it is more than half done for sure :)

comment:10 Changed 3 years ago by matt (mattab)

  • Milestone changed from Feature requests to 1.x - Piwik 1.x

comment:12 Changed 21 months ago by matt (mattab)

@halfdan Have you had a chance to make progress on the feature? Even if not complete we'd be very interested to see what you've done and if we can reuse it. Thanks!

comment:13 Changed 17 months ago by Codalibre

Hi, I would be very interested to help or to continue this project. @halfdan, it would be very nice if you could report what is working and what not, and which kind of problems you encountered.

comment:14 Changed 17 months ago by halfdan

Hi @Codalibre,
that is very kind of you. I am however still working on this plugin and hope to have it finished by the end of the year. I have not responded to matt's request since I am in direct contact with him.

I will let you know if I need any help.

comment:15 Changed 14 months ago by matt (mattab)

If you are interested in this feature, please pledge now! We are running a campaign for Custom Alerts: http://crowdfunding.piwik.org/custom-alerts-plugin/

It will be awesome, but we need your support!

comment:16 Changed 14 months ago by matt (mattab)

  • Milestone changed from 1.x - Piwik 1.x to 1.12.x - Piwik 1.12.x

comment:17 Changed 14 months ago by matt (mattab)

  • Milestone changed from 1.12.x - Piwik 1.12.x to 1.12 - Piwik 1.12

comment:18 Changed 13 months ago by mohelten

I'd love to see another feature.
Piwik could become a helpful tool in order to be alerted when it comes to problems with the website.

There are a lot of things that can go wrong after updating your website. You have a lot of log files and tests but nevertheless sometimes there goes something wrong on the website (e.g. browser problems with js, layout problems, ...).

Sure, you could have an alert for higher bounce rates on critical pages (e.g. product pages).

It would be great if there were alerts that are a bit more complex.
My scenario:
Alert if the ratio of visitors that view a product page, add a product to cart and checkout decreases.
This means the checkout funnel should alert me if there is a problem in one of the steps.

Last edited 13 months ago by mohelten (previous) (diff)

comment:19 Changed 13 months ago by matt (mattab)

@mohelten Piwik does not yet support conversion funnel, so this is not possible, but it's a good idea for the future.

comment:20 Changed 13 months ago by matt (mattab)

  • Milestone changed from 1.12 - The Great 1.x Backlog to 2.x - Piwik 2.x

comment:21 Changed 8 months ago by matt (mattab)

  • Milestone changed from 2.x - Piwik 2.x to 2.0 - Piwik 2.0

comment:22 Changed 6 months ago by matt (mattab)

I got a few emails in the last few weeks from funders asking about Custom Alerts.

We have some news: while unfortunately Fabian @halfdan is right now in hospital recovering from accident but will be out soon. AND he has a version of the plugin for beta which we'll release as part of Piwik 2.0 as a core plugin, as announced. Stay tuned & halfdan hope you get well!

comment:23 Changed 5 months ago by matt (mattab)

@Fabian any update for our community of funders eagerly waiting to hear an update? hope you are well!

comment:24 Changed 5 months ago by matt (mattab)

  • Priority changed from major to critical

comment:25 Changed 5 months ago by matt (mattab)

hey @Fabian any update?

comment:26 Changed 4 months ago by matt (mattab)

  • Milestone changed from 2.0 - Piwik 2.0 to 2.1 - Piwik 2.1

No update yet, so likely won't make it to 2.0 (sorry for such long wait to all funders!)

comment:27 Changed 4 months ago by tsteur

  • Owner changed from halfdan to tsteur
  • Status changed from new to assigned

comment:28 Changed 4 months ago by tsteur

In f9ff7412c7fe8deec56f41a1b54f3409a72d1d09:

refs #1486 made previous alerts plugin compatible with Piwik 2.0 and added the possibility to actually list, create and edit alerts

comment:29 Changed 4 months ago by tsteur

In 1e6cece49d0838e6518bb8ce2c9077a9c0555dfe:

refs #1486 code cleanup, implemented some comparisons, bugfixes

comment:30 Changed 4 months ago by tsteur

In 3bd9c0b2d09a0654540a1f86601416bb6074bdbe:

refs #1486 we do not want to have sql statements in controller, api and plugin class. Put them in a model, makes it also better testable

comment:31 Changed 4 months ago by tsteur

In 450e861b69b3dafff24f10f446dd41932270bf31:

refs #1486 some more cleanup, bugfixes and moved defined metrics/group conditions to the processor as only this class can know which ones it supports

comment:32 Changed 4 months ago by tsteur

In 853b18cd1a5722650f18189e26afbea13bb20d92:

refs #1486 more refactoring, bugfixing, display report name is list of all alerts

comment:33 Changed 4 months ago by matt (mattab)

In 8d02c09125bdfb308bee630687c0adf730b9c241:

Deleting custom alerts plugin for 2.0.2 release refs #1486

comment:34 Changed 4 months ago by matt (mattab)

@tsteur

  • nice progress!
  • please "refs #1486" when making changes to Custom Alerts so we get nice history in here
  • On: https://travis-ci.org/piwik/piwik-ui-tests#L110 There is the following error:
    PHP Strict standards: Declaration of Piwik\Plugins\CustomAlerts\tests\CustomProcessor::getMetricFromTable() should be compatible with Piwik\Plugins\CustomAlerts\Processor::getMetricFromTable($dataTable, $metric, $filterCond = '', $filterValue = '') in /home/travis/build/piwik/piwik/plugins/CustomAlerts/tests/ProcessorTest.php on line 15
    

comment:35 Changed 4 months ago by tsteur

I stopped using refs once we created the CustomAlerts repository. We have a nice history just for this plugin there and the refs are not being displayed here anyway. I think only for Piwik repository. We changed this recently with the Git problems and I think we even did not process all repositories before.

comment:36 Changed 4 months ago by matt (mattab)

The git problems should now be fixed, now that git was upgraded on the server.

What would it take to process all repositories again and have trac Refs work on any github.com/piwik/* repository?

  • This would be nice as it brings openness and awareness of our work. Also it will help a lot users who come here from the changelog and like to see detailed list of changes.

comment:37 Changed 4 months ago by tsteur

Trac config ini needs to be adjusted for this. To work for all repositories automatically won't work

comment:38 Changed 4 months ago by tsteur

In c2d80690b16dc937747e0d3ba1c92f5718b57ef4:

refs #1486 make it possible to reuse only phoneNumbers and emails without the html for graph type

comment:39 Changed 4 months ago by tsteur

In 87696120917c4b21bbaa9352c6299974f362851f:

refs #1486 fix link to MobileMessage settings works only if current action is index

comment:40 Changed 4 months ago by tsteur

In 8b76960d3575d8fa98bd5c608a68893afab75e79:

refs #1486 move script tag into td to make sure nth-child(2n) css selector works. Otherwise sometimes two rows have same backgroundcolor in an entitytable

comment:41 Changed 4 months ago by tsteur

In e13682cfbaaa14ecd1d5807eb6687f0b55278498:

refs #1486 fix selected phone numbers are not recoginzed

comment:42 Changed 4 months ago by tsteur

In c18171d4c4c15a3106c4ed5865541a8f8bfc014a:

refs #1486 trigger an event when a phone number is deleted

comment:43 Changed 4 months ago by tsteur

In 5773706aa201b003ad10790de4d67b2709e802fa:

refs #1486 added some methods needed in CustomAlerts plugin to validate report and metric is still valid

comment:44 Changed 4 months ago by tsteur

In 329369d5117c082377d75dd6d10ed3e61241c495:

refs #1486 fix website id is not updated but website name if someone switches the website in website selector

comment:45 Changed 4 months ago by tsteur

In 32f40ab9525256f1f7654d54f3307a4e9e1bc7c4:

refs #1486 added method to detect whether current year is a leap year

comment:46 Changed 4 months ago by tsteur

In a9d8fc6d48e6c823c501dd834170d6d175b4b818:

refs #1486 added method to create a scheduled time depending on the websites timezone

comment:47 Changed 4 months ago by tsteur

In 66bc3b1d60e2e995781dcfb9fe25b0ef81f9132e:

refs #1486 if AllWebsites was selected in the site selector, the value was of the site selector was never updated. Also do not toggle the name of the current selected website with the clicked website as it is kinda confusing and leads to website names appearing multiple times

comment:48 Changed 3 months ago by tsteur

In 3664f48d1d0f523f572c038acb15868701e6f16c:

refs #1486 added an anchor so we can directly link to a specific plugin

comment:49 Changed 3 months ago by tsteur

In 15853da8b03f0ab7a858a2335e9dbd62b7db4116:

refs #1486 execute applyQueuedFilters only if not disabled, prevents an issue when filter_truncate and flat=1 is used in Actions.getPageUrls: Trying to add two strings values in DataTable\Row::sumRowArray

comment:50 Changed 3 months ago by matt (mattab)

  • Milestone changed from 2.1 - Piwik 2.1 to 2.0.4 - Piwik 2.0.4

comment:51 Changed 3 months ago by tsteur

In 04ac332bde0a9f3629722aa4cf31e12f031afda7:

refs #1486 updated submodule where we now store the reportUniqueId instead of module.action to support all reports (goals etc)

comment:52 Changed 3 months ago by tsteur

In 4b10f3953498ce8c77a073816f9e4f5c0a82721c:

refs #1486 load fixtures from plugins when running UI tests

comment:53 Changed 3 months ago by tsteur

In 961f0e95553a3cf4afcc5e9ecec86579a87be980:

refs #1486 do not apply queued filters when they are disabled

comment:54 Changed 3 months ago by tsteur

In 9dbe600d619736e6e8136c9f047eb100fde6e1d5:

refs #1486 in case we do not apply the filters directly, keep them in the new data table. Otherwise we cannot apply them later in the customAlerts plugin manually

comment:55 Changed 3 months ago by matt (mattab)

  • Description modified (diff)

comment:56 Changed 3 months ago by matt (mattab)

  • Description modified (diff)

comment:57 Changed 3 months ago by tsteur

In 711456dea7e892b7f0adfb8e87a03d49b2ada310:

refs #1486 this should fix the scheduled alerts for month or week run one day too late under circumstances.

Note: See TracTickets for help on using tickets.