Opened 5 years ago

Closed 3 years ago

Last modified 3 years ago

#572 closed New feature (fixed)

Core: Provide custom date range feature

Reported by: matt Owned by: matt
Priority: critical Milestone: Piwik 1.3
Component: Core Keywords:
Cc: Sensitive: no

Description (last modified by matt)

Piwik provides reports for days, weeks, month and year, but not for custom date ranges.

The mechanism used to process reports for weeks, month and year use a generic code that could be used to generate reports on any custom date range.

List of changes to make this happen:

  • the date range should be set in the "date" parameter and not in a new parameter date_range, and "period" parameter would be "range"
  • Archiving: when period=range, the Archiving code will test if an archive exists already. If it doesn't, it will look at the most efficient way to process the range: the minimum number of sub date ranges.
    • For example, requesting data from Jan 1st to March 15th 2009:
      • we can sum jan month + feb + sum of the first 15 days of july? which is 17 archives to select and sum
      • but if we minimize the nb of date ranges in the requested range, by doing: jan + feb + day 1st of march + week from 2nd to 8th + week from 9th to 15th, we only select & sum 5 archives which will be a lot more efficient and better user experience.

The archive sum of all sub archives will then be recorded and read as normal.

  • UI: period selector will change to use a calendar that allows for date range selector (example). We could add "Date range" after "day, month, year" buttons, when clicked would show 2 calendars.
  • UI: maybe it would be nice to show date ranges shortcut, "last 7 days", "last 30 days" somehow ?
  • QA: add new integration tests that test with the date range "date=2011-01-01,2011-03-22&period=range"
  • Note: none of the plugins code for Archiving, API querying etc. will need to be modified, since all the logic is hidden

Any ideas? Any volunteer to work on this interesting challenge? Please comment :)

Change History (35)

comment:1 Changed 5 years ago by matt (mattab)

  • Priority changed from major to critical

comment:2 Changed 5 years ago by matt (mattab)

  • Summary changed from Custom date range to Core: Provide custom date range feature

comment:3 Changed 5 years ago by kolchak

Think this is a must as a core feature, and was the first thing one of my users asked :)

comment:4 Changed 4 years ago by yan

I think this is a must-have. Amazing that there is so little request for this feauture.

comment:5 Changed 4 years ago by amnigos

Agree, this is a must have feature.

comment:6 Changed 4 years ago by greg (gka)

I would suggest a different spec than mentioned in the ticket desc:

  • Instead of introducing a new "custom" period, I would keep the periods day, week, month and year.
  • There are two modes: the first is the current behavior where there is a parameter "date" and the exact date range is selected automatically, (either the day itself or the week/month/year that contains this date). The second mode (the new one) uses the parameters fromDate and toDate instead of just date. This will define the custom date range.
  • The selected period now defines how to sum up the data, if the range is from Jan 1st until March 15th and the period is week, the weekly values are summed up.
  • If the user selects a date range that would require much computing time (eg. period: day and range: Jan 2009 until Jul 2010), Piwik will display a warning before the computation starts..

comment:7 Changed 4 years ago by matt (mattab)

  • Sensitive unset

See also 'Week beginning setting' #1633

comment:8 Changed 3 years ago by matt (mattab)

When this is implemented, we can improve user settings, and allow to set the default date to "Last 7 days", "Last 30 days", etc.

comment:9 Changed 3 years ago by mrisher

I agree with @greg's suggestion of implementation. Has anyone begun on this? If not, I can look into it, but I don't want to step on someone else's half-completed implementation.

Thank you.

comment:10 Changed 3 years ago by matt (mattab)

mrisher, work hasn't started on this so feel free to pick it up. However, I think the proposed implementation should be followed (Piwik chooses which is the most efficient way of archiving custom date ranges, selecting weeks and/or months when this is faster).

comment:11 Changed 3 years ago by mrisher

Hi, Matt:
Unfortunately, our project needs have shifted a bit and so I don't think that we'll be working on this feature in the short term.

I wanted to update the comment so nobody else was scared off.

Sorry,
--Mark

comment:12 Changed 3 years ago by code-kobold

I'm working on this preferring a last N Days range since I do not need a in between Dates range. Hope to come up with this soon.

Regards, Ron

comment:13 Changed 3 years ago by matt (mattab)

  • Description modified (diff)

comment:14 Changed 3 years ago by matt (mattab)

  • Description modified (diff)

comment:15 Changed 3 years ago by matt (mattab)

(In [4159]) Refs #572

  • Implementing Custom Date range in Core, eg "period=range&date=2010-01-05,2010-02-03"
  • Only the requested plugin's report will be processed, ensuring fast responses
  • when a large period is requested, the code will try and select an optimal number of sub periods (see Range.test.php) - a mix of weeks, months and days inside the requested range
  • adding integration testing + unit test for optimal subperiods algorithm

TODO

  • we should also use Yearly archives whenever possible (when date range is very large)
  • if custom date range start date == website creation date or before, can use more optimal subperiods (using starting month / year / week)
  • Test/implement "last N days"
  • Test with timezones when the custom date range ends today
  • UI to allow selecting custom range (show 2 calendars)
  • Update API reference doc + add example in API page listing
  • Add "last 7 days", "last 30 days" in the user settings for "Default date to load"


NB: 3 tests fail in Main.test.php on my box, but I'm really not sure why... let's see if Jenkins fails as well

comment:16 Changed 3 years ago by matt (mattab)

  • Milestone changed from Feature requests to 1.3 - Piwik 1.3
  • Owner set to matt

comment:17 Changed 3 years ago by matt (mattab)

(In [4170]) Refs #572 Fixing bug when the range goes in the future of current month, forcing to stop at the specified end date

comment:18 Changed 3 years ago by matt (mattab)

(In [4171]) Refs #572 Health checks

comment:19 Changed 3 years ago by matt (mattab)

Note: update FAQ: http://piwik.org/faq/how-to/#faq_33 when feature released

comment:20 Changed 3 years ago by matt (mattab)

(In [4174]) Refs #572 Fixing known archiving issues when period=range

comment:21 Changed 3 years ago by matt (mattab)

(In [4188]) Refs #572

  • Adding Date Range calendar UI, with "Apply Date Range" button. Possibility to edit the INPUT fields directly rather than clicking in the calendar. Testing that dates are valid (from < to, valid string).
  • Updating calendar to show Loading.. on click (hopefully it makes it less confusing)
  • Fixing few bugs when period=range
  • All reports should load correctly when period=range, including sparklines, graphs & standard tables
  • Refs #2145 renaming parameter to $filter_limit which seems to work OK - it's not even a hack isn't it? :)

TODO:

  • refactor period_select.tpl JS in helper,
  • Metadata compatibility with period=range (at least the World map doesn't work for now)
  • previous TODO still valid

Testing is welcome, please report any bug you find!

comment:22 Changed 3 years ago by matt (mattab)

(In [4189]) Refs #572

  • Metadata supports period=range
  • Support for period=range&date=lastN

comment:23 Changed 3 years ago by matt (mattab)

(In [4190]) Adding unit tests metadata & range & last7 refs #572

comment:24 Changed 3 years ago by matt (mattab)

(In [4192]) Refactor JS & css + i18n Refs #572

comment:25 Changed 3 years ago by matt (mattab)

I'm thinking it would be nice to show, in the calendar, "Last 7 days", "Last 30 days", etc. but I don't know how to show these without making the calendar look complicated/ugly. Any idea?

comment:26 Changed 3 years ago by matt (mattab)

(In [4193]) Fixing broken admin UI (Anthon, could we add a simple webtest request that checks no JS errors?) refs #572

comment:27 Changed 3 years ago by vipsoft (robocoder)

Checking all the screens and widgets for errors is on the todo list. It will have to wait until we get Selenium tests as there are some blocking webtest bugs.

comment:28 Changed 3 years ago by matt (mattab)

(In [4207]) Refs #572 All Websites dashboard now work when a range is selected

comment:29 Changed 3 years ago by matt (mattab)

(In [4215]) Refs #572 PDF reports now work with a Custom Date Range

comment:30 Changed 3 years ago by matt (mattab)

(In [4216]) Refs #572 Fixing Widgetize screen with Custom Date Range. All known bug fixed! RC release soon?

comment:31 Changed 3 years ago by matt (mattab)

(In [4217]) Refs #572

  • Adding Last 7 days, Last 30 days to available "default date to load Piwik"

comment:32 Changed 3 years ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from new to closed

Marking as fixed since all known issues are fixed. Please test on trunk and report any problem you have with this feature!

comment:33 Changed 3 years ago by matt (mattab)

(In [4252]) Refs #572

  • X legends appears more often for graphs in general, and in particular when a small date range selected
  • Graphs "link" now disabled for date range since the data returned is huge and the graph wasn't displaying at all on large date range
  • Fix bug click on sparkline reloading main graph wasn't working on range

comment:34 Changed 3 years ago by matt (mattab)

(In [4255]) Refs #572 Deleting "Custom Date Range" reports after 1 day, since they can be re-processed and would take up unecessary space

comment:35 Changed 3 years ago by matt (mattab)

(In [4342]) Refs #572 lastN and previousN were buggy for custom date ranges, added tests + fixed algorithm

Note: See TracTickets for help on using tickets.