Opened 7 months ago

Closed 6 weeks ago

#4189 closed New feature (fixed)

UI unit tests

Reported by: capedfuzz Owned by: capedfuzz
Priority: major Milestone: 2.3.0 - Piwik 2.3.0
Component: Core Keywords:
Cc: Sensitive: no


We should increase our JavaScript code coverage by adding unit tests for individual UI controls. To do this, we need to:

  • Allow JavaScript to be injected before taking screenshots.
  • Create unit tests of all individual UI widgets (eg, site selector, datatable) that compare screenshots after triggering certain events.

Change History (22)

comment:1 Changed 6 months ago by matt (mattab)

  • Milestone set to 2.0 - Piwik 2.0

comment:2 Changed 6 months ago by matt (mattab)

  • Owner set to diosmosis

comment:3 Changed 5 months ago by matt (mattab)

In 5e644be468448be0707e5e23c0ef2103f38dfc63:

Refs #4087 #4189 We would like ui tests build to run every commit even when the Integration tests fail. This will help know exactly which commit introduced a regression.

comment:4 Changed 4 months ago by capedfuzz (diosmosis)

In a68ed87a60f246e97073c202413a3ce0575dce7b:

Refs #4189, use new Omni test fixture for UI tests that combines all other fixtures.

comment:5 Changed 4 months ago by matt (mattab)

@capedfuzz the omni fixture broke the UI tests:

comment:6 Changed 4 months ago by matt (mattab)

  • Owner changed from diosmosis to capedfuzz

comment:7 Changed 3 months ago by capedfuzz (diosmosis)

In be9b410ac5d0947447097f175b93114f15662e3a:

Refs #4189, added UI unit test base class and tests for ViewDataTable. Refactored capture.js to make new type of testing possible.

comment:8 Changed 3 months ago by capedfuzz (diosmosis)

In 84a22be56b65aa7bed4ae3cce3c447b49238cbad:

Refs #4189, add segment selector/editor unit tests & tweak CSS so tests look right.

comment:9 Changed 3 months ago by matt (mattab)

  • Priority changed from normal to major

comment:10 Changed 8 weeks ago by capedfuzz (diosmosis)

In f4e39c9a87e38ee17f1b2ddaf0136543e42a47cc:

Refs #4739, #4189, remove persist_fixture_data config option code, placed OmniFixture into a namespace, removed PHP screenshot testcase classes and added fixture for use w/ JavaScript screenshot tests.

comment:11 Changed 8 weeks ago by capedfuzz (diosmosis)

In 7c7a8776f526d274fae687d7c74fa682c6a527c5:

Refs #4189, add Annotations.deleteAll API method, work around popover issue in phantomjs (modifying window.location.href directly erases JavaScript state for page even if the URL is the same) and tweak user creation in test fixture setup.

comment:12 Changed 8 weeks ago by capedfuzz (diosmosis)

In 13a08d58897ba2921a037b309e75593c3fbbabcd:

Refs #4189, added expected screenshot for DBStats, added actual & mock data access class to DBStats, fix bug that occurs when deleting merged assets that do not exist and fix diffviewer generation regression (expected URL was incorrect).

comment:13 Changed 8 weeks ago by capedfuzz (diosmosis)

In 716106915c1407485aac66dd34747492230d68e9:

Refs #4189, add action to render site selector, add more sites for UI tests for site selector and make site selector has a margin top in UI tests.

comment:14 Changed 7 weeks ago by capedfuzz (diosmosis)

In e1d338cf179de9146b94f4fb79c918e264ef9116:

Refs #4189, add controller method to CoreHome to render period selector, add periodSelector CSS class to period selector.

comment:15 Changed 7 weeks ago by capedfuzz (diosmosis)

In 67d34ebc9eef1fdcb8d5676bb5302f067b465576:

Refs #4189, allow widget factory to be included in getDashboardSettingsControl() action output for tests, make sure last dashboard always overwritten by UI test start.

comment:16 Changed 7 weeks ago by capedfuzz (diosmosis)

In 57bf5d0c1452e56621b1ff266fdf7154c5321040:

Refs #4189, bug fixes for dashboard & changes to make dashboard UI tests possible. Includes:

  • fix for bug where top controls not displayed correctly when dashboard is displayed in a widget iframe.
  • do not display dashboard manager after dashboard action selected.
  • truncate tables when running UI tests w/ --persist-fixture-data so existing database data won't be used if the fixture setup didn't complete before.
  • do not create super user more than once in Fixture.
  • don't include Test_Piwik_Fixture_CustomAlerts in omni fixture.
  • make sure UITestFixture executes the correct code every time regardless of whether --persist-fixture-data isn't used.
  • add mouseup/mousedown events to screenshot testing page rendere.
  • add ability to call controller methods in screenshot tests and fix API calling mechanism.

comment:17 Changed 7 weeks ago by capedfuzz (diosmosis)

In 8b48ff24b96d411d5c18eba6a6ad714f3af7908f:

Refs #4189, allow screenshot tests to change viewport size.

comment:18 Changed 7 weeks ago by capedfuzz (diosmosis)

In c8d8818e9dd3ed209f579e6fea4d1972a8a69521:

Refs #4189, allow default config file paths to be overridden for tests, rewrite proxy/index.php to use hooks for config instead of disabling dispatch.php and remove no longer used config override code.

comment:19 Changed 7 weeks ago by capedfuzz (diosmosis)

In 8a525ad008a5235744de7f6c922516a36d45c314:

Refs #4189, more work getting UI tests to pass and adding admin screenshots, including:

  • make updatetoken.php use TestingEnvironment in testmode,
  • allow persisted fixture to be dropped using new --drop command line option (for ui test runner),
  • fix debugging mistake in realtime map, fix merge error in empty.twig,
  • make sure Fixture prints to screen only for UI test setup, re-enable CustomAlerts fixture in OmniFixture,
  • make sure forced now value for realtime map is constant,
  • use *.getDatabaseConfig events to override database name in TestingEnvironment
  • use TestingEnvironment in proxy/piwik.php
  • add wait() method to screenshot testing framework's PageRenderer
  • hide current UTC time in manage websites admin screenshot

comment:20 Changed 7 weeks ago by capedfuzz (diosmosis)

In 5c66663d11aec26579b7c66928695f03ddb11dff:

Refs #4189, adding Overlay screenshot tests:

  • Make 'idsite' parameter in certain Overlay URLs 'idSite'.
  • Make sure Overlay sidebar has width so screenshot test framework can find position.
  • Add overlay test site so Overlay can be loaded in phantomjs.
  • Remove createSuperUser call from TrackerTest (done in Fixture setUp).
  • Make sure piwik.js link is created for screenshot tests & make sure the test environment JSON file is loaded after the database is setup.
  • Modify logic that waits for images to load in screenshot testing framework's page renderer to wait for CSS image URLs.

comment:21 Changed 6 weeks ago by matt (mattab)

UI Unit Tests Project Summary:

  • Testing framework for UI testing using mocha & chai, both cutting edge libraries.
    • Framework increases the speed of running tests. Even with more than double the amount of tests (from 82 -> 181), the new tests run in less time (5 min less time).
    • Ability to perform benchmarking of UI code, both page load, AJAX requests and individual JavaScript actions.
    • Framework can be used to write other UI tests that run in the browser (such as tests for the AngularJS code).
    • Plugins can now have their own UI tests.
    • Writing UI tests is easier & w/ included README creating tests will be easier for other team members to start writing them.
  • Refactoring of Fixture/IntegrationTest setup code.
    • Opens up the possibility more easily re-introducing benchmarks.
    • Consistent use of TestingEnvironment so testing code is in one place.
  • Significantly increased code coverage, including tests to following logic:
    • Overlay
    • Row Evolution + Annotations (not just load, but switching series, create/update annotations, etc.)
    • Dashboard + Dashboard Manager
    • Site Selector
    • Pie Graph / Bar Graph / Evolution Graph / Actions Data Table / Goals Table
    • Login + Reset Password
    • + Others including DBStats + No Config message + other normal & admin pages, etc.

Intended Outcomes

  • Reduced time to release.
  • Reduced regressions.
  • Reduced time to finished feature.
  • Reduced time to refactor.
  • Increased developer involvement in QA.

Next Steps:

  • We should try and measure the effectiveness of UI unit tests by:
    • Counting all regressions & bugs that appear in 2.1.0 release & recording the amount of days between 2.1.0's intended release day and actual release day. This is the before stats.
    • Counting all regressions & bugs that appear in 2.2.0 release & recording how long it takes to go from 'done' to 'release'.
  • There is currently a large list of UI tests that must be manually done before a list. This entire list could be translated into screenshot tests.

Room for improvement:

  • waiting for one build to finish before starting the UI tests build is a problem
  • debugging travis is still a problem
  • could add benchmarking stats for AJAX (overall & per specific API/Controller method)
  • TestingEnvironment integration can be improved (ie, move Tracker::setTestEnvironment to file)
  • the screenshot testing lib could use an async action JS lib since calling multiple API methods in before()/after() is used
  • when new fixtures are added, the new data is used in the screenshot tests, changing the screenshot. not an issue for us, but potentially one for plugin developers.
  • allowing plugins to add their own PHP code to the UITestFixture isn't possible. can be done w/ an event.
  • to make it exactly like manual UI testing, we can use real data for the tests (ie, from the data could be in an SQL dump we'd modify when making new test cases. this way integration tests can test tracker/aggregation/API, and UI tests can test just the UI.

comment:22 Changed 6 weeks ago by capedfuzz (diosmosis)

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.