Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UI unit tests #4189

Closed
diosmosis opened this issue Oct 1, 2013 · 17 comments
Closed

UI unit tests #4189

diosmosis opened this issue Oct 1, 2013 · 17 comments
Assignees
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.
Milestone

Comments

@diosmosis
Copy link
Member

We should increase our JavaScript code coverage by adding unit tests for individual UI controls. To do this, we need to:

  • Allow JavaScript to be injected before taking screenshots.
  • Create unit tests of all individual UI widgets (eg, site selector, datatable) that compare screenshots after triggering certain events.
@mattab
Copy link
Member

mattab commented Nov 22, 2013

In 5e644be: Refs #4087 #4189 We would like ui tests build to run every commit even when the Integration tests fail. This will help know exactly which commit introduced a regression.

@diosmosis
Copy link
Member Author

In a68ed87: Refs #4189, use new Omni test fixture for UI tests that combines all other fixtures.

@mattab
Copy link
Member

mattab commented Dec 26, 2013

@capedfuzz the omni fixture broke the UI tests: https://travis-ci.org/piwik/piwik-ui-tests

@diosmosis
Copy link
Member Author

In be9b410: Refs #4189, added UI unit test base class and tests for ViewDataTable. Refactored capture.js to make new type of testing possible.

@diosmosis
Copy link
Member Author

In 84a22be: Refs #4189, add segment selector/editor unit tests & tweak CSS so tests look right.

@diosmosis
Copy link
Member Author

In f4e39c9: Refs #4739, #4189, remove persist_fixture_data config option code, placed OmniFixture into a namespace, removed PHP screenshot testcase classes and added fixture for use w/ JavaScript screenshot tests.

@diosmosis
Copy link
Member Author

In 7c7a877: Refs #4189, add Annotations.deleteAll API method, work around popover issue in phantomjs (modifying window.location.href directly erases JavaScript state for page even if the URL is the same) and tweak user creation in test fixture setup.

@diosmosis
Copy link
Member Author

In 13a08d5: Refs #4189, added expected screenshot for DBStats, added actual & mock data access class to DBStats, fix bug that occurs when deleting merged assets that do not exist and fix diffviewer generation regression (expected URL was incorrect).

@diosmosis
Copy link
Member Author

In 7161069: Refs #4189, add action to render site selector, add more sites for UI tests for site selector and make site selector has a margin top in UI tests.

@diosmosis
Copy link
Member Author

In e1d338c: Refs #4189, add controller method to CoreHome to render period selector, add periodSelector CSS class to period selector.

@diosmosis
Copy link
Member Author

In 67d34eb: Refs #4189, allow widget factory to be included in getDashboardSettingsControl() action output for tests, make sure last dashboard always overwritten by UI test start.

@diosmosis
Copy link
Member Author

In 57bf5d0: Refs #4189, bug fixes for dashboard & changes to make dashboard UI tests possible. Includes:

  • fix for bug where top controls not displayed correctly when dashboard is displayed in a widget iframe.
  • do not display dashboard manager after dashboard action selected.
  • truncate tables when running UI tests w/ --persist-fixture-data so existing database data won't be used if the fixture setup didn't complete before.
  • do not create super user more than once in Fixture.
  • don't include Test_Piwik_Fixture_CustomAlerts in omni fixture.
  • make sure UITestFixture executes the correct code every time regardless of whether --persist-fixture-data isn't used.
  • add mouseup/mousedown events to screenshot testing page rendere.
  • add ability to call controller methods in screenshot tests and fix API calling mechanism.

@diosmosis
Copy link
Member Author

In 8b48ff2: Refs #4189, allow screenshot tests to change viewport size.

@diosmosis
Copy link
Member Author

In c8d8818: Refs #4189, allow default config file paths to be overridden for tests, rewrite proxy/index.php to use hooks for config instead of disabling dispatch.php and remove no longer used config override code.

@diosmosis
Copy link
Member Author

In 8a525ad: Refs #4189, more work getting UI tests to pass and adding admin screenshots, including:

  • make updatetoken.php use TestingEnvironment in testmode,
  • allow persisted fixture to be dropped using new --drop command line option (for ui test runner),
  • fix debugging mistake in realtime map, fix merge error in empty.twig,
  • make sure Fixture prints to screen only for UI test setup, re-enable CustomAlerts fixture in OmniFixture,
  • make sure forced now value for realtime map is constant,
  • use *.getDatabaseConfig events to override database name in TestingEnvironment
  • use TestingEnvironment in proxy/piwik.php
  • add wait() method to screenshot testing framework's PageRenderer
  • hide current UTC time in manage websites admin screenshot

@diosmosis
Copy link
Member Author

In 5c66663: Refs #4189, adding Overlay screenshot tests:

  • Make 'idsite' parameter in certain Overlay URLs 'idSite'.
  • Make sure Overlay sidebar has width so screenshot test framework can find position.
  • Add overlay test site so Overlay can be loaded in phantomjs.
  • Remove createSuperUser call from TrackerTest (done in Fixture setUp).
  • Make sure piwik.js link is created for screenshot tests & make sure the test environment JSON file is loaded after the database is setup.
  • Modify logic that waits for images to load in screenshot testing framework's page renderer to wait for CSS image URLs.

@mattab
Copy link
Member

mattab commented Mar 9, 2014

UI Unit Tests Project Summary:

  • Testing framework for UI testing using mocha & chai, both cutting edge libraries.
    • Framework increases the speed of running tests. Even with more than double the amount of tests (from 82 -> 181), the new tests run in less time (5 min less time).
    • Ability to perform benchmarking of UI code, both page load, AJAX requests and individual JavaScript actions.
    • Framework can be used to write other UI tests that run in the browser (such as tests for the AngularJS code).
    • Plugins can now have their own UI tests.
    • Writing UI tests is easier & w/ included README creating tests will be easier for other team members to start writing them.
  • Refactoring of Fixture/IntegrationTest setup code.
    • Opens up the possibility more easily re-introducing benchmarks.
    • Consistent use of TestingEnvironment so testing code is in one place.
  • Significantly increased code coverage, including tests to following logic:
    • Overlay
    • Row Evolution + Annotations (not just load, but switching series, create/update annotations, etc.)
    • Dashboard + Dashboard Manager
    • Site Selector
    • Pie Graph / Bar Graph / Evolution Graph / Actions Data Table / Goals Table
    • Login + Reset Password
    • + Others including DBStats + No Config message + other normal & admin pages, etc.

Intended Outcomes

  • Reduced time to release.
  • Reduced regressions.
  • Reduced time to finished feature.
  • Reduced time to refactor.
  • Increased developer involvement in QA.

Next Steps:

  • We should try and measure the effectiveness of UI unit tests by:
    • Counting all regressions & bugs that appear in 2.1.0 release & recording the amount of days between 2.1.0's intended release day and actual release day. This is the before stats.
    • Counting all regressions & bugs that appear in 2.2.0 release & recording how long it takes to go from 'done' to 'release'.
  • There is currently a large list of UI tests that must be manually done before a list. This entire list could be translated into screenshot tests.

Room for improvement:

  • waiting for one build to finish before starting the UI tests build is a problem
  • debugging travis is still a problem
  • could add benchmarking stats for AJAX (overall & per specific API/Controller method)
  • TestingEnvironment integration can be improved (ie, move Tracker::setTestEnvironment to file)
  • the screenshot testing lib could use an async action JS lib since calling multiple API methods in before()/after() is used
  • when new fixtures are added, the new data is used in the screenshot tests, changing the screenshot. not an issue for us, but potentially one for plugin developers.
  • allowing plugins to add their own PHP code to the UITestFixture isn't possible. can be done w/ an event.
  • to make it exactly like manual UI testing, we can use real data for the tests (ie, from demo.piwik.org). the data could be in an SQL dump we'd modify when making new test cases. this way integration tests can test tracker/aggregation/API, and UI tests can test just the UI.

@diosmosis diosmosis added this to the 2.2.1 - Piwik 2.2.1 milestone Jul 8, 2014
@diosmosis diosmosis self-assigned this Jul 8, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…run every commit even when the Integration tests fail. This will help know exactly which commit introduced a regression.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…wDataTable. Refactored capture.js to make new type of testing possible.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…nfig option code, placed OmniFixture into a namespace, removed PHP screenshot testcase classes and added fixture for use w/ JavaScript screenshot tests.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…nd popover issue in phantomjs (modifying window.location.href directly erases JavaScript state for page even if the URL is the same) and tweak user creation in test fixture setup.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…tual & mock data access class to DBStats, fix bug that occurs when deleting merged assets that do not exist and fix diffviewer generation regression (expected URL was incorrect).
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…tes for UI tests for site selector and make site selector has a margin top in UI tests.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…iod selector, add periodSelector CSS class to period selector.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…oardSettingsControl() action output for tests, make sure last dashboard always overwritten by UI test start.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…oard UI tests possible. Includes:

- fix for bug where top controls not displayed correctly when dashboard is displayed in a widget iframe.
- do not display dashboard manager after dashboard action selected.
- truncate tables when running UI tests w/ --persist-fixture-data so existing database data won't be used if the fixture setup didn't complete before.
- do not create super user more than once in Fixture.
- don't include Test_Piwik_Fixture_CustomAlerts in omni fixture.
- make sure UITestFixture executes the correct code every time regardless of whether --persist-fixture-data isn't used.
- add mouseup/mousedown events to screenshot testing page rendere.
- add ability to call controller methods in screenshot tests and fix API calling mechanism.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…n for tests, rewrite proxy/index.php to use hooks for config instead of disabling dispatch.php and remove no longer used config override code.
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…dmin screenshots, including:

- make updatetoken.php use TestingEnvironment in testmode,
- allow persisted fixture to be dropped using new --drop command line option (for ui test runner),
- fix debugging mistake in realtime map, fix merge error in empty.twig,
- make sure Fixture prints to screen only for UI test setup, re-enable CustomAlerts fixture in OmniFixture,
- make sure forced now value for realtime map is constant,
- use *.getDatabaseConfig events to override database name in TestingEnvironment
- use TestingEnvironment in proxy/piwik.php
- add wait() method to screenshot testing framework's PageRenderer
- hide current UTC time in manage websites admin screenshot
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
- Make 'idsite' parameter in certain Overlay URLs 'idSite'.
- Make sure Overlay sidebar has width so screenshot test framework can find position.
- Add overlay test site so Overlay can be loaded in phantomjs.
- Remove createSuperUser call from TrackerTest (done in Fixture setUp).
- Make sure piwik.js link is created for screenshot tests & make sure the test environment JSON file is loaded after the database is setup.
- Modify logic that waits for images to load in screenshot testing framework's page renderer to wait for CSS image URLs.
@mattab mattab added the c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. label Oct 12, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.
Projects
None yet
Development

No branches or pull requests

2 participants