Opened 2 years ago

Closed 2 weeks ago

#2935 closed Task (fixed)

Write Installer/Updater screenshot tests

Reported by: vipsoft Owned by: capedfuzz
Priority: major Milestone: 2.3.0 - Piwik 2.3.0
Component: Core Keywords:
Cc: Sensitive: no

Description (last modified by matt)

The last big missing functionnality not yet tested is our Installation process, as well as the update process.

Let's add screenshot tests for:

  • Installer process
  • Upgrade to GIT repo from 1.0

The result will be that we always know whether installation + update to latest Piwik work. This will result in more agility and ability to release new versions faster.

Change History (27)

comment:1 Changed 2 years ago by matt (mattab)

  • Milestone changed from 1.8 Piwik 1.8 to 1.7.x - Piwik 1.7.1

comment:2 Changed 2 years ago by vipsoft (robocoder)

(In [5866]) refs #2935 - remove broken webtests =P

comment:3 Changed 2 years ago by vipsoft (robocoder)

(In [5992]) refs #2935 - re-enable JavaScript (hack)

comment:4 Changed 2 years ago by vipsoft (robocoder)

(In [5993]) refs #2935 - add delay for form to auto-post

comment:5 Changed 2 years ago by vipsoft (robocoder)

(In [6007]) refs #2935 - simulate auto-post form

comment:6 Changed 21 months ago by matt (mattab)

  • Milestone changed from 1.8.x - Piwik 1.8.x to 1.8.3 - Piwik 1.8.3
  • Summary changed from Replace WebTest with Selenium driver to [QA] Re-enable Webtests! Replace WebTest with Selenium driver for compat with new jquery

It is really critical for our QA process to re-enable the Webtests on Jenkins.

@Anthon would you have an estimate for this task? is it half day or 2 days work? :)

comment:7 Changed 21 months ago by vipsoft (robocoder)

Much more than 2, I suspect -- a lot of web tests were disabled.

Moreover, the ticket involves rewriting all the web tests.

We would switch to either PHPUnit+Mink or Behat+Mink. Using Mink will give us more driver choices, eg Selenium (or Selenium2) for pages with JavaScript (eg testing software update), and non-Selenium drivers for non-JavaScript pages (eg login).

comment:8 Changed 21 months ago by vipsoft (robocoder)

(oh another option is phpspec + mink)

comment:9 Changed 21 months ago by matt (mattab)

It sounds really hard and might never happen.. :(

Maybe we could find a workaround to re-enable at least the install/update tests which were an incredibly useful part of our QA toolset... ?

Is there a way to make the existing tests work again using selenium? I remember the problem was about Javascript files/Jquery not being executed properly to click on some buttons?

comment:10 Changed 21 months ago by vipsoft (robocoder)

the javascript engine used by HtmlUnit couldn't handle the dynamic changes to the DOM. I'm subscribed to the webtest mailing list and we are not the only ones feeling the pain.

It might be nice to use Behat+Mink. I've already done work here to parallelize Selenium testing.

comment:11 Changed 18 months ago by matt (mattab)

  • Milestone changed from 1.9.1 - Piwik 1.9.1 to 1.9.x - Piwik 1.9.x
  • Priority changed from critical to normal

Webtests are not that critically important, compared to getting Jenkins back to working. Decreasing priority...

Anthon, do you reckon I should create a ticket for Jenkins setup?

comment:12 Changed 16 months ago by matt (mattab)

Many times our sql update code wasn't failure-friendly and would fail when re-executed as it left the DB in a half-updated state.

Having the following test would ensure that all future update scripts written are working in a failure scenario.

A new idea for a webtest:

Download 1.8.3 release
Download latest
Run Auto update - but make sure the auto update code fails at the last step
Re-run auto update: the code should still run and the SQL not fail. For this, some queries will throw exceptions which must be specifically ignored, eg. "1060 Duplicate column name 'xx'" can be ignored for an ALTER that adds a column.
The update re-running after initial failure should succeed.

comment:13 Changed 14 months ago by matt (mattab)

  • Priority changed from normal to major

I've been told we should use Webdriver eg. https://github.com/facebook/php-webdriver

and we could run the test on Travis CI as well.

If anyone is fluent in webdriver/selenium, please contact me at matt@… :)

comment:14 Changed 13 months ago by matt (mattab)

  • Description modified (diff)
  • Milestone changed from 1.12.x - Piwik 1.12.x to 2.0 - Piwik 2.0
  • Summary changed from [QA] Re-enable Webtests! Replace WebTest with Selenium driver for compat with new jquery to Write webdriver/selenium tests for Piwik

comment:15 Changed 13 months ago by vipsoft (robocoder)

  • Owner vipsoft deleted

comment:16 Changed 8 months ago by matt (mattab)

  • Milestone changed from 2.0 - Piwik 2.0 to 2.x - Piwik 2.x

comment:17 Changed 8 months ago by matt (mattab)

  • Milestone changed from 2.x - Piwik 2.x to 2.0 - Piwik 2.0
  • Summary changed from Write webdriver/selenium tests for Piwik to Write Installer/Updater webdriver/selenium tests for Piwik

comment:18 Changed 8 months ago by matt (mattab)

  • Priority changed from major to normal

comment:19 Changed 7 months ago by matt (mattab)

  • Milestone changed from 2.0 - Piwik 2.0 to Piwik 2.1

Piwik 2.0 full focus mode: Moving normal + low priorities to Piwik 2.1 milestone.

comment:20 Changed 3 months ago by matt (mattab)

  • Milestone changed from 2.1 - Piwik 2.1 to 2.0.4 - Piwik 2.0.4

Consolidating milestones FTW

comment:21 Changed 2 months ago by matt (mattab)

  • Milestone changed from 2.1 - Piwik 2.1 to 2.2 - Piwik 2.2

Entering 2.1 RC cycle, moving all other tickets to 2.2 milestone!

comment:22 Changed 6 weeks ago by matt (mattab)

  • Description modified (diff)
  • Milestone changed from 2.2 - Piwik 2.2 to 2.1.1 - Piwik 2.1.1
  • Owner set to capedfuzz
  • Priority changed from normal to major
  • Summary changed from Write Installer/Updater webdriver/selenium tests for Piwik to Write Installer/Updater screenshot tests

comment:23 Changed 6 weeks ago by capedfuzz (diosmosis)

In 50742af6d5784d78bdf36281880c45310875a698:

Refs #2935, add installation UI test, allow no fixture to be specified for screenshot test and fix following installation process regressions:

  • pending events should be executed after all plugins successfully loaded so plugin order in config file is unimportant
  • do not set piwikUrl on View instances if the DB cannot be connected to
  • create Twig instance when clearing template cache instead of new View

comment:25 Changed 3 weeks ago by matt (mattab)

@capedfuzz well done on adding those tests. Please close the ticket if it's fixed :)

comment:26 Changed 3 weeks ago by matt (mattab)

Feedback:

comment:27 Changed 2 weeks ago by capedfuzz (diosmosis)

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.