Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write Installer/Updater screenshot tests #2935

Closed
robocoder opened this issue Feb 16, 2012 · 17 comments
Closed

Write Installer/Updater screenshot tests #2935

robocoder opened this issue Feb 16, 2012 · 17 comments
Assignees
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Milestone

Comments

@robocoder
Copy link
Contributor

The last big missing functionnality not yet tested is our Installation process, as well as the update process.

Let's add screenshot tests for:

  • Installer process
  • Upgrade to GIT repo from 1.0

The result will be that we always know whether installation + update to latest Piwik work. This will result in more agility and ability to release new versions faster.

@robocoder
Copy link
Contributor Author

(In [5866]) refs #2935 - remove broken webtests =P

@robocoder
Copy link
Contributor Author

(In [5992]) refs #2935 - re-enable JavaScript (hack)

@robocoder
Copy link
Contributor Author

(In [5993]) refs #2935 - add delay for form to auto-post

@robocoder
Copy link
Contributor Author

(In [6007]) refs #2935 - simulate auto-post form

@mattab
Copy link
Member

mattab commented Jul 27, 2012

It is really critical for our QA process to re-enable the Webtests on Jenkins.

@anthon would you have an estimate for this task? is it half day or 2 days work? :)

@robocoder
Copy link
Contributor Author

Much more than 2, I suspect -- a lot of web tests were disabled.

Moreover, the ticket involves rewriting all the web tests.

We would switch to either PHPUnit+Mink or Behat+Mink. Using Mink will give us more driver choices, eg Selenium (or Selenium2) for pages with JavaScript (eg testing software update), and non-Selenium drivers for non-JavaScript pages (eg login).

@robocoder
Copy link
Contributor Author

(oh another option is phpspec + mink)

@mattab
Copy link
Member

mattab commented Jul 29, 2012

It sounds really hard and might never happen.. :(

Maybe we could find a workaround to re-enable at least the install/update tests which were an incredibly useful part of our QA toolset... ?

Is there a way to make the existing tests work again using selenium? I remember the problem was about Javascript files/Jquery not being executed properly to click on some buttons?

@robocoder
Copy link
Contributor Author

the javascript engine used by HtmlUnit couldn't handle the dynamic changes to the DOM. I'm subscribed to the webtest mailing list and we are not the only ones feeling the pain.

It might be nice to use Behat+Mink. I've already done work here to parallelize Selenium testing.

@mattab
Copy link
Member

mattab commented Oct 16, 2012

Webtests are not that critically important, compared to getting Jenkins back to working. Decreasing priority...

Anthon, do you reckon I should create a ticket for Jenkins setup?

@mattab
Copy link
Member

mattab commented Dec 13, 2012

Many times our sql update code wasn't failure-friendly and would fail when re-executed as it left the DB in a half-updated state.

Having the following test would ensure that all future update scripts written are working in a failure scenario.

A new idea for a webtest:

Download 1.8.3 release
Download latest
Run Auto update - but make sure the auto update code fails at the last step
Re-run auto update: the code should still run and the SQL not fail. For this, some queries will throw exceptions which must be specifically ignored, eg. "1060 Duplicate column name 'xx'" can be ignored for an ALTER that adds a column.
The update re-running after initial failure should succeed. 

@mattab
Copy link
Member

mattab commented Feb 21, 2013

I've been told we should use Webdriver eg. https://github.com/facebook/php-webdriver

and we could run the test on Travis CI as well.

If anyone is fluent in webdriver/selenium, please contact me at matt@piwik.org :)

@mattab
Copy link
Member

mattab commented Feb 1, 2014

Consolidating milestones FTW

@diosmosis
Copy link
Member

In 50742af: Refs #2935, add installation UI test, allow no fixture to be specified for screenshot test and fix following installation process regressions:

  • pending events should be executed after all plugins successfully loaded so plugin order in config file is unimportant
  • do not set piwikUrl on View instances if the DB cannot be connected to
  • create Twig instance when clearing template cache instead of new View

@diosmosis
Copy link
Member

In f05c080: Refs #2935, fixing Installation UI test.

@mattab
Copy link
Member

mattab commented Apr 1, 2014

@capedfuzz well done on adding those tests. Please close the ticket if it's fixed :)

@mattab
Copy link
Member

mattab commented Apr 1, 2014

Feedback:

@robocoder robocoder added this to the 2.2.1 - Piwik 2.2.1 milestone Jul 8, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…e specified for screenshot test and fix following installation process regressions:

- pending events should be executed after all plugins successfully loaded so plugin order in config file is unimportant
- do not set piwikUrl on View instances if the DB cannot be connected to
- create Twig instance when clearing template cache instead of new View
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
@mattab mattab added the c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. label Oct 12, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Projects
None yet
Development

No branches or pull requests

3 participants