Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Continuous Integration server improvements #1470

Closed
mattab opened this issue Jul 12, 2010 · 44 comments
Closed

Continuous Integration server improvements #1470

mattab opened this issue Jul 12, 2010 · 44 comments
Assignees
Labels
Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Milestone

Comments

@mattab
Copy link
Member

mattab commented Jul 12, 2010

High priority/ Super useful projects

Improvement ideas

  • Run code coverage during the tests and produce reports:
    The tests are already sometimes failing without running the integration tests.
  • Enable SchemaSpy http://qa.piwik.org:8080/schema/
  • Enable Nightly Build at: http://qa.piwik.org:8080/nightly/
  • Currently we must manually run the /tests/javascript/. We could use Selenium (and Sauce Labs) to run ./tests/javascript/. Selenium has the advantage of native browser testing.
  • Nightly builds for Piwik Mobile (Android), see test install script
  • automated API doc generation for piwik.js tracker
    • jsphpdoc - does some text transformations and feeds the output to phpDocumentor; since it doesn't parse the JS, it doesn't recognize some of the syntactic sugar to define methods. The result is missing the public methods, but includes many private methods.
    • jsdoc-toolkit - uses rhino/java; despite failing miserably (initial output has only three functions: cvt_hex, piwik_log, and piwik_track), it's probably the more promising of the two given its feature set (and usage by other projects). It appears to we have to add more tags as hints...

Created as a followup / summary of the 2 year old ticket #1470

More information about piwik QA: http://piwik.org/qa/

@mattab
Copy link
Member Author

mattab commented Jul 12, 2010

Other feature ideas

Note: I read on PHP sonar "PHP at least 5.2.6 is required for this procedure and the plugins to work smoothly". It should probably only run in one build anyway (more recent php version).

I would say that following features are important to have to make Hudson very useful:

  • the console output doesn't include the webtest output
    • also need to provide links to artifacts
  • hudson can send out email notifications re: build results
    • shall I create a mailing list similar to piwik-svn?
    • Hudson could send all build results to the committer email (do you have access to this info?), as well as to the mailing list.

@robocoder
Copy link
Contributor

Hudson doesn't archive artifacts for failed builds. This doesn't appear to be configurable.

I have scripted a partial workaround, copying the webtest results to http://qa.piwik.org/webtest/ and the Piwik documentation to http://qa.piwik.org/phpdocs/ (if generated).

@robocoder
Copy link
Contributor

Build results are now emailed to the piwik-builds mailing list.

@robocoder
Copy link
Contributor

To test release update:

Set up build server to:

  • provide service which mimics api.piwik.org/1.0/getLatestVersion
  • serve up latest.zip (svn export trunk and bundle it up in a zip file)
  • restrict to local access (127.0.0.1) to prevent abuse

Add web test:

  • if core/Version.php differs from (real) api.piwik.org/1.0/getLatestVersion
    • svn export using tag
    • install
    • generate fake visits
    • override piwik_url and piwik_host in config.ini.php
    • delete last update check timestamp (in options table)
    • visit dashboard and start the auto-update
    • run through the updater
    • check for errors

@robocoder
Copy link
Contributor

I created a master job called "CI" which does a single "svn up". This then triggers two jobs, "Docs" (phpdoc generation) and "Piwik" (unit tests, web tests, and schemaspy analysis), which can run in parallel.

The SchemaSpy analysis can now be found at http://qa.piwik.org/schema/

Note: since the integration unit tests are now run 3 times (non-webtest pdo_mysql, non-webtest mysqli, and webtest pdo_mysql), the "processed folder" is renamed between runs. If you browse to http://qa.piwik.org:8080/hudson/job/Piwik/ws/build/tests/integration/ you'll see the folder contains:

  • expected
  • processed
  • processed-mysqli
  • processed-pdo-mysql
  • Integration.php
  • Main.test.php
  • proxy-piwik.php

@robocoder
Copy link
Contributor

Ok, default error_reporting is now E_ALL | E_STRICT. all_tests.php is now failing with/without the webtest wrapper on php 5.2.14.

@mattab
Copy link
Member Author

mattab commented Jul 23, 2010

Anthon, unit tests are failing now, it seems the tracking is not working anymore.

Relevant part of http://qa.piwik.org/webtest/002_UnitTests/001_response_invoke.xml is:

<span class="fail">Fail</span>: ../tests/integration/Main.test.php -&gt; Test_Piwik_Integration_Main -&gt; test_noVisit -&gt;  at [/home/www/data/root/hudson.private/jobs/Piwik/workspace/build/tests/integration/Integration.php line 159]<br />

Expected GIF beacon, got: <br/>

is there a way to access error logs of the build? would be great to have them viewable online (or inside the zip artifacts).

I expect that something is wrong there. AT the same time, it is on display_errors On so we should see the error in the response, but it seems tracker is just returning blank response? any idea or anything in error logs?

@robocoder
Copy link
Contributor

Unit tests are finally green. (Setting allow_url_fopen=On solves the PiwikTracker problem.)

php-cgi errors are now logged to /var/log/jetty/error.log

@mattab
Copy link
Member Author

mattab commented Jul 27, 2010

Great job Anthon! it's all green now :) http://qa.piwik.org:8080/hudson/

@robocoder
Copy link
Contributor

Updated webtest to R_1807 and hudson to 1.369.

@robocoder
Copy link
Contributor

Rolled comment:2 and comment:6 ideas into main description. (LDAP login for Hudson isn't doable at this time per matt's email.)

@robocoder
Copy link
Contributor

Updated server to hudson 1.388, and switched to a single executor. Two executors allowed doc generation and unit testing in parallel. But having 2 also sometimes resulting in double builds, and false build failures.

@robocoder
Copy link
Contributor

(In [3433]) refs #1470 - add nightly build script; tweak the webtest build script

@robocoder
Copy link
Contributor

(In [3434]) refs #1470 - build environment changes

@mattab
Copy link
Member Author

mattab commented Jan 8, 2011

A good article pointing out Code coverage report in Hudson (via phpunit again, should we migrate...) and : http://techportal.ibuildings.com/2010/09/20/building-a-continuous-integration-server-for-php-with-hudson/

Also interesting: PHP Mess detector and PHP Copy Paste.

@mattab
Copy link
Member Author

mattab commented Jan 8, 2011

We have to install the Chuck Norris plugin

@halfdan
Copy link
Member

halfdan commented Jan 8, 2011

+1 for PHP Unit, simple test really bugs me sometimes.

For the locale, we should check a locale other than german/english. Maybe some chinese/japanese/cyrillic.

@robocoder
Copy link
Contributor

Matt: We're now running Jenkins. For the Hudson Trac plugin, please update the URL and re-enable it. Thanks.

@mattab
Copy link
Member Author

mattab commented Feb 5, 2011

done.

Looking at jenkins-php.org I have the feeling we will have to migrate to PHPUnit to benefit all the cool tools around PHP QA in Jenkins. Not sure how easy it will be to change the testing library, hopefuly not too difficult.

@robocoder
Copy link
Contributor

(In [3889]) refs #1470 - add WebTests for piwik.js (tracking-related tests are disabled because HtmlUnit behaves differently from native browser)

@robocoder
Copy link
Contributor

(In [3891]) refs #1470 - guard against a false positive, e.g., 0 tests of 0 passed, 0 failed.

@robocoder
Copy link
Contributor

(In [4109]) refs #1470 - test piwik.js against third-party javascript frameworks (dojo, extjs, jquery, prototype, mootools)

@robocoder
Copy link
Contributor

(In [4110]) refs #1470

@robocoder
Copy link
Contributor

(In [4111]) refs #1470

@robocoder
Copy link
Contributor

Tracking requests are disabled in WebTest because the HtmlUnit wrapper loads the href on simulated clicks into the current response instead of the designated iframe (WT-428); setting evt.returnValue to false or calling evt.preventDefault() doesn't work as intended, so can't be used as a workaround. Also, GET requests (fetching the web bug/beacon/pixel) are silently ignored (by design).

@mattab
Copy link
Member Author

mattab commented Mar 30, 2011

It would be nice that the build runs faster, for example by only running unit tests first, then another build would run the webtests (which seem to take few minutes).

It is sometimes useful to have the unit tests feedback after a few seconds than the current 7minutes build time, especially because running the full tests on my box takes 3min and I often just run one or two and rely on Jenkins for the full run.

@robocoder
Copy link
Contributor

Good point. I'll break this up into separate jobs, and only trigger the webtests if the unit tests pass.

@robocoder
Copy link
Contributor

(In [4286]) refs #1470 - fail build early if unit tests fail

@robocoder
Copy link
Contributor

Reminder to self of recent changes on dev6 that I have to update in the slack files:

  • php-cli: set memory_limit to -1
  • piwik_qa db user: GRANT FILE

@robocoder
Copy link
Contributor

jsphpdoc - does some text transformations and feeds the output to phpDocumentor; since it doesn't parse the JS, it doesn't recognize some of the syntactic sugar to define methods. The result is missing the public methods, but includes many private methods.

jsdoc-toolkit - uses rhino/java; despite failing miserably (initial output has only three functions: cvt_hex, piwik_log, and piwik_track), it's probably the more promising of the two given its feature set (and usage by other projects). It appears to we have to add more tags as hints...

@robocoder
Copy link
Contributor

(In [4380]) refs #1470 - add jsdoc-toolkit

@tsteur
Copy link
Member

tsteur commented May 22, 2011

Attachment:
install.sh

@robocoder
Copy link
Contributor

ok, Piwik Mobile docs are online at http://qa.piwik.org:8080/jsdocs/

I've updated the QA page, too.

@robocoder
Copy link
Contributor

PHP 5.4 dropped the sqlite extension -- required by tests/javascript/.

@mattab
Copy link
Member Author

mattab commented Sep 17, 2011

@sgiehl
Copy link
Member

sgiehl commented Sep 17, 2011

I allready migrated most core and plugin simpletests to phpunit on my local copy. When I got some time I'll have a close look at the integration tests. don't know wether they are easy to migrate or not.

@mattab
Copy link
Member Author

mattab commented Sep 17, 2011

Great to hear about your tests with Phpunit!!! exciting.

Migration tests should be the same to migrate as other tests, they don't use any special logic.

@sgiehl
Copy link
Member

sgiehl commented Sep 18, 2011

There is a small problem with the integration tests and some other tests. As simpletests are called in a browser there is a local URL to send api-requests to. phpunit is executed on command line and so there is no url. guess we need to put that to a config file in order to have the tests run correctly with phpunit.

@robocoder
Copy link
Contributor

Right. Jenkins sets the following environment vars:

export DISPLAY=:0.0
export REMOTE_ADDR=127.0.0.1
export HTTP_HOST=localhost:8080
export REQUEST_URI=/path/to/all_tests.php

@robocoder
Copy link
Contributor

Upgraded qa.piwik.org to WebTest R_1812 last nite. No regressions. http://qa.piwik.org:8080/jenkins/job/Piwik/2372/

@mattab
Copy link
Member Author

mattab commented Jun 18, 2012

Work for Simpletest to PHPUnit migration is done in: #3227
Will be a great improvement to have code coverage metrics, can't wait to see how we are doing...

EDIT: see also this presentation about Jenkins and PHP http://www.slideshare.net/sebastian_bergmann/integrate-your-php-project-with-jenkins-oscon-2012 by PHPUnit dev

@mattab
Copy link
Member Author

mattab commented Aug 9, 2012

Many times our sql update code wasn't failure-friendly and would fail when re-executed as it left the DB in a half-updated state.

Having the following test would ensure that all future update scripts written are working in a failure scenario.

A new idea for a webtest:

  • Download 1.8.3 release
  • Download latest
  • Run Auto update - but make sure the auto update code fails at the last step
  • Re-run auto update: the code should still run and the SQL not fail. For this, some queries will throw exceptions which must be specifically ignored, eg. "1060 Duplicate column name 'xx'" can be ignored for an ALTER that adds a column.
  • The update re-running after initial failure should succeed.

@robocoder
Copy link
Contributor

Dropping Sonar from the todo list.

@mattab
Copy link
Member Author

mattab commented Dec 13, 2012

See ticket for Webtests request: #2935

This ticket is mostly implemented after 2 years! See #3606 for the follow up ticket listing all ideas and improvements to our continuous integration server and processes. Good times!

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Projects
None yet
Development

No branches or pull requests

5 participants