Opened 4 years ago

Closed 16 months ago

#1470 closed Task (fixed)

Continuous Integration server improvements

Reported by: matt Owned by: vipsoft
Priority: major Milestone: 1.10 - Piwik 1.10
Component: Core Keywords:
Cc: Sensitive: no

Description (last modified by matt)

High priority/ Super useful projects

Improvement ideas

  • Enable SchemaSpy ​http://qa.piwik.org:8080/schema/
  • Enable Nightly Build at: ​http://qa.piwik.org:8080/nightly/
  • Currently we must manually run the /tests/javascript/. We could use Selenium (and Sauce Labs) to run ./tests/javascript/. Selenium has the advantage of native browser testing.
  • Nightly builds for Piwik Mobile (Android), see test install script
  • automated API doc generation for piwik.js tracker
    • jsphpdoc - does some text transformations and feeds the output to phpDocumentor; since it doesn't parse the JS, it doesn't recognize some of the syntactic sugar to define methods. The result is missing the public methods, but includes many private methods.
    • jsdoc-toolkit - uses rhino/java; despite failing miserably (initial output has only three functions: cvt_hex, piwik_log, and piwik_track), it's probably the more promising of the two given its feature set (and usage by other projects). It appears to we have to add more tags as hints...

Created as a followup / summary of the 2 year old ticket #1470

More information about piwik QA: http://piwik.org/qa/

Attachments (1)

install.sh (4.2 KB) - added by tsteur 3 years ago.

Download all attachments as: .zip

Change History (63)

comment:1 Changed 4 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:2 Changed 4 years ago by matt (mattab)

Other feature ideas

Note: I read on PHP sonar "PHP at least 5.2.6 is required for this procedure and the plugins to work smoothly". It should probably only run in one build anyway (more recent php version).


I would say that following features are important to have to make Hudson very useful:

  • the console output doesn't include the webtest output
    • also need to provide links to artifacts
  • hudson can send out email notifications re: build results
    • shall I create a mailing list similar to piwik-svn?
    • Hudson could send all build results to the committer email (do you have access to this info?), as well as to the mailing list.

comment:3 Changed 4 years ago by vipsoft (robocoder)

  • Description modified (diff)

Hudson doesn't archive artifacts for failed builds. This doesn't appear to be configurable.

I have scripted a partial workaround, copying the webtest results to http://qa.piwik.org/webtest/ and the Piwik documentation to http://qa.piwik.org/phpdocs/ (if generated).

comment:4 Changed 4 years ago by vipsoft (robocoder)

  • Description modified (diff)
  • Owner set to vipsoft

comment:5 Changed 4 years ago by vipsoft (robocoder)

  • Description modified (diff)

Build results are now emailed to the piwik-builds mailing list.

comment:6 Changed 4 years ago by vipsoft (robocoder)

To test release update:

Set up build server to:

  • provide service which mimics api.piwik.org/1.0/getLatestVersion
  • serve up latest.zip (svn export trunk and bundle it up in a zip file)
  • restrict to local access (127.0.0.1) to prevent abuse

Add web test:

  • if core/Version.php differs from (real) api.piwik.org/1.0/getLatestVersion
    • svn export using tag
    • install
    • generate fake visits
    • override piwik_url and piwik_host in config.ini.php
    • delete last update check timestamp (in options table)
    • visit dashboard and start the auto-update
    • run through the updater
    • check for errors

comment:7 Changed 4 years ago by matt (mattab)

  • Description modified (diff)

comment:8 Changed 4 years ago by vipsoft (robocoder)

I created a master job called "CI" which does a single "svn up". This then triggers two jobs, "Docs" (phpdoc generation) and "Piwik" (unit tests, web tests, and schemaspy analysis), which can run in parallel.

The SchemaSpy analysis can now be found at http://qa.piwik.org/schema/

Note: since the integration unit tests are now run 3 times (non-webtest pdo_mysql, non-webtest mysqli, and webtest pdo_mysql), the "processed folder" is renamed between runs. If you browse to http://qa.piwik.org:8080/hudson/job/Piwik/ws/build/tests/integration/ you'll see the folder contains:

  • expected
  • processed
  • processed-mysqli
  • processed-pdo-mysql
  • Integration.php
  • Main.test.php
  • proxy-piwik.php

comment:9 Changed 4 years ago by vipsoft (robocoder)

  • Description modified (diff)

Ok, default error_reporting is now E_ALL | E_STRICT. all_tests.php is now failing with/without the webtest wrapper on php 5.2.14.

comment:10 Changed 4 years ago by matt (mattab)

Anthon, unit tests are failing now, it seems the tracking is not working anymore.

Relevant part of http://qa.piwik.org/webtest/002_UnitTests/001_response_invoke.xml is:

<span class="fail">Fail</span>: ../tests/integration/Main.test.php -&gt; Test_Piwik_Integration_Main -&gt; test_noVisit -&gt;  at [/home/www/data/root/hudson.private/jobs/Piwik/workspace/build/tests/integration/Integration.php line 159]<br />

Expected GIF beacon, got: <br/>

is there a way to access error logs of the build? would be great to have them viewable online (or inside the zip artifacts).

I expect that something is wrong there. AT the same time, it is on display_errors On so we should see the error in the response, but it seems tracker is just returning blank response? any idea or anything in error logs?

comment:11 Changed 4 years ago by vipsoft (robocoder)

Unit tests are finally green. (Setting allow_url_fopen=On solves the PiwikTracker problem.)

php-cgi errors are now logged to /var/log/jetty/error.log

comment:12 Changed 4 years ago by matt (mattab)

Great job Anthon! it's all green now :) http://qa.piwik.org:8080/hudson/

comment:13 Changed 4 years ago by vipsoft (robocoder)

Updated webtest to R_1807 and hudson to 1.369.

comment:14 Changed 4 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:15 Changed 4 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:16 Changed 3 years ago by matt (mattab)

  • Description modified (diff)

comment:17 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

Rolled comment:2 and comment:6 ideas into main description. (LDAP login for Hudson isn't doable at this time per matt's email.)

comment:18 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:19 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:20 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:21 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:22 Changed 3 years ago by vipsoft (robocoder)

Updated server to hudson 1.388, and switched to a single executor. Two executors allowed doc generation and unit testing in parallel. But having 2 also sometimes resulting in double builds, and false build failures.

comment:23 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:24 Changed 3 years ago by vipsoft (robocoder)

(In [3433]) refs #1470 - add nightly build script; tweak the webtest build script

comment:25 Changed 3 years ago by vipsoft (robocoder)

(In [3434]) refs #1470 - build environment changes

comment:26 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:27 Changed 3 years ago by matt (mattab)

A good article pointing out Code coverage report in Hudson (via phpunit again, should we migrate...) and : http://techportal.ibuildings.com/2010/09/20/building-a-continuous-integration-server-for-php-with-hudson/

Also interesting: PHP Mess detector and PHP Copy Paste.

comment:28 Changed 3 years ago by matt (mattab)

We have to install the Chuck Norris plugin

comment:29 Changed 3 years ago by halfdan

+1 for PHP Unit, simple test really bugs me sometimes.

For the locale, we should check a locale other than german/english. Maybe some chinese/japanese/cyrillic.

comment:30 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:31 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:32 Changed 3 years ago by vipsoft (robocoder)

Matt: We're now running Jenkins. For the Hudson Trac plugin, please update the URL and re-enable it. Thanks.

comment:33 Changed 3 years ago by matt (mattab)

done.

Looking at jenkins-php.org I have the feeling we will have to migrate to PHPUnit to benefit all the cool tools around PHP QA in Jenkins. Not sure how easy it will be to change the testing library, hopefuly not too difficult.

comment:34 Changed 3 years ago by vipsoft (robocoder)

(In [3889]) refs #1470 - add WebTests for piwik.js (tracking-related tests are disabled because HtmlUnit behaves differently from native browser)

comment:35 Changed 3 years ago by vipsoft (robocoder)

(In [3891]) refs #1470 - guard against a false positive, e.g., 0 tests of 0 passed, 0 failed.

comment:36 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:37 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:38 Changed 3 years ago by vipsoft (robocoder)

(In [4109]) refs #1470 - test piwik.js against third-party javascript frameworks (dojo, extjs, jquery, prototype, mootools)

comment:41 Changed 3 years ago by matt (mattab)

  • Priority changed from normal to major

comment:42 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

Tracking requests are disabled in WebTest because the HtmlUnit wrapper loads the href on simulated clicks into the current response instead of the designated iframe (WT-428); setting evt.returnValue to false or calling evt.preventDefault() doesn't work as intended, so can't be used as a workaround. Also, GET requests (fetching the web bug/beacon/pixel) are silently ignored (by design).

comment:43 Changed 3 years ago by matt (mattab)

It would be nice that the build runs faster, for example by only running unit tests first, then another build would run the webtests (which seem to take few minutes).

It is sometimes useful to have the unit tests feedback after a few seconds than the current 7minutes build time, especially because running the full tests on my box takes 3min and I often just run one or two and rely on Jenkins for the full run.

comment:44 Changed 3 years ago by vipsoft (robocoder)

Good point. I'll break this up into separate jobs, and only trigger the webtests if the unit tests pass.

comment:45 Changed 3 years ago by vipsoft (robocoder)

(In [4286]) refs #1470 - fail build early if unit tests fail

comment:46 Changed 3 years ago by vipsoft (robocoder)

Reminder to self of recent changes on dev6 that I have to update in the slack files:

  • php-cli: set memory_limit to -1
  • piwik_qa db user: GRANT FILE

comment:47 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

jsphpdoc - does some text transformations and feeds the output to phpDocumentor; since it doesn't parse the JS, it doesn't recognize some of the syntactic sugar to define methods. The result is missing the public methods, but includes many private methods.

jsdoc-toolkit - uses rhino/java; despite failing miserably (initial output has only three functions: cvt_hex, piwik_log, and piwik_track), it's probably the more promising of the two given its feature set (and usage by other projects). It appears to we have to add more tags as hints...

comment:48 Changed 3 years ago by vipsoft (robocoder)

(In [4380]) refs #1470 - add jsdoc-toolkit

comment:49 Changed 3 years ago by tsteur

  • Description modified (diff)

comment:50 Changed 3 years ago by tsteur

  • Description modified (diff)

Changed 3 years ago by tsteur

comment:51 Changed 3 years ago by vipsoft (robocoder)

  • Description modified (diff)

ok, Piwik Mobile docs are online at http://qa.piwik.org:8080/jsdocs/

I've updated the QA page, too.

comment:52 Changed 3 years ago by vipsoft (robocoder)

PHP 5.4 dropped the sqlite extension -- required by tests/javascript/.

comment:53 Changed 3 years ago by matt (mattab)

  • Milestone changed from Feature requests to 1.x - Piwik 1.x

comment:54 Changed 3 years ago by SteveG (sgiehl)

I allready migrated most core and plugin simpletests to phpunit on my local copy. When I got some time I'll have a close look at the integration tests. don't know wether they are easy to migrate or not.

comment:55 Changed 3 years ago by matt (mattab)

Great to hear about your tests with Phpunit!!! exciting.

Migration tests should be the same to migrate as other tests, they don't use any special logic.

comment:56 Changed 3 years ago by SteveG (sgiehl)

There is a small problem with the integration tests and some other tests. As simpletests are called in a browser there is a local URL to send api-requests to. phpunit is executed on command line and so there is no url. guess we need to put that to a config file in order to have the tests run correctly with phpunit.

comment:57 Changed 3 years ago by vipsoft (robocoder)

Right. Jenkins sets the following environment vars:

export DISPLAY=:0.0
export REMOTE_ADDR=127.0.0.1
export HTTP_HOST=localhost:8080
export REQUEST_URI=/path/to/all_tests.php

comment:58 Changed 3 years ago by vipsoft (robocoder)

Upgraded qa.piwik.org to WebTest R_1812 last nite. No regressions. http://qa.piwik.org:8080/jenkins/job/Piwik/2372/

comment:59 Changed 22 months ago by matt (mattab)

Work for Simpletest to PHPUnit migration is done in: #3227
Will be a great improvement to have code coverage metrics, can't wait to see how we are doing...

EDIT: see also this presentation about Jenkins and PHP http://www.slideshare.net/sebastian_bergmann/integrate-your-php-project-with-jenkins-oscon-2012 by PHPUnit dev

comment:60 Changed 21 months ago by matt (mattab)

Many times our sql update code wasn't failure-friendly and would fail when re-executed as it left the DB in a half-updated state.

Having the following test would ensure that all future update scripts written are working in a failure scenario.

A new idea for a webtest:

  • Download 1.8.3 release
  • Download latest
  • Run Auto update - but make sure the auto update code fails at the last step
  • Re-run auto update: the code should still run and the SQL not fail. For this, some queries will throw exceptions which must be specifically ignored, eg. "1060 Duplicate column name 'xx'" can be ignored for an ALTER that adds a column.
  • The update re-running after initial failure should succeed.

comment:61 Changed 21 months ago by vipsoft (robocoder)

  • Description modified (diff)

Dropping Sonar from the todo list.

comment:62 Changed 16 months ago by matt (mattab)

  • Component changed from Website piwik.org to Core
  • Description modified (diff)
  • Milestone changed from 1.x - Piwik 1.x to 1.9.3 - Piwik 1.9.3
  • Reporter changed from vipsoft to matt
  • Resolution set to fixed
  • Status changed from new to closed
  • Summary changed from Continuous integration server - improvements to Continuous Integration server improvements
  • Type changed from New feature to Task

See ticket for Webtests request: #2935

This ticket is mostly implemented after 2 years! See #3606 for the follow up ticket listing all ideas and improvements to our continuous integration server and processes. Good times!

Note: See TracTickets for help on using tickets.