Opened 5 years ago

Closed 4 years ago

Last modified 4 years ago

#818 closed New feature (fixed)

Setup continuous integration server for Piwik builds

Reported by: matt Owned by: vipsoft
Priority: critical Milestone: Piwik 0.6.4
Component: Website piwik.org Keywords:
Cc: Sensitive: no

Description (last modified by matt)

We use bamboo for continuous integration: https://bamboo.openx.org/

currently it runs the webtest that tests the installation process + reset password feature.
it should also run the unit tests and fail when at least one test doesn't pass. there is some magic involved to rewrite simpletests results in XML to be parsed by the tool.

The tool has been failing lately and we would like to setup our own Build integration software on 2 servers we have available for the task.

Change History (58)

comment:1 Changed 5 years ago by matt (mattab)

  • Component changed from Core to Website piwik.org

comment:2 Changed 5 years ago by koteiko

comment:3 Changed 4 years ago by vipsoft (robocoder)

  • Sensitive unset

bamboo.openx.org server appears to be no longer available; should we close this or look into setting up our own (eg using xinc)?

comment:4 Changed 4 years ago by matt (mattab)

vipsoft, we should consider running our own bamboo, or maybe Hudson as it seems better. We could have 1 or 2 boxes by our sponsor Free to support it - if you want to have a try I can give you the server details

comment:5 Changed 4 years ago by matt (mattab)

I looked at xing, interesting to build this in php! the project hasn't seen an update for a while though.. I used Hudson in the past, and people were very happy with it; http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson

comment:6 Changed 4 years ago by matt (mattab)

  • Description modified (diff)
  • Summary changed from bamboo should run unit tests as well as webtests to Setup continuous integration servers for Piwik builds

comment:7 Changed 4 years ago by vipsoft (robocoder)

I thought we could use xinc to avoid using a java server, but since webtest is java-based, we might as well use bamboo or hudson.

comment:8 Changed 4 years ago by matt (mattab)

  • Type changed from Bug to New feature

comment:9 Changed 4 years ago by matt (mattab)

Note: the current Bamboo is at http://bamboo.openx.org:8085/browse/PIWIK-TRUNK but seems to fail due to some errors. We would like to setup our own integration build server at eg. tests.piwik.org

comment:10 Changed 4 years ago by matt (mattab)

I asked atlassian and they offered a free license of Bamboo. However, it might be better for us to pay for hosted continuous integration server (bamboo or hudson). If anyone knows such a hosted service, let us know!

comment:11 Changed 4 years ago by vipsoft (robocoder)

System requirements are:

  • multiple PHP CGI versions (e.g., 5.1.3, 5.2.0, 5.2.6, 5.2-SVN, 5.3.1, 5.3-SVN)
  • Tomcat (or other Java server) to support bamboo or hudson; or Apache if we go with xinc
  • Java runtime, ant, JDK for WebTest
  • MySQL (and in the future, other databases)

Is Free still offering a box (or two)? Maybe we can set this up ourselves.

comment:12 Changed 4 years ago by matt (mattab)

Anthon,
Yes the servers are still available. I can give you the details by email if you'd like to give it a try.

comment:13 Changed 4 years ago by vipsoft (robocoder)

Should test Piwik with:

  • safe_mode disabled and no open_basedir restriction
  • safe_mode enabled
  • open_basedir restriction
  • safe_mode enabled and open_basedir restriction

To test the one click update process, a script could download and install the latest release (latest.zip). Then instead of api.piwik.org/1.0/getLatestVersion, it could check api.piwik.org/1.0/getLatestSvnRevision, and update to the latest svn snapshot, e.g., http://dev.piwik.org/trac/changeset/{latest revision}/trunk?old_path=%2F&format=zip. (Unlike the build script, Trac packages it in the "trunk" folder instead of "piwik".) It gets a little trickier because the piwik_option table stores the version number in UpdateCheck_LatestVersion, not the SVN version.

comment:14 follow-up: Changed 4 years ago by vipsoft (robocoder)

It would be nice if there was some support for tinderbox-like scripts to collect build results from remote build servers because it would be desirable to have builds continuously tested on Windows IIS and XAMPP.

comment:15 Changed 4 years ago by vipsoft (robocoder)

Should also include testing with E_STRICT because some environments don't permit the error_reporting level to be overridden by .htaccess or scripts.

comment:16 Changed 4 years ago by vipsoft (robocoder)

(In [2269]) refs #818 - unit tests with both pdo_mysql and mysqli adapters

comment:17 Changed 4 years ago by vipsoft (robocoder)

(In [2270]) refs #818 - fix typo in target

comment:18 Changed 4 years ago by vipsoft (robocoder)

(In [2271]) refs #818 - bamboo.openx.org doesn't have mysqli extension ... comment out for now

comment:19 in reply to: ↑ 14 Changed 4 years ago by vipsoft (robocoder)

Replying to vipsoft:

It would be nice if there was some support for tinderbox-like scripts to collect build results from remote build servers because it would be desirable to have builds continuously tested on Windows IIS and XAMPP.

Cool. Hudson and Bamboo support remote agents, or on the cloud (Amazon EC2) -- what Atlassian calls "elastic agents". Neither provide Amazon Machine Images for Windows, nor support for Microsoft Azure.

comment:20 Changed 4 years ago by vipsoft (robocoder)

  • Owner changed from matt to vipsoft

comment:22 Changed 4 years ago by vipsoft (robocoder)

(In [2318]) refs #818 - build directory missing a folder

comment:23 Changed 4 years ago by vipsoft (robocoder)

Note to self: It appears there's a bug in HtmlUnit 2.7 that causes WebTest R_1804 to fail the web test after sign in. Sizzle.filter (in jquery.js 1.4.2) is throwing an exception (Syntax error, unrecognized expression: ul li) for this selector in superfish-modified.js, "li:has(ul li)".

comment:24 follow-up: Changed 4 years ago by matt (mattab)

Anthon, I have a few questions

  • Will Hudson run on Mysql, mysqli all tests?
  • Is there a way to trigger the JS tests for piwik.js, as well as unit tests and webtests?
  • Would it be better if we migrate to PhpUnit if we want to have the unit tests ran easily (right now Bamboo doesn't parse Simpletest output)

I'm working on acceptance tests that will test Tracker with known data sets, and then call all API methods and Widgets, and check the XML/ HTML output. I hope to finish this in the next few days, which would give great coverage to Piwik testing. Do you know when Hudson service could be available, any chance before end of Month/1.0 release?

Thanks

comment:25 in reply to: ↑ 24 Changed 4 years ago by vipsoft (robocoder)

Replying to matt:

Anthon, I have a few questions

  • Will Hudson run on Mysql, mysqli all tests?

Eventually. I commented out the mysqli unit test because bamboo.openx.org (or rather its elastic agent) doesn't have the mysqli extension installed.

  • Is there a way to trigger the JS tests for piwik.js, as well as unit tests and webtests?

We could use webtest (via HtmlUnit browser simulation). Need to investigate further.

  • Would it be better if we migrate to PhpUnit if we want to have the unit tests ran easily (right now Bamboo doesn't parse Simpletest output)

I might be able to wrap it in a webtest testcase.

I'm working on acceptance tests that will test Tracker with known data sets, and then call all API methods and Widgets, and check the XML/ HTML output. I hope to finish this in the next few days, which would give great coverage to Piwik testing. Do you know when Hudson service could be available, any chance before end of Month/1.0 release?

As of last nite, I have it running locally but the webtest is failing on the second-to-last step. I tried using Webtest 3.0 and the same version of webtest as on bamboo.openx.org (r 1783) but both choked, so I'm now running the latest snapshot and trying to locate the source of the failure in HtmlUnit. Right now, it's not looking hopeful for end-of-the-month.

comment:26 Changed 4 years ago by vipsoft (robocoder)

Re: piwik.js. It looks like we can use Selenium for native browser testing. (There's a Hudson plugin for this.)

comment:27 Changed 4 years ago by vipsoft (robocoder)

(In [2414]) refs #818 - phpdoc 1.4.3 expects this to be an absolute path

comment:28 Changed 4 years ago by vipsoft (robocoder)

Looks like #660 works around the problem in HtmlUnit.

I'm going to start moving my build setup over to dev6.

comment:29 Changed 4 years ago by matt (mattab)

Awesome.

The continuous build server running Unit tests as well as webtests will be a life saver!! (not to note the possibility of running mysqli and later postgresql...)

comment:30 Changed 4 years ago by vipsoft (robocoder)

Another option for native browser testing of piwik.js is to run our own Testswarm instance. http://wiki.github.com/jeresig/testswarm/

comment:31 Changed 4 years ago by vipsoft (robocoder)

It looks like bamboo.openx.org is down, so there's greater urgency to get this setup on dev6.

In r2429, Matt indicates that Config.test.php fails when run standalone. We should definitely consider iterating through the unit tests, in addition to all_tests.php, as this can catch some side-effects (due to globals).

comment:32 Changed 4 years ago by matt (mattab)

Fixed the config.test.php in r2433

comment:33 Changed 4 years ago by vipsoft (robocoder)

(In [2445]) refs #818 - add webtest wrapper around unit tests

comment:34 Changed 4 years ago by vipsoft (robocoder)

  • Milestone changed from 3 - Piwik 0.9 - Surviving The Wild to 0 - Piwik 0.6.4
  • Priority changed from major to critical

comment:35 Changed 4 years ago by vipsoft (robocoder)

  • Resolution set to fixed
  • Status changed from new to closed
  • Summary changed from Setup continuous integration servers for Piwik builds to Setup continuous integration server for Piwik builds

Hudson is now running at http://qa.piwik.org:8080/ with some the unit tests are failing. (See other tickets.)

I'll open a new ticket for improvements.

comment:36 Changed 4 years ago by halfdan

Nice work! The feeds are broken though:

http://qa.piwik.org:8080/hudson/rssFailed

The items links to iproxy.ca:8080.

comment:37 Changed 4 years ago by vipsoft (robocoder)

Thanks. Hopefully that's the last config migration change.

comment:38 Changed 4 years ago by vipsoft (robocoder)

(In [2471]) refs #818 - set-up database_tests section for unit testing via webtest wrapper

comment:39 Changed 4 years ago by matt (mattab)

(In [2472]) Refs #1446 Refs #818

  • Fixing unit tests on local box by ignoring ExampleAPI.getPiwikVersion which would fail at every new version

Note: on my box, when I run all_tests.php, I don't see the green bar at the bottom of the page anymore. It simply displays the Time and Memory delta. I didn't see any commit that coudl have broken this so I'm a bit confused as to why the green/red bar doesn't display anymore?

comment:40 Changed 4 years ago by matt (mattab)

  • Resolution fixed deleted
  • Status changed from closed to reopened

Reopened as it seems unit tests are not being executed?
See for example output in: http://qa.piwik.org:8080/hudson/job/Piwik/93/console

Also few other questions

  • can we access webtests artifacts? this is important to see what webtests failed, and see what the output was like.
  • can we access integration tests artifacts (eg. in tests/integration/processed/ ?). This is important in case an integration test fails on Hudson but not in local box.
  • are you planning to use Piwik_QA twitter to post build results?

thanks!

comment:41 Changed 4 years ago by matt (mattab)

(In [2474]) Refs #1125

  • fixing integration tests
  • fixing bug when no visit (called function on non object)

Refs #818: all_tests.php is now green on my box

comment:42 Changed 4 years ago by matt (mattab)

(In [2478]) Refs #818
When tests fail with a fatal error, it will now display as expected in the test page. This was the cause for the green/red bar not displaying (a fatal error was thrown in the UserSettings API called from the Integration tests, and was hidden).

comment:43 Changed 4 years ago by vipsoft (robocoder)

(In [2485]) Refs #818 - upcase words in installation steps; use "JavaScript" consistently; fixes WebTest (installation)

Artifacts can be access through the "Workspace" link, once you're logged in, e.g.,

(There's a configuration option to archive artifacts somewhere...)

Yes, the plan is to use Piwik_QA to tweet build results.

comment:44 Changed 4 years ago by halfdan

Is it possible to make Artifacts public? I can't login and would very much like to see the current build status and possible errors.

http://trac-hacks.org/wiki/HudsonTracPlugin would be nice for trac to see the build history from hudson in the trac timeline.

comment:45 Changed 4 years ago by vipsoft (robocoder)

re: public artifacts are possible, but we'll have to review the artifacts to ensure no passwords and internal paths are disclosed.

re: hudson trac plugin. Good idea. We don't currently use the Bamboo Trac plugin that's currently installed.

comment:46 Changed 4 years ago by vipsoft (robocoder)

I've rebuilt php with --enable-mbstring, so the mb_convert_encoding calls work (requires for .csv and .tsv output).

The webtest wrapper on the unit tests is currently failing because the output isn't proper html, eg text before the <html> and after the </html>.

http://qa.piwik.org:8080/hudson/job/Piwik/ws/default/tests/results/002_UnitTests/001_response_invoke.xml/*view*/

comment:47 Changed 4 years ago by vipsoft (robocoder)

(In [2488]) refs #818 - clean up output

comment:49 Changed 4 years ago by vipsoft (robocoder)

Current status of our Hudson server:

  • php 5.2.13 and pdo_mysql
  • unit tests via php-cli (excludes tests/integration) passes
  • webtest install passes
  • webtest unit tests (includes tests/integration) passes
  • phpdocs generated

comment:50 Changed 4 years ago by matt (mattab)

Are the phpdocs artifacts available for download? It would be cool to link the http://piwik.org/docs/tracking-api/ doc page to the PiwikTracker API page (to the doc from latest successful build maybe?).

comment:51 Changed 4 years ago by vipsoft (robocoder)

I made some configuration changes to the job, but they're not having any effect. It appears that webtest results are treated as build artifacts, and artifacts are not published if the build fails.

I'll try scripting a workaround.

comment:52 Changed 4 years ago by vipsoft (robocoder)

comment:53 Changed 4 years ago by vipsoft (robocoder)

  • Resolution set to fixed
  • Status changed from reopened to closed

In [2502], exclude integration tests if REQUEST_UR or HTTP_HOST not defined; this means we can now fully test with php-cli if these CGI environment variables are defined.

comment:54 Changed 4 years ago by vipsoft (robocoder)

(In [2503]) refs #818 - PATH_INFO and SCRIPT_NAME are also accepted

comment:55 Changed 4 years ago by matt (mattab)

Great for artifacts. Should the URL be prefixed by the job ID? (or do we only keep artifacts for last build?)

comment:56 Changed 4 years ago by matt (mattab)

When you say, "if generated", are the latest successful build doc pages always going to be there (ie. failing builds don't delete old docs). Can I safely link to http://qa.piwik.org/phpdocs/Piwik/PiwikTracker.html ?

comment:57 Changed 4 years ago by vipsoft (robocoder)

Webtest results are only for the latest build.

Yes, should be safe to link to the docs. It's only overwritten if the build succeeds.

comment:58 Changed 4 years ago by matt (mattab)

I linked the page from the bottom of: http://piwik.org/docs/tracking-api/

Note: See TracTickets for help on using tickets.