New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Setup continuous integration server for Piwik builds #818
Comments
bamboo.openx.org server appears to be no longer available; should we close this or look into setting up our own (eg using xinc)? |
vipsoft, we should consider running our own bamboo, or maybe Hudson as it seems better. We could have 1 or 2 boxes by our sponsor Free to support it - if you want to have a try I can give you the server details |
I looked at xing, interesting to build this in php! the project hasn't seen an update for a while though.. I used Hudson in the past, and people were very happy with it; !http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson |
I thought we could use xinc to avoid using a java server, but since webtest is java-based, we might as well use bamboo or hudson. |
Note: the current Bamboo is at http://bamboo.openx.org:8085/browse/PIWIK-TRUNK but seems to fail due to some errors. We would like to setup our own integration build server at eg. tests.piwik.org |
I asked atlassian and they offered a free license of Bamboo. However, it might be better for us to pay for hosted continuous integration server (bamboo or hudson). If anyone knows such a hosted service, let us know! |
System requirements are:
Is Free still offering a box (or two)? Maybe we can set this up ourselves. |
Anthon, |
Should test Piwik with:
To test the one click update process, a script could download and install the latest release (latest.zip). Then instead of api.piwik.org/1.0/getLatestVersion, it could check api.piwik.org/1.0/getLatestSvnRevision, and update to the latest svn snapshot, e.g., !http://dev.piwik.org/trac/changeset/{latest revision}/trunk?old_path=%2F&format=zip. (Unlike the build script, Trac packages it in the "trunk" folder instead of "piwik".) It gets a little trickier because the piwik_option table stores the version number in UpdateCheck_LatestVersion, not the SVN version. |
It would be nice if there was some support for tinderbox-like scripts to collect build results from remote build servers because it would be desirable to have builds continuously tested on Windows IIS and XAMPP. |
Should also include testing with E_STRICT because some environments don't permit the error_reporting level to be overridden by .htaccess or scripts. |
(In [2269]) refs #818 - unit tests with both pdo_mysql and mysqli adapters |
(In [2270]) refs #818 - fix typo in target |
(In [2271]) refs #818 - bamboo.openx.org doesn't have mysqli extension ... comment out for now |
Replying to vipsoft:
Cool. Hudson and Bamboo support remote agents, or on the cloud (Amazon EC2) -- what Atlassian calls "elastic agents". Neither provide Amazon Machine Images for Windows, nor support for Microsoft Azure. |
Minor issue with Hudson on Jetty: http://issues.hudson-ci.org/browse/HUDSON-6760 |
(In [2318]) refs #818 - build directory missing a folder |
Note to self: It appears there's a bug in HtmlUnit 2.7 that causes WebTest R_1804 to fail the web test after sign in. Sizzle.filter (in jquery.js 1.4.2) is throwing an exception (Syntax error, unrecognized expression: ul li) for this selector in superfish-modified.js, "li:has(ul li)". |
Anthon, I have a few questions
I'm working on acceptance tests that will test Tracker with known data sets, and then call all API methods and Widgets, and check the XML/ HTML output. I hope to finish this in the next few days, which would give great coverage to Piwik testing. Do you know when Hudson service could be available, any chance before end of Month/1.0 release? Thanks |
Replying to matt:
Eventually. I commented out the mysqli unit test because bamboo.openx.org (or rather its elastic agent) doesn't have the mysqli extension installed.
We could use webtest (via HtmlUnit browser simulation). Need to investigate further.
I might be able to wrap it in a webtest testcase.
As of last nite, I have it running locally but the webtest is failing on the second-to-last step. I tried using Webtest 3.0 and the same version of webtest as on bamboo.openx.org (r 1783) but both choked, so I'm now running the latest snapshot and trying to locate the source of the failure in HtmlUnit. Right now, it's not looking hopeful for end-of-the-month. |
Re: piwik.js. It looks like we can use Selenium for native browser testing. (There's a Hudson plugin for this.) |
(In [2414]) refs #818 - phpdoc 1.4.3 expects this to be an absolute path |
Looks like #660 works around the problem in HtmlUnit. I'm going to start moving my build setup over to dev6. |
Awesome. The continuous build server running Unit tests as well as webtests will be a life saver!! (not to note the possibility of running mysqli and later postgresql...) |
Another option for native browser testing of piwik.js is to run our own Testswarm instance. http://wiki.github.com/jeresig/testswarm/ |
It looks like bamboo.openx.org is down, so there's greater urgency to get this setup on dev6. In r2429, Matt indicates that Config.test.php fails when run standalone. We should definitely consider iterating through the unit tests, in addition to all_tests.php, as this can catch some side-effects (due to globals). |
Fixed the config.test.php in r2433 |
(In [2445]) refs #818 - add webtest wrapper around unit tests |
Hudson is now running at http://qa.piwik.org:8080/ with some the unit tests are failing. (See other tickets.) I'll open a new ticket for improvements. |
Nice work! The feeds are broken though: http://qa.piwik.org:8080/hudson/rssFailed The items links to iproxy.ca:8080. |
(In [2485]) Refs #818 - upcase words in installation steps; use "JavaScript" consistently; fixes WebTest (installation) Artifacts can be access through the "Workspace" link, once you're logged in, e.g., (There's a configuration option to archive artifacts somewhere...) Yes, the plan is to use Piwik_QA to tweet build results. |
Is it possible to make Artifacts public? I can't login and would very much like to see the current build status and possible errors. http://trac-hacks.org/wiki/HudsonTracPlugin would be nice for trac to see the build history from hudson in the trac timeline. |
re: public artifacts are possible, but we'll have to review the artifacts to ensure no passwords and internal paths are disclosed. re: hudson trac plugin. Good idea. We don't currently use the Bamboo Trac plugin that's currently installed. |
I've rebuilt php with --enable-mbstring, so the mb_convert_encoding calls work (requires for .csv and .tsv output). The webtest wrapper on the unit tests is currently failing because the output isn't proper html, eg text before the <html> and after the </html>. |
(In [2488]) refs #818 - clean up output |
(In [2489]) refs #818 |
Current status of our Hudson server:
|
Are the phpdocs artifacts available for download? It would be cool to link the http://piwik.org/docs/tracking-api/ doc page to the PiwikTracker API page (to the doc from latest successful build maybe?). |
I made some configuration changes to the job, but they're not having any effect. It appears that webtest results are treated as build artifacts, and artifacts are not published if the build fails. I'll try scripting a workaround. |
|
In [2502], exclude integration tests if REQUEST_UR or HTTP_HOST not defined; this means we can now fully test with php-cli if these CGI environment variables are defined. |
(In [2503]) refs #818 - PATH_INFO and SCRIPT_NAME are also accepted |
Great for artifacts. Should the URL be prefixed by the job ID? (or do we only keep artifacts for last build?) |
When you say, "if generated", are the latest successful build doc pages always going to be there (ie. failing builds don't delete old docs). Can I safely link to http://qa.piwik.org/phpdocs/Piwik/PiwikTracker.html ? |
Webtest results are only for the latest build. Yes, should be safe to link to the docs. It's only overwritten if the build succeeds. |
I linked the page from the bottom of: http://piwik.org/docs/tracking-api/ |
We use bamboo for continuous integration: https://bamboo.openx.org/
currently it runs the webtest that tests the installation process + reset password feature.
it should also run the unit tests and fail when at least one test doesn't pass. there is some magic involved to rewrite simpletests results in XML to be parsed by the tool.
The tool has been failing lately and we would like to setup our own Build integration software on 2 servers we have available for the task.
The text was updated successfully, but these errors were encountered: