#3314 closed New feature (fixed)

Add support for Composer, Vagrant and Puppet

Reported by: tsteur Owned by: tsteur
Priority: normal Milestone: 1.9.2 - Piwik 1.9.2
Component: Core Keywords:
Cc: Sensitive: no

Description (last modified by matt)

DONE! :-) See: Get started with Piwik development – Piwik Puppet Module + Vagrant

We should support Composer ( http://getcomposer.org ) and add our repository to http://packagist.org/

We should also add a Vagrant/Puppet configuration. All files should be located under "misc/deploy".

I'd suggest to start with a Lucid64 base box ( http://files.vagrantup.com/lucid64.box ) and add different boxes later. Network Interface should be "Bridged"?

Required configurations/pacakges in Puppet:

  • PHP + all required extensions like GD, MySQL, Subversion, Git, Apache or Nignx, Composer, Vim :)
  • The Piwik project itself

Related Puppet links:

Tasks:

  • Nginx + PHP-FPM + Host nginx.piwik:8080
  • Apache2 + Mod PHP + Host apache.piwik
  • PHP Modules required by Piwik
  • PHPUnit (and other Tools like PHP_PMD, PHP_Depend, PHP_CodeCoverage, PHP_CodeSniffer, phpDox)
  • Git
  • Subversion
  • MySQL / PDO
  • Percona Tools
  • Vim
  • python-setuptools python-dev build-essential
  • Composer
  • easy_install
  • simplejson (via easy_install)
  • Once everything is done: Create a puppet-piwik module
  • Allow creation of multiple DB users
  • Blog post
  • FAQ, see ticket #3319
  • Create a repo "puppet-piwik" on Github and move source to this repository
  • Create a repo "piwik-dev-environment" on Github which contains all required Vagrant/Puppet files. Including all required Puppet modules (via Git submodule)
  • Composer detects only trunk / Jenkins does not sync our tags to Github

Change History (70)

comment:1 Changed 21 months ago by tsteur

(In [6722]) refs #3314 added composer.json

comment:2 Changed 21 months ago by vipsoft (robocoder)

  • Milestone set to 1.x - Piwik 1.x

I've published piwik/piwik to packagist. (Matt: see email for packagist credentials.) At minimum, we're reserving our namespace. Do you see using composer.json for Piwik's dependencies (currently in libs), or Piwik as a dependency (when included in another project)?

Contrary to http://getcomposer.org/doc/02-libraries.md#tags, it doesn't look like Packagist recognizes our tag (without the 'v' prefix). I've created an issue here: https://github.com/composer/packagist/issues/191

I'm a puppet n00b, but I want to help with a Precise64 box. I was planning on using a "Host-only" network interface, assuming the VM was for dev/test. "Bridged" makes sense if we want to provide users with a config that they can use to spin up a VM and test drive Piwik.

comment:3 Changed 21 months ago by tsteur

I created a Packagist account too :) Then I saw it's already added :) I'll delete my account again.

I'm not sure whether we should include other libs like Zend via composer. I see it more as a dependency. If we include Zend and other libs via composer, other developers would have to use composer too when checking out the repository. This would require PHP 5.3.

OK, I'll start with an Ubuntu base box and add a few packages to Puppet. It's easy to add a Precise64 box afterwards.

comment:4 Changed 21 months ago by tsteur

(In [6723]) refs #3314 added lucid64 vagrant base box

comment:5 Changed 21 months ago by tsteur

(In [6729]) refs #3314 a very first puppet configuration for Piwik. I will add the required modules later

comment:6 Changed 21 months ago by tsteur

(In [6730]) refs #3314 added precise64 base box

comment:7 Changed 21 months ago by tsteur

A list of available base boxes can be found here:

http://www.vagrantbox.es/

comment:8 Changed 21 months ago by tsteur

(In [6731]) refs #3314 change file permissions of config directory after checkout

comment:9 Changed 21 months ago by tsteur

(In [6732]) refs #3314 change file permissions of tmp directory after checkout

comment:10 Changed 21 months ago by tsteur

(In [6735]) refs #3314 configured nginx, piwik is running now but mysqli is still missing

comment:11 Changed 21 months ago by tsteur

(In [6736]) refs #3314 fix php, mysql naming conflict

comment:12 Changed 21 months ago by JulienM (JulienMoumne)

I have logged the task of writing a FAQ entry about this superb initiative in #3319

comment:13 Changed 21 months ago by tsteur

  • Description modified (diff)

comment:14 Changed 21 months ago by tsteur

(In [6759]) refs #3314 nginx.piwik:8080 is now fully working with nginx and php-fpm. piwik installation works

comment:15 Changed 21 months ago by tsteur

  • Description modified (diff)

comment:16 Changed 21 months ago by tsteur

  • Description modified (diff)

comment:17 Changed 21 months ago by tsteur

(In [6760]) refs #3314 added required packages for testing log analytics

comment:18 Changed 21 months ago by tsteur

  • Description modified (diff)

comment:19 Changed 21 months ago by tsteur

(In [6765]) refs #3314 include PHPUnit and lots of php qa tools

comment:20 Changed 21 months ago by tsteur

  • Description modified (diff)

comment:21 Changed 21 months ago by tsteur

(In [6766]) refs #3314 apach2 is up and running as well. both use the same piwik directoy. apache.piwik listens on port 80

comment:22 Changed 21 months ago by tsteur

  • Description modified (diff)

comment:23 Changed 21 months ago by tsteur

(In [6775]) refs #3314 removing the lucid64 vagrant configuration for the first

comment:24 Changed 21 months ago by tsteur

(In [6777]) refs #3314 removed deprecated files

comment:25 Changed 21 months ago by tsteur

(In [6778]) refs #3314 create user after repo checkout. the directory does not exist before

comment:26 Changed 21 months ago by tsteur

(In [6779]) refs #3314 install simplejson via pip and install those packages only if they are not already installed

comment:27 Changed 21 months ago by tsteur

(In [6780]) refs #3314 do not notify fpm, the fpm service is not created at this moment

comment:28 Changed 21 months ago by tsteur

  • Description modified (diff)

comment:29 Changed 21 months ago by tsteur

(In [6795]) refs #3314 added some requires to make sure the packages will be installed in the correct order. It still needs further improvements

comment:30 Changed 21 months ago by tsteur

(In [6796]) refs #3314 added percona toolkit package

comment:31 Changed 21 months ago by tsteur

  • Description modified (diff)

comment:32 Changed 21 months ago by tsteur

(In [6797]) refs #3314 added composer. usage: "php composer.phar ..."

comment:33 Changed 21 months ago by tsteur

  • Description modified (diff)

comment:34 Changed 21 months ago by tsteur

(In [6798]) refs #3314 added simple test

comment:35 Changed 21 months ago by tsteur

  • Description modified (diff)

comment:36 Changed 20 months ago by tsteur

(In [6809]) refs #3314 install composer only once and make it globally available

comment:37 Changed 20 months ago by tsteur

(In [6810]) refs #3314 removed users dependency

comment:38 Changed 20 months ago by tsteur

(In [6816]) refs #3314 adding a precise 32bit base box

comment:39 Changed 20 months ago by tsteur

(In [6817]) refs #3314 change owner of piwik directory

comment:40 Changed 20 months ago by tsteur

(In [6823]) refs #3314 added a class to create the user

comment:41 Changed 20 months ago by tsteur

(In [6825]) refs #3314 more docs

comment:42 Changed 20 months ago by tsteur

(In [6831]) refs #3314 moved user definition, this allows us to host several different Piwik versions

comment:43 Changed 20 months ago by tsteur

(In [6832]) refs #3314 added todo note

comment:44 Changed 20 months ago by tsteur

Are there any command line tools (bash, php, ...) to install/configure Piwik? To add users and websites and so on?

Would be great to have the possibility to configure Piwik totally with Puppet in a future release. Especially for a production environment.

Example could be:

piwik:install { path => '...' }
piwik:website {name =>'myexample1', url => 'example.org/1',  timezone => '', 'use_ecommerce' => true}
piwik:website {name =>'myexample2', url => 'example.org/2',  timezone => '', 'use_ecommerce' => false}
piwik:user {name => 'username1', websites => 'myexample1', access => 'view'}
piwik:user {name => 'username2', websites => 'myexample2', access => 'view'}

comment:45 Changed 20 months ago by matt (mattab)

It is the ticket: #1586 Headless install / command line piwik remote install.

Definitely would be nice to have and have as a scriptable piwik setup for several use cases!

comment:46 Changed 20 months ago by tsteur

(In [6833]) refs #3314 better description of the requirements

comment:47 Changed 20 months ago by tsteur

The piwik-puppet module is now available here:

https://github.com/piwik/puppet-piwik

Once I've created the "piwik-dev-environment" I'll delete the files in svn (misc/deploy)

comment:48 Changed 20 months ago by tsteur

  • Description modified (diff)

comment:49 Changed 20 months ago by tsteur

  • Description modified (diff)

The piwik-dev-environment is now available here:

https://github.com/piwik/piwik-dev-environment

Contains all available vagrant boxes, the Puppet configuration as well as all required modules

comment:50 Changed 20 months ago by tsteur

(In [6835]) refs #3314 removing the configuration from SVN repository. Those files are now on Github

comment:51 Changed 20 months ago by tsteur

Can someone follow the instructions from https://github.com/piwik/piwik-dev-environment and try it? Anthon?

comment:52 Changed 20 months ago by tsteur

  • Description modified (diff)

comment:53 Changed 20 months ago by vipsoft (robocoder)

Sure. I'll give it a spin later tonight.

comment:54 follow-up: Changed 20 months ago by vipsoft (robocoder)

Awesome work, Thomas!

Installing the pear modules took much longer than I expected. (Total runtime for vagrant > 50 minutes.)

As an alternative to modifying one's /etc/hosts file, maybe we could set up a couple of DNS A records both pointing to 192.168.33.10. (Maybe Matt could use one of the alternate piwik domains?)

Do the puppet scripts set up a MySQL user with sufficient privileges? (site.pp doesn't contain working credentials) I manually created a piwik user. (BTW I noticed something odd in the mysql.user table, e.g., SELECT COUNT(User) from mysql.user where User = ; returns 2 records.)

The www-data user's home directory is /var/www, it defaults to: AllowOverride FileInfo AuthConfig Limit Indexes. We should remove "Indexes" from the config (to reflect best practices).

comment:55 in reply to: ↑ 54 Changed 20 months ago by matt (mattab)

As an alternative to modifying one's /etc/hosts file, maybe we could set up a couple of DNS A records both pointing to 192.168.33.10. (Maybe Matt could use one of the alternate piwik domains?)

It would be nice not to have to edit hosts file indeed. Could we use a subdomain such as virtual.piwik.org ?

comment:56 Changed 20 months ago by tsteur

  • Description modified (diff)
  • Owner set to tsteur
  • Status changed from new to assigned

Thanks for testing, Anthon!

Yes, that's right. Without Pear modules it takes about 10 minutes, with Pear modules it takes about 20-30 minutes on my machine. Shall I remove some probably not needed QA Tools? It currently installs the following packages: https://github.com/rafaelfelix/puppet-phpqatools/blob/master/manifests/init.pp among a few others.

Great idea regarding set up a couple of DNS A records! Let me know the domains and I'll changed it.

Database user should work now. But only from localhost. I've updated dev-environment as well as puppet module. The created user was "piwik@%". That means it is possible to access the machine from any host. But "piwik@%" works only if there's also at least one "piwik@localhost" entry. This was not the case... I'll refactor the module to allow the creation of multiple DB users. For example "piwik@localhost" as well as "piwik@%", "piwik@192.168.1.2", ...
The "grants" are currently set to "all", see https://github.com/piwik/puppet-piwik/blob/master/manifests/db.pp#L45 . Shall I change it?

The www-data user's home directory should be "/var/www/piwik". I'll have a look into this. Do we need UserDir Module at all? Guess I have to overwrite the file.

comment:57 Changed 20 months ago by tsteur

  • Milestone changed from 1.x - Piwik 1.x to 1.8.4 - Piwik 1.8.4

comment:58 Changed 20 months ago by tsteur

It's now possible to define multiple initial database users within site.pp this way:

db_user => ['piwik@localhost' , 'piwik@192.168.0.0/255.255.0.0']

But I don't think we need the ability to connect from any host, do we? Some tools like "Navicat" are able to establish a connection via ssh. I think the actual problem is that MySQL doesn't listen to the external IP at the moment.

comment:59 Changed 20 months ago by vipsoft (robocoder)

I think we can leave the home dir as /var/www, and UserDir module as is. The closer we have it to the default, the more familiar it'll be to others, and at the same time, not further prolong the startup time.

I agree localhost access to the MySQL server is sufficient.

I'm going to take a look at the pear puppet module. It shouldn't be that slow.

comment:60 Changed 20 months ago by tsteur

  • Description modified (diff)

The home dir is now /var/www again (default directory).

comment:61 Changed 20 months ago by tsteur

FYI: Added possibility to define local vagrant settings, see step 3 https://github.com/piwik/piwik-dev-environment#installation

There's a gitignore on "vagrantconfig_local.yaml". So, any changes will be ignored (not commited). It is now possible for example to set your svn username/password there. See https://github.com/piwik/piwik-dev-environment/blob/master/vagrantconfig.yaml

Let me know if we need more settings. For example mount points (shared folders) and so on.

comment:62 Changed 20 months ago by JulienM (JulienMoumne)

I have successfully configured my IDE to sync Piwik source files via SFTP.

I had to perform several user rights adjustments so the Vagrant user had the rights to update Piwik source files :

  • add Vagrant user to the www-data group
  • perform a chmod g+w on all Piwik files

I was wondering if I was correct in using the Vagrant user in that case or if there was a better way.

The main idea is to be able to set-up a complete development environment using Vagrant.

comment:63 Changed 20 months ago by tsteur

Thanks Julien. Puppet will do this now for you when creating a new Piwik VM :)
I changed the owner of Piwik files to "vagrant:www-data" and added "vagrant" to group "www-data".

See https://github.com/piwik/piwik-dev-environment/commit/6fbc2eeb9aa5199163fe27ab63b65200333adb61

That's my main idea too ;)

comment:64 Changed 20 months ago by JulienM (JulienMoumne)

Because I am no sysadmin expert, I do not know the best practices to resolve the issue.

The commands I gave was to illustrate the issue and how I resolved it, but it may be 'quick and dirty'.

There are maybe several other ways to solve this issue such as :

  • adding a non-sudoer piwik user (to contrast with vagrant which is a sudoer)
  • using www-data instead of Vagrant?

comment:65 Changed 20 months ago by JulienM (JulienMoumne)

As a side note, when running phpunit tests, one needs to configure the [database_tests] config section to set

password = secure

It is possible to edit config.ini.php via SSH but it would be nice if it was possible to do so using the IDE.

The way permissions are currently set-up it is not possible with the vagrant user because of the rights associated to config.ini.php :

-rw-r--r-- 1 www-data www-data 1756 Sep  2 19:06 /var/www/piwik/config/config.ini.php

Another related issue is happening when running PHPUnit integration tests within the virtual machine.

The following error occurs :

SmartyException: unable to write to $compile_dir '/var/www/piwik/tmp/templates_c'. Be sure $compile_dir is writable by the web server user.

Because '/var/www/piwik/tmp/templates_c' is not writable by the Vagrant user :

drwxr-xr-x  2 www-data www-data 4096 Sep  2 19:06 templates_c/

comment:66 Changed 20 months ago by tsteur

Thanks for this feedback!

I created a new VM and verified config.ini.php is editable now with vagrant user because the files do now belong to the "vagrant" user. I changed this yesterday.

Using the "vagrant" user is ok and not "quick and dirty" I think.

comment:67 Changed 20 months ago by tsteur

  • Description modified (diff)

comment:68 Changed 20 months ago by tsteur

  • Description modified (diff)

Packagist recognizes our tag now. Version 1.8.3 is available. I've tried it and it works! Thanks for syncing our tags!

comment:69 Changed 20 months ago by matt (mattab)

Nice team work... Love the look of this Tasks list now ;-)

comment:70 Changed 18 months ago by matt (mattab)

  • Description modified (diff)
  • Resolution set to fixed
  • Status changed from assigned to closed

DONE! :-) See: Get started with Piwik development – Piwik Puppet Module + Vagrant

well done Thomas! this will be useful for the build server/jenkins :)

Note: See TracTickets for help on using tickets.