PiwikTracker.java

[exactt]

hi folks,

I transformed PiwikTracker.php to Piwiktracker.java and wanted to share the code for further improvement/fixes. There is still stuff missing but it works for me so far.

To track a (JSP-)page I use the following code:

<% PiwikTracker pt = new PiwikTracker(1, " http://localhost/piwik", request); ResponseData rd = pt.doTrackPageView("Tracker Test"); if(rd.getCookies().size() > 0) response.addCookie(rd.getCookies().get(rd.getCookies().size()-1)); %>

I am still a newbie to Piwik so I might have gotten something wrong...

cheers

[exactt]

Cookie Handling Helper Class

[exactt]

The PiwikTracker.php equivalent in Java

[vipsoft]

Looks good (except for some formatting inconsistencies). Can we use org.piwik and redistribute under a BSD license?

[matt]

Exactt, thanks for your submission. This is the first third party language we will support after PHP, good news!

Code review:

The code looks very close to the PHP version which is good as well. I noticed json_encode is not implemented? this is important since Custom Variables are sent json encoded to the tracker.

You can safely remove all XDEBUG related code.

Also, maybe you haven't used the latest 1.2 PiwikTracker to do the conversion. There are a few fixes and now the setIp and set date of visit work with token_auth, so they can be passed in getRequest() method. Implementation has slightly changed (ie. was finalized) in 1.2 around Custom Variables (eg. the parameter is called cvar not _cvar)

for readability, you can remove the PHP functions not implemented

---

Thanks for this, Java users from the piwik community will definitely appreciate. We can also let you know if there are further updates to PiwikTracker.php if you'd like to maintain

[exactt]

hi guys, thx for your comments!

@vipsoft: I removed the license part because it contained too many URLs. These kept me from uploading because the file was recognized as spam. Maybe someone can change the spam recognition setting so uploads with a handfull of URLs work... Please change the licensing and package as you prefer!

@matt: I will investigate the changes and upload a new version in the next couple of days. I would love to get informed about updates and will maintain the file as far as possible.

[vipsoft]

You can upload a .zip or .jar to get past the attachment filter.

[exactt]

here my refined calling code:

<% String pageTitle = "Tracker Test"; PiwikTracker pt = new PiwikTracker(1, " http://localhost/piwik", request); ResponseData rd = pt.doTrackPageView(pageTitle); if(rd.getCookies() != null && rd.getCookies().size() > 0)

response.addCookie(rd.getCookies().get(rd.getCookies().size()-1));

%> <title><%= pageTitle %></title>

[exactt]

again latest version. NPE fix + protected fields and methods. please test!

[matt]

exactt, thanks for the updates, it looks really cool.

Have you tested all the functions? Are you confident it all works?

If so, and because you are keen to maintain the lib, I propose that we commit this class to the SVN and link it from the Tracking API page (like we do for the PHP version). Also could you please post a simple forum post and then we can ask for testers on twitter and facebook, I'm sure there are many java fans using Piwik :)

Let me know your thoughts, great work anyway! Now on to Python, ruby :)

[exactt]

I just posted in the forum:  http://forum.piwik.org/read.php?2,73439

After some success reports we can take the next steps...

[matt]

exactt,

We posted the link on twitter as well so hopefully Java people will find it!

Maybe it is easier if you edit the ticket description and put the README there (full sample code to track a page view, other interesting notes).

[mheuser]

Hi folks, yesterday i have taken a look on the sources, because i need easy access to the trcker api from apache wicket, and it's a good start, but IMHO not yet production ready. J'am right now on doing some changes and will post it on my  github page in the next few hours.

The changes involve:

• Build & dependency resolution via maven
• http calls in a non blocking fashion via jetty client
• error logging via slf4j
• unit tests with testng
• a more "javaisher" handling of the api.

Background: I just started to know about piwik (from the heise.de newsticker) but have 10 years of java web development skills (jsp, wicket, gwt, vaadin, grails).

So far my few points. But without the start from exactt i would right now fight with the js api :) so thanks for the commit!

[matt]

mheuser, this sounds great. Please post a comment here when you have done work in your github, and when you are done please post the Zip here.

Ideally, we will include the Java API in the Piwik SVN and then give you access to SVN so you can maintain it (even though there shouldn't be much work, we might add new features in the API later).

[mheuser]

It's been a while, just want to update this ticket. I had to do some more important tasks since my last post, but i will post the code till Friday.

[mfochler]

hi, nice to read this ticket.

I just started working with the provided code. I'm really interested in the changed code, cause I use maven dependency management as well and don't want to make the changes you have listed before by myself. ;)

[matt]

mheuser, exactt, how is the PiwikTracker.java going? Can we submit it on SVN and link it from the product and documentation as the official Java client?

[knarf]

Thanks for the effort! They have been a very good starting point. Unfortunately I do not find the API well documented, i.e. which parameters are used with which data type so I was very happy to find this class.

We have also used the class in a project now but had to adjust them to our needs. The PiwikTracker.java attached here requires JRE 6 for cookie handling which is not feasible for all project setups - and has not been for ours. Furthermore it uses a synchronous approach which might block further processing if the PIWIK server is unavailable.

We have gone for an asynchronous approach and a handling w/o cookies (we are identifying unique visitors differently). I would provide a diff, but we are using PiwikTracker as POJO without any logic in it except building the URL.

We have found some small issues and here are changes that need to be done to URL building in PiwikTracker#getRequest(int) regardless of the project setup:

Parameter _refts requires epoch time in seconds, a Date was given. Java furthermore counts epoch times in milliseconds.

"&_refts=" + ( forcedDatetime != null ? forcedDatetime.getTime() : System.currentTimeMillis()) / 1000

The parameter forcedDateTime needs to be encoded in a way PHP's strtotime() can handle. We are using the following formatter and had no issues with it:

private static SimpleDateFormat SDF = new SimpleDateFormat("yyyyMMdd HH:mm:ssZ");
[..]
(forcedDatetime != null ? "&cdt=" + urlencode(SDF.format(forcedDatetime)) : "")

As the #empty(int) is a wrong method (returns true if value != 0), the resolution was not transmitted correctly. We have changed the line to:

(width > 0 && height > 0 ? "&res=" + width + "x" + height : "")

The "biggest" change affected custom variables. It fails with an IndexOutOfBoundsException because the position is given although there are no elements in the List. So we are "adding" the variables to the list rather than "putting" them in. I have no clue if the order of custom variables is that important. However as we are always calling the method in the same order, the order of the variables remain the same anyway. The second change affected the json_encode which should JSONObject instead JSONArray:

private List<List<Object>> visitorCustomVar;

public void setCustomVariable(Object name, Object value)
{
    if (visitorCustomVar == null) {
        visitorCustomVar = new ArrayList<List<Object>>();
    }
    List<Object> list = new ArrayList<Object>();
    list.add(name);
    list.add(value);
    visitorCustomVar.add(list);
}

private String json_encode(List<List<Object>> visitorCustomVar) throws JSONException
{
    JSONObject json = new JSONObject();
    int pos = 1;
    for (Iterator<List<Object>> iterator = visitorCustomVar.iterator(); iterator.hasNext();)
    {
        List<Object> objects = iterator.next();
        json.put(Integer.toString(pos++), objects);
    }
    return json.toString();
}

[exactt]

Thx for the valuable feedback. I adopted the changes except the last one.

Could one of the PHP developers shed some light on the custom variable thingy please!? Where can we find find the documentation regarding ordering etc.?

[matt]

I hear your complain about lack of documentation :) I think the best way to start is to look at the PHP Tracker, the code should be fairly easy to understand:  http://dev.piwik.org/trac/browser/trunk/libs/PiwikTracker/PiwikTracker.php

Regarding Custom Variables, here are the spec: {"1":["VAR NAME 1","VAR VALUE 1"],"4":["VAR NAME 4","VAR VALUE 4"]}

The order is not important but the [name, value] array must be indexed by the custom variable slot.

Hope it helps!

[matt]

Few more details:

• the name & value should be truncated each to 100 as a health check to limit URL length (I will add this in the PHP tracker soon)
• slot should be from 1 to 5 if you want to check and throw an exception if this isn't the case

[exactt]

the latest and greatest - all others can be ignored/deleted

[exactt]

just uploaded another version.

@knarf: what do you think? looking forward to your feedback!

@all: please test and report back! thx!

[matt]

(In [4350]) PiwikTracker

• doc improvements
• adding stubs for upcoming functionnality in Refs #2222
• Refs #2172 PiwikTracker.java will need a few new functions when this work is finalized (will ping you when new API is final)

[mfochler]

Mavenproject of the java implementation

[mfochler]

I added my maven project of the implementation to the ticket. I have done some java 1.5 changes and rewrite the hole java class in a more java stylish way. Please look into the code and give me feedback.

[JulienM]

In pom.xml,

<url>http://maven.apache.org</url>

should be

<url>http://piwik.org</url>

Is there a reason why org.json classes are included as project source files and not as a maven dependency ?

[matt]

(In [4378]) Fixes #2168

• Now crediting the right referrer for any goal conversion
• Referrer URLs, timestamp, and Campaign name & keyword (parsed from the landing page URL) are now stored in a first party cookie (JSON encoded)
• Added getters to the piwik.js to allow passing the values from the client, to store for later use in the PHP/Java Tracking API (paypal use case)
• Added integration tests and basic JS tests

Refs #2172

• Added a new API function: setAttributionInfo( $jsonEncoded ) which accepts JSON encoded array of 4 values (see implementation for details what to do with these values)
• 2 new parameters are: _rcn and _rck to pass to piwik the campaign name and campaign keyword that will be creditted for the Goal conversion (if a goal is converted)
• Also renamed setUrlReferer to setUrlReferrer -- Important that all public facing APIs use the proper writing

Refs #2222 Accurate Paypal tracking (or any other third party "after the fact" Goal conversion): implementation is now done, we must write some kind of guide and test

[matt]

As well as the previous comment, the Java client must be updated with the new functions:

• setVisitorId
• getVisitorId

All this code is pretty trivial hopefully you can reach an agreement soon for the Piwik Java client and we can start advertising it and commit it to SVN!

Thanks Java devs

[mfochler]

Replying to JulienM:

In pom.xml, {{{ <url> http://maven.apache.org</url> }}} should be {{{ <url>http://piwik.org</url> }}} Is there a reason why org.json classes are included as project source files and not as a maven dependency ?

The pom.xml change is trivial. The problem of the dependency is that I don't find a proper one.

The getVisitorId function is also trivial to add.

Is there a way to get access to svn or the get a path where I can upload my source? The attachments will get very confusing if we do the source providing in that way.

[JulienM]

have you tried from  mvnrepository.com ?

<dependency>
    <groupId>org.json</groupId>
    <artifactId>json</artifactId>
    <version>20090211</version>
</dependency>

we really need a decentralized VCS!

[matt]

It is no problem to get access to Piwik SVN to work on the PiwikTracker.java please contact us at hello att piwik.org with your request and we will add you!

[matt]

(In [4411]) Fixes #2299, refs #2172 (in case it is easier for you, now the function will accept a timestamp as well)

[mfochler]

Replying to matt:

It is no problem to get access to Piwik SVN to work on the PiwikTracker.java please contact us at hello att piwik.org with your request and we will add you!

I have send a mail to the given address but got now answer so far.

[matt]

mfochler, we found your email in spam folder sorry. You should now have access, per email.

[mfochler]

uploaded all Data with r4563 Sadly I have committed it with the wrong ticket number.

[matt]

Thanks Martin! To all Java developers in here, would you mind checking out Martin's code from  http://dev.piwik.org/svn/trunk/libs/PiwikTracker/java/ and give your feedback here?

Also, in order to add the Java lib in the Analytics tracking documentation, would you be able to prepare a 3-4 lines description of the tool, and add the code documentation "How to use it" in the Class header comment?

Thank you for the great work!

[mfochler]

I will try to create more documentation to the classes this week, but can not promise it ;)

[user08]

Hi,

I'm using your last code (from SVN), but the JSP code is not updated.

Can you add it please.

Thanks.

[user08]

Hello,

I tried to adapt the old JSP code to this one :

<%

String pageTitle = "Tracker Test"; SimplePiwikTracker pt = new SimplePiwikTracker(1, " http://localhost/piwik", request); ResponseData rd = pt.sendRequest(pt.getPageTrackURL(pageTitle)); if(rd.getCookies() != null && rd.getCookies().size() > 0)

response.addCookie(rd.getCookies().get(rd.getCookies().size()-1));

%>

But i have an exception in SimpleWikiTracker.java at this ligne :

this.setUrlReferer(request.getHeader("Referer"));

wich is in readRequestInfos fonction. It triggers an exception because i haven't "Referer" in my request (urlReferer=null)..

So what's the point? and what's the solution ?

Thank you :)

[user08]

Hi,

I found one error in SimplePiwikTracker.java , you must replace "user-agent" by "User-Agent", in all instances, to make it work ;)

[matt]

mfochler, thanks for your update in trunk!! Excellent class for Java developers.

In Piwik 1.5 (currently in  RC Testing), We have added Tracker API support for Ecommerce analytics tracking.

The Tracking API is now stable. The new functions are:

• void addEcommerceItem (string $sku, [string $name = false], [string $category = false], [float|int $price = false], [int $quantity = false])
• void doTrackEcommerceCartUpdate (float $grandTotal)
• void doTrackEcommerceOrder (string|int $orderId, float $grandTotal, [float $subTotal = false], [float $tax = false], [float $shipping = false], [float $discount = false])
• void setEcommerceView ([string $sku = false], [string $name = false], [string $category = false])

AS well as these new functions, the Tracking API functions added in 1.4 haven't been ported in the Java client:

• string getAttributionInfo ()
• void setAttributionInfo (string $jsonEncoded)

Finally, the following functions have been updated ( a new parameter added, for Custom Variables for scope "page" support)

• array|false getCustomVariable (int $id, [string $scope = 'visit'])
• void setCustomVariable (int $id, string $name, string $value, [string $scope = 'visit'])

(the last parameter "scope" was added)

The code for these functions is fairly straightforward (very small code) and don't introduce any new concept so it is mostly reusing existing code. The PHP client is at:  http://dev.piwik.org/trac/browser/trunk/libs/PiwikTracker/PiwikTracker.php and the auto generated doc at:  http://qa.piwik.org:8080/phpdocs/PiwikTracker/PiwikTracker.html

Thank you for your useful help so far! Hoping that someone can submit a patch which mfochler can then commit to trunk :-)

[mfochler]

(In [4848]) Workaround for the NullPointer in urlReferer, will set the referer to null. This has to be testet for other consequences. Changes both appearances of user-agent to the capital letter writing. refs #2172

[mfochler]

I will create a patch in near future.

In the new code of the piwikTracker the param "_ref" is used for the attributionInfo element 4. In the previous version the param "ref" was used by the same param as "urlref". I implemented it in this way. Is it right that the old param "ref" was never used? I'm really confused cause the attributionInfo element 3 is now the param "_refts" with was used by the 'forcedDatetime'. Please enlighten me so I can make the changes to be compatible with the older versions of piwik.

The other changes can be made easily.

I hope I can make the changes at the weekend.

[matt]

Summary:

• _ref is set to the content of the attribution first party cookie, which is the Full Referrer URL used to reach the site the "last" time (or the "first" time if customized)
• urlref is set to the Referrer URL of the current pageview
• I don't think "ref" was ever used? but maybe I'm wrong. Anyway it is not used now
• ForcedDateTime is only used in the parameter: {{{

(!empty($this->forcedDatetime) ? '&cdt=' . urlencode($this->forcedDatetime) : ) . }}}

There is no need to make it compatible with older version of Piwik, the version of PiwikTracker.java is only compatible with the latest version of Piwik only. Most Piwik users upgrade to the latest version all the time, so it is not a problem and would be too complicated to do (also hard to test etc.)

So, maybe just look at the PiwikTracker.php and follow the implementation in this class. we test a lot this class, so if the Java Tracker is using the same code more or less it should be good :)

Another change I didn't mention above:

• setUrlReferer is deprecated, the new function is setUrlReferrer - it is the same just fixing the typo

Thanks Martin for your help, let me know if you have more question!

[vipsoft]

(In [5079]) refs #2172 - add header

[matt]

Hello Java developers, any update on PiwikTracker.java? Thanks guys!

[rie miyazaki]

Hi! I have a problem with sending "custom variables" via "org.piwik.SimplePiwikTracker.java".

If we try to send more than one custom variable, only one custom variable of them will be shown on Piwik application.

The problem is that SimplePiwikTracker seems not to send custom variables with startindex 0, and this is wrong.

From the document of JavaScript tracking(http://piwik.org/docs/javascript-tracking/), custom variables must be sent with "1" as startindex.

So following format should be correct : {"1":["Modelname","BROWSER Firefox 8.0"], "2":["Prefered Markup","html"]}

But SimplePiwikTracker.java generates a different format : [["Modelname","BROWSER Firefox 8.0"],["Prefered Markup","html"]]

(Full example URL of Piwik Tracking ->  http://piwik.local/piwik.php?idsite=111&rec=1&apiv=1&url=http%3A%2F%2Flocalhost%3A8080%2Finm%2Fadesso%2F&_id=BD2F6502EA8ABE3D0FF7F9DC749A3114&cip=127.0.0.1&token_auth=34f6ee3769a9e5f6239573b6637b6336&pdf=true=false&res=400x700&cookie=false&_cvar=%5B%5B%22Modelname%22%2C%22BROWSER+Firefox+8.0%22%5D%2C%5B%22Prefered+Markup%22%2C%22html%22%5D%5D&rand=0.20301514584441116&action_name=http%3A%2F%2Flocalhost%3A8080%2Finm%2Fadesso%2F )

So I guess that Piwik application doesn't parse all custom variables correctly.

I tried to fix this issue.

Class : org.piwik.SimplePiwikTracker.java Method : public final String getGeneralQuery() (Line:421) Line 461 - 467

(Now)

                       final JSONArray json = new JSONArray();
            for (final Map.Entry<String, String> entry : this.customVar.entrySet()) {
                final List<String> list = new ArrayList<String>();
                list.add(entry.getKey());
                list.add(entry.getValue());
                json.put(list);
            }
            withCustomVar = this.addParameter(withCustomData, "_cvar", json.toString());

(Should be)

        	 final JSONObject json = new JSONObject();
            int i=1;
            for (final Map.Entry<String, String> entry : this.customVar.entrySet()) {
               
            	 final JSONArray jsonArray = new JSONArray();
            	 jsonArray.put(entry.getKey());
            	 jsonArray.put(entry.getValue());
            	 
            	 try 
            	 {
					json.put(Integer.toString(i), jsonArray);
				} catch (JSONException e) {
					e.printStackTrace();
				}
            	 i++;

            }
            withCustomVar = this.addParameter(withCustomData, "_cvar", json.toString());

Can you apply the fix, and release the update version for us ?

Thanks a lot for help! Rie

[rie miyazaki]

INFO :

We are using the JAVA tracker API from :  http://dev.piwik.org/trac/browser/tags/1.6/libs/PiwikTracker/java