Opened 3 years ago

Closed 2 years ago

Last modified 23 months ago

#2172 closed New feature (fixed)

Java Tracking Client V1 - PiwikTracker.java

Reported by: exactt Owned by:
Priority: major Milestone: 1.x - Piwik 1.x
Component: Core Keywords: PiwikTracker Java
Cc: Sensitive: no

Description

hi folks,

I transformed PiwikTracker.php to Piwiktracker.java and wanted to share the code for further improvement/fixes. There is still stuff missing but it works for me so far.

To track a (JSP-)page I use the following code:

<% PiwikTracker pt = new PiwikTracker(1, "http://localhost/piwik", request);
ResponseData rd = pt.doTrackPageView("Tracker Test");
if(rd.getCookies().size() > 0)
response.addCookie(rd.getCookies().get(rd.getCookies().size()-1));
%>

I am still a newbie to Piwik so I might have gotten something wrong...

cheers

Attachments (6)

ResponseData.java (1.7 KB) - added by exactt 3 years ago.
Cookie Handling Helper Class
PiwikTracker.java (19.1 KB) - added by exactt 3 years ago.
The PiwikTracker.php equivalent in Java
piwik.2.zip (64.0 KB) - added by exactt 3 years ago.
again latest version. NPE fix + protected fields and methods. please test!
piwik.jar (13.7 KB) - added by exactt 3 years ago.
the latest and greatest - all others can be ignored/deleted
JavaPiwikTracker.tar.gz (65.8 KB) - added by mfochler 3 years ago.
Mavenproject of the java implementation
piwik_java_plugin_1.0.2.zip (43.9 KB) - added by klaus 2 years ago.
New version of PIWIK JAVA plugin.

Download all attachments as: .zip

Change History (64)

Changed 3 years ago by exactt

Cookie Handling Helper Class

Changed 3 years ago by exactt

The PiwikTracker.php equivalent in Java

comment:1 Changed 3 years ago by vipsoft (robocoder)

Looks good (except for some formatting inconsistencies). Can we use org.piwik and redistribute under a BSD license?

comment:2 Changed 3 years ago by matt (mattab)

Exactt, thanks for your submission. This is the first third party language we will support after PHP, good news!

Code review:

The code looks very close to the PHP version which is good as well. I noticed json_encode is not implemented? this is important since Custom Variables are sent json encoded to the tracker.

You can safely remove all XDEBUG related code.

Also, maybe you haven't used the latest 1.2 PiwikTracker to do the conversion. There are a few fixes and now the setIp and set date of visit work with token_auth, so they can be passed in getRequest() method. Implementation has slightly changed (ie. was finalized) in 1.2 around Custom Variables (eg. the parameter is called cvar not _cvar)

for readability, you can remove the PHP functions not implemented

---

Thanks for this, Java users from the piwik community will definitely appreciate. We can also let you know if there are further updates to PiwikTracker.php if you'd like to maintain

comment:3 Changed 3 years ago by exactt

hi guys, thx for your comments!

@vipsoft: I removed the license part because it contained too many URLs.
These kept me from uploading because the file was recognized as spam. Maybe someone can change the spam recognition setting so uploads with a handfull of URLs work...
Please change the licensing and package as you prefer!

@matt: I will investigate the changes and upload a new version in the next couple of days.
I would love to get informed about updates and will maintain the file as far as possible.

comment:4 Changed 3 years ago by vipsoft (robocoder)

You can upload a .zip or .jar to get past the attachment filter.

comment:5 Changed 3 years ago by exactt

here my refined calling code:

<% String pageTitle = "Tracker Test";
PiwikTracker pt = new PiwikTracker(1, "http://localhost/piwik", request);
ResponseData rd = pt.doTrackPageView(pageTitle);
if(rd.getCookies() != null && rd.getCookies().size() > 0)

response.addCookie(rd.getCookies().get(rd.getCookies().size()-1));

%>
<title><%= pageTitle %></title>

Changed 3 years ago by exactt

again latest version. NPE fix + protected fields and methods. please test!

comment:6 Changed 3 years ago by matt (mattab)

exactt, thanks for the updates, it looks really cool.

Have you tested all the functions? Are you confident it all works?

If so, and because you are keen to maintain the lib, I propose that we commit this class to the SVN and link it from the Tracking API page (like we do for the PHP version).
Also could you please post a simple forum post and then we can ask for testers on twitter and facebook, I'm sure there are many java fans using Piwik :)

Let me know your thoughts, great work anyway! Now on to Python, ruby :)

comment:7 Changed 3 years ago by exactt

I just posted in the forum: http://forum.piwik.org/read.php?2,73439

After some success reports we can take the next steps...

comment:8 Changed 3 years ago by matt (mattab)

exactt,

We posted the link on twitter as well so hopefully Java people will find it!

Maybe it is easier if you edit the ticket description and put the README there (full sample code to track a page view, other interesting notes).

comment:9 Changed 3 years ago by mheuser

Hi folks,
yesterday i have taken a look on the sources, because i need easy access to the trcker api from apache wicket, and it's a good start, but IMHO not yet production ready.
J'am right now on doing some changes and will post it on my github page in the next few hours.

The changes involve:

  • Build & dependency resolution via maven
  • http calls in a non blocking fashion via jetty client
  • error logging via slf4j
  • unit tests with testng
  • a more "javaisher" handling of the api.

Background:
I just started to know about piwik (from the heise.de newsticker) but have 10 years of java web development skills (jsp, wicket, gwt, vaadin, grails).

So far my few points. But without the start from exactt i would right now fight with the js api :) so thanks for the commit!

comment:10 Changed 3 years ago by matt (mattab)

mheuser, this sounds great. Please post a comment here when you have done work in your github, and when you are done please post the Zip here.

Ideally, we will include the Java API in the Piwik SVN and then give you access to SVN so you can maintain it (even though there shouldn't be much work, we might add new features in the API later).

comment:11 Changed 3 years ago by mheuser

It's been a while, just want to update this ticket. I had to do some more important tasks since my last post, but i will post the code till Friday.

comment:12 Changed 3 years ago by vipsoft (robocoder)

  • Milestone set to 1.3 - Piwik 1.3

comment:13 Changed 3 years ago by mfochler

hi, nice to read this ticket.

I just started working with the provided code. I'm really interested in the changed code, cause I use maven dependency management as well and don't want to make the changes you have listed before by myself. ;)

comment:14 Changed 3 years ago by matt (mattab)

mheuser, exactt, how is the PiwikTracker.java going? Can we submit it on SVN and link it from the product and documentation as the official Java client?

comment:15 in reply to: ↑ description Changed 3 years ago by knarf

Thanks for the effort! They have been a very good starting point. Unfortunately I do not find the API well documented, i.e. which parameters are used with which data type so I was very happy to find this class.

We have also used the class in a project now but had to adjust them to our needs. The PiwikTracker.java attached here requires JRE 6 for cookie handling which is not feasible for all project setups - and has not been for ours. Furthermore it uses a synchronous approach which might block further processing if the PIWIK server is unavailable.

We have gone for an asynchronous approach and a handling w/o cookies (we are identifying unique visitors differently). I would provide a diff, but we are using PiwikTracker as POJO without any logic in it except building the URL.

We have found some small issues and here are changes that need to be done to URL building in PiwikTracker#getRequest(int) regardless of the project setup:

Parameter _refts requires epoch time in seconds, a Date was given. Java furthermore counts epoch times in milliseconds.

"&_refts=" + ( forcedDatetime != null ? forcedDatetime.getTime() : System.currentTimeMillis()) / 1000

The parameter forcedDateTime needs to be encoded in a way PHP's strtotime() can handle. We are using the following formatter and had no issues with it:

private static SimpleDateFormat SDF = new SimpleDateFormat("yyyyMMdd HH:mm:ssZ");
[..]
(forcedDatetime != null ? "&cdt=" + urlencode(SDF.format(forcedDatetime)) : "")

As the #empty(int) is a wrong method (returns true if value != 0), the resolution was not transmitted correctly. We have changed the line to:

(width > 0 && height > 0 ? "&res=" + width + "x" + height : "")

The "biggest" change affected custom variables. It fails with an IndexOutOfBoundsException because the position is given although there are no elements in the List. So we are "adding" the variables to the list rather than "putting" them in. I have no clue if the order of custom variables is that important. However as we are always calling the method in the same order, the order of the variables remain the same anyway. The second change affected the json_encode which should JSONObject instead JSONArray:

private List<List<Object>> visitorCustomVar;

public void setCustomVariable(Object name, Object value)
{
    if (visitorCustomVar == null) {
        visitorCustomVar = new ArrayList<List<Object>>();
    }
    List<Object> list = new ArrayList<Object>();
    list.add(name);
    list.add(value);
    visitorCustomVar.add(list);
}

private String json_encode(List<List<Object>> visitorCustomVar) throws JSONException
{
    JSONObject json = new JSONObject();
    int pos = 1;
    for (Iterator<List<Object>> iterator = visitorCustomVar.iterator(); iterator.hasNext();)
    {
        List<Object> objects = iterator.next();
        json.put(Integer.toString(pos++), objects);
    }
    return json.toString();
}

comment:16 Changed 3 years ago by exactt

Thx for the valuable feedback. I adopted the changes except the last one.

Could one of the PHP developers shed some light on the custom variable thingy please!? Where can we find find the documentation regarding ordering etc.?

comment:17 Changed 3 years ago by matt (mattab)

I hear your complain about lack of documentation :) I think the best way to start is to look at the PHP Tracker, the code should be fairly easy to understand: http://dev.piwik.org/trac/browser/trunk/libs/PiwikTracker/PiwikTracker.php

Regarding Custom Variables, here are the spec:
{"1":["VAR NAME 1","VAR VALUE 1"],"4":["VAR NAME 4","VAR VALUE 4"]}

The order is not important but the [name, value] array must be indexed by the custom variable slot.

Hope it helps!

comment:18 Changed 3 years ago by matt (mattab)

Few more details:

  • the name & value should be truncated each to 100 as a health check to limit URL length (I will add this in the PHP tracker soon)
  • slot should be from 1 to 5 if you want to check and throw an exception if this isn't the case

Changed 3 years ago by exactt

the latest and greatest - all others can be ignored/deleted

comment:19 Changed 3 years ago by exactt

just uploaded another version.

@knarf: what do you think? looking forward to your feedback!

@all: please test and report back! thx!

comment:20 Changed 3 years ago by matt (mattab)

(In [4350]) PiwikTracker

  • doc improvements
  • adding stubs for upcoming functionnality in Refs #2222
  • Refs #2172 PiwikTracker.java will need a few new functions when this work is finalized (will ping you when new API is final)

Changed 3 years ago by mfochler

Mavenproject of the java implementation

comment:21 Changed 3 years ago by mfochler

I added my maven project of the implementation to the ticket. I have done some java 1.5 changes and rewrite the hole java class in a more java stylish way. Please look into the code and give me feedback.

comment:22 follow-up: Changed 3 years ago by JulienM (JulienMoumne)

In pom.xml,

<url>http://maven.apache.org</url>

should be

<url>http://piwik.org</url>

Is there a reason why org.json classes are included as project source files and not as a maven dependency ?

comment:23 Changed 3 years ago by matt (mattab)

(In [4378]) Fixes #2168

  • Now crediting the right referrer for any goal conversion
  • Referrer URLs, timestamp, and Campaign name & keyword (parsed from the landing page URL) are now stored in a first party cookie (JSON encoded)
  • Added getters to the piwik.js to allow passing the values from the client, to store for later use in the PHP/Java Tracking API (paypal use case)
  • Added integration tests and basic JS tests

Refs #2172

  • Added a new API function: setAttributionInfo( $jsonEncoded ) which accepts JSON encoded array of 4 values (see implementation for details what to do with these values)
  • 2 new parameters are: _rcn and _rck to pass to piwik the campaign name and campaign keyword that will be creditted for the Goal conversion (if a goal is converted)
  • Also renamed setUrlReferer to setUrlReferrer -- Important that all public facing APIs use the proper writing

Refs #2222 Accurate Paypal tracking (or any other third party "after the fact" Goal conversion): implementation is now done, we must write some kind of guide and test

comment:24 Changed 3 years ago by matt (mattab)

As well as the previous comment, the Java client must be updated with the new functions:

  • setVisitorId
  • getVisitorId

All this code is pretty trivial hopefully you can reach an agreement soon for the Piwik Java client and we can start advertising it and commit it to SVN!

Thanks Java devs

comment:25 in reply to: ↑ 22 Changed 3 years ago by mfochler

Replying to JulienM:

In pom.xml,

<url>http://maven.apache.org</url>

should be

<url>http://piwik.org</url>

Is there a reason why org.json classes are included as project source files and not as a maven dependency ?

The pom.xml change is trivial. The problem of the dependency is that I don't find a proper one.

The getVisitorId function is also trivial to add.

Is there a way to get access to svn or the get a path where I can upload my source? The attachments will get very confusing if we do the source providing in that way.

comment:26 Changed 3 years ago by JulienM (JulienMoumne)

have you tried from mvnrepository.com ?

<dependency>
    <groupId>org.json</groupId>
    <artifactId>json</artifactId>
    <version>20090211</version>
</dependency>

we really need a decentralized VCS!

comment:27 follow-up: Changed 3 years ago by matt (mattab)

It is no problem to get access to Piwik SVN to work on the PiwikTracker.java please contact us at hello att piwik.org with your request and we will add you!

comment:28 Changed 3 years ago by matt (mattab)

(In [4411]) Fixes #2299, refs #2172 (in case it is easier for you, now the function will accept a timestamp as well)

comment:29 in reply to: ↑ 27 Changed 3 years ago by mfochler

Replying to matt:

It is no problem to get access to Piwik SVN to work on the PiwikTracker.java please contact us at hello att piwik.org with your request and we will add you!

I have send a mail to the given address but got now answer so far.

comment:30 Changed 3 years ago by matt (mattab)

mfochler, we found your email in spam folder sorry. You should now have access, per email.

comment:31 Changed 3 years ago by mfochler

uploaded all Data with r4563 Sadly I have committed it with the wrong ticket number.

comment:32 Changed 3 years ago by matt (mattab)

Thanks Martin! To all Java developers in here, would you mind checking out Martin's code from http://dev.piwik.org/svn/trunk/libs/PiwikTracker/java/ and give your feedback here?

Also, in order to add the Java lib in the Analytics tracking documentation, would you be able to prepare a 3-4 lines description of the tool, and add the code documentation "How to use it" in the Class header comment?

Thank you for the great work!

comment:33 Changed 3 years ago by mfochler

I will try to create more documentation to the classes this week, but can not promise it ;)

comment:34 Changed 3 years ago by user08

Hi,

I'm using your last code (from SVN), but the JSP code is not updated.

Can you add it please.

Thanks.

comment:35 Changed 3 years ago by user08

Hello,

I tried to adapt the old JSP code to this one :

<%

String pageTitle = "Tracker Test";
SimplePiwikTracker pt = new SimplePiwikTracker(1, "http://localhost/piwik", request);
ResponseData rd = pt.sendRequest(pt.getPageTrackURL(pageTitle));
if(rd.getCookies() != null && rd.getCookies().size() > 0)

response.addCookie(rd.getCookies().get(rd.getCookies().size()-1));

%>

But i have an exception in SimpleWikiTracker.java at this ligne :

this.setUrlReferer(request.getHeader("Referer"));

wich is in readRequestInfos fonction. It triggers an exception because i haven't "Referer" in my request (urlReferer=null)..

So what's the point? and what's the solution ?

Thank you :)

comment:36 Changed 3 years ago by user08

Hi,

I found one error in SimplePiwikTracker.java , you must replace "user-agent" by "User-Agent", in all instances, to make it work ;)

comment:37 Changed 3 years ago by matt (mattab)

mfochler, thanks for your update in trunk!! Excellent class for Java developers.

In Piwik 1.5 (currently in RC Testing), We have added Tracker API support for Ecommerce analytics tracking.

The Tracking API is now stable. The new functions are:

  • void addEcommerceItem (string $sku, [string $name = false], [string $category = false], [float|int $price = false], [int $quantity = false])
  • void doTrackEcommerceCartUpdate (float $grandTotal)
  • void doTrackEcommerceOrder (string|int $orderId, float $grandTotal, [float $subTotal = false], [float $tax = false], [float $shipping = false], [float $discount = false])
  • void setEcommerceView ([string $sku = false], [string $name = false], [string $category = false])

AS well as these new functions, the Tracking API functions added in 1.4 haven't been ported in the Java client:

  • string getAttributionInfo ()
  • void setAttributionInfo (string $jsonEncoded)

Finally, the following functions have been updated ( a new parameter added, for Custom Variables for scope "page" support)

  • array|false getCustomVariable (int $id, [string $scope = 'visit'])
  • void setCustomVariable (int $id, string $name, string $value, [string $scope = 'visit'])

(the last parameter "scope" was added)

The code for these functions is fairly straightforward (very small code) and don't introduce any new concept so it is mostly reusing existing code. The PHP client is at: http://dev.piwik.org/trac/browser/trunk/libs/PiwikTracker/PiwikTracker.php and the auto generated doc at: http://qa.piwik.org:8080/phpdocs/PiwikTracker/PiwikTracker.html

Thank you for your useful help so far! Hoping that someone can submit a patch which mfochler can then commit to trunk :-)

comment:38 Changed 3 years ago by matt (mattab)

  • Milestone changed from 1.5 - Piwik 1.5 to 1.x - Piwik 1.x

comment:39 Changed 3 years ago by mfochler

(In [4848]) Workaround for the NullPointer in urlReferer, will set the referer to null. This has to be testet for other consequences. Changes both appearances of user-agent to the capital letter writing. refs #2172

comment:40 Changed 3 years ago by mfochler

I will create a patch in near future.

In the new code of the piwikTracker the param "_ref" is used for the attributionInfo element 4.
In the previous version the param "ref" was used by the same param as "urlref". I implemented it in this way. Is it right that the old param "ref" was never used? I'm really confused cause the attributionInfo element 3 is now the param "_refts" with was used by the 'forcedDatetime'. Please enlighten me so I can make the changes to be compatible with the older versions of piwik.

The other changes can be made easily.

I hope I can make the changes at the weekend.

comment:41 Changed 3 years ago by matt (mattab)

Summary:

  • _ref is set to the content of the attribution first party cookie, which is the Full Referrer URL used to reach the site the "last" time (or the "first" time if customized)
  • urlref is set to the Referrer URL of the current pageview
  • I don't think "ref" was ever used? but maybe I'm wrong. Anyway it is not used now
  • ForcedDateTime is only used in the parameter: {{{

(!empty($this->forcedDatetime) ? '&cdt=' . urlencode($this->forcedDatetime) : ) .
}}}

There is no need to make it compatible with older version of Piwik, the version of PiwikTracker.java is only compatible with the latest version of Piwik only. Most Piwik users upgrade to the latest version all the time, so it is not a problem and would be too complicated to do (also hard to test etc.)

So, maybe just look at the PiwikTracker.php and follow the implementation in this class. we test a lot this class, so if the Java Tracker is using the same code more or less it should be good :)

Another change I didn't mention above:

  • setUrlReferer is deprecated, the new function is setUrlReferrer - it is the same just fixing the typo

Thanks Martin for your help, let me know if you have more question!

comment:42 Changed 3 years ago by vipsoft (robocoder)

(In [5079]) refs #2172 - add header

comment:43 Changed 3 years ago by matt (mattab)

Hello Java developers, any update on PiwikTracker.java? Thanks guys!

comment:44 Changed 2 years ago by rie miyazaki

Hi! I have a problem with sending "custom variables" via "org.piwik.SimplePiwikTracker.java".

If we try to send more than one custom variable, only one custom variable of them will be shown on Piwik application.

The problem is that SimplePiwikTracker seems not to send custom variables with startindex 0, and this is wrong.

From the document of JavaScript tracking(http://piwik.org/docs/javascript-tracking/), custom variables must be sent with "1" as startindex.

So following format should be correct :
{"1":["Modelname","BROWSER Firefox 8.0"], "2":["Prefered Markup","html"]}

But SimplePiwikTracker.java generates a different format :
Modelname","BROWSER Firefox 8.0"],["Prefered Markup","html?

(Full example URL of Piwik Tracking -> http://piwik.local/piwik.php?idsite=111&rec=1&apiv=1&url=http%3A%2F%2Flocalhost%3A8080%2Finm%2Fadesso%2F&_id=BD2F6502EA8ABE3D0FF7F9DC749A3114&cip=127.0.0.1&token_auth=34f6ee3769a9e5f6239573b6637b6336&pdf=true=false&res=400x700&cookie=false&_cvar=%5B%5B%22Modelname%22%2C%22BROWSER+Firefox+8.0%22%5D%2C%5B%22Prefered+Markup%22%2C%22html%22%5D%5D&rand=0.20301514584441116&action_name=http%3A%2F%2Flocalhost%3A8080%2Finm%2Fadesso%2F )

So I guess that Piwik application doesn't parse all custom variables correctly.

I tried to fix this issue.

Class : org.piwik.SimplePiwikTracker.java
Method : public final String getGeneralQuery() (Line:421)
Line 461 - 467

(Now)

                       final JSONArray json = new JSONArray();
            for (final Map.Entry<String, String> entry : this.customVar.entrySet()) {
                final List<String> list = new ArrayList<String>();
                list.add(entry.getKey());
                list.add(entry.getValue());
                json.put(list);
            }
            withCustomVar = this.addParameter(withCustomData, "_cvar", json.toString());

(Should be)

        	 final JSONObject json = new JSONObject();
            int i=1;
            for (final Map.Entry<String, String> entry : this.customVar.entrySet()) {
               
            	 final JSONArray jsonArray = new JSONArray();
            	 jsonArray.put(entry.getKey());
            	 jsonArray.put(entry.getValue());
            	 
            	 try 
            	 {
					json.put(Integer.toString(i), jsonArray);
				} catch (JSONException e) {
					e.printStackTrace();
				}
            	 i++;

            }
            withCustomVar = this.addParameter(withCustomData, "_cvar", json.toString());

Can you apply the fix, and release the update version for us ?

Thanks a lot for help!
Rie

comment:45 Changed 2 years ago by rie miyazaki

INFO :

We are using the JAVA tracker API from : http://dev.piwik.org/trac/browser/tags/1.6/libs/PiwikTracker/java

Changed 2 years ago by klaus

New version of PIWIK JAVA plugin.

comment:46 Changed 2 years ago by klaus

I improved the existing official PIWIK JAVA connector:

  • changed JDK to 1.6 in pom.xml
  • added page custom variables
  • fixed visitor custom variables
  • added comments and made it generally more Checkstyle-compliant
  • renamed methods getDownloadTrackURL, getPageTrackURL and set the others to deprecated (was probably a typo)
  • fixed tests
  • renamed methods (naming conventions)

comment:47 Changed 2 years ago by matt (mattab)

@klaus, thank you for your patch and improvements!

@mfochler would you be by any chance available to review the changes and commit to SVN?

comment:48 Changed 2 years ago by thivnor

Hi I've been using piwik for awhile but just recently tried using this Java code. Things are just about working, but there is one odd behavior. I'm not sure if what I'm seeing is a bug or user-error.

Most of my tracking is done by putting the normal JavaScript code into an HTML page. Now I want to add additional tracking inside my Java code which runs server-side. I'm specifically interested in goal-level tracking: when they hit the homepage they have achieved goal #1. When they get into the deeper part where my Java code is running, they should achieve goal #2.

My problem is that Piwik isn't recognizing these two goals as being achieved by the same visitor. In the database, specifically piwik_log_conversion, the two goals are logged with different idvisit and idvisitor values. Also, for Goal #2, which is recoded via Java, the location_continent is 'unk' and location_country is 'xx' So somewhere the visitor tracking is breaking down.

Currently I'm in development, so the IP address of the requests are the same. That is, both the JavaScript and Java code are executing from the same computer.

I looked into the code, and followed along with a debugging proxy server. The Java code is forwarding the "piwik_visitor" cookie. But the actual cookies I have in my browser are _pk_id.6.1fff and _pk_ses.6.1fff. I modified the Java code so that _all_ cookies are forwarded, and can confirm from the debugging proxy server that they are being passed along. I was careful to ensure the "cookie" URL parameter was set to "true." Even with these changes, however, piwik is still not tracking the goals as having been completed by the same visitor.

My server is up-to-date, at v1.7.1.

This code is called a 'plug-in' but I'm assuming I don't need to actually install a plug-in on the server. All I need to use is this Java code which makes a GET request to piwik.php .. correct?

Is this a bug? User error? Any suggestions on how to address it?

Todd Hivnor

comment:49 Changed 2 years ago by matt (mattab)

This code is called a 'plug-in' but I'm assuming I don't need to actually install a plug-in on the server. All I need to use is this Java code which makes a GET request to piwik.php .. correct?

Yes correct.. We should really put up a documentation for the JAVA client in http://piwik.org/docs/tracking-api - if anyone would like to help with this it would be great.

Is this a bug? User error? Any suggestions on how to address it?

  • Are you also setting/forcing the visitor IP correctly?
  • Can you try to do: setVisitorId( this.getVisitorId() ) This might help somehow.

Also to debug the Piwik.php request to find out more about th elogic piwik is doing, you can see: http://piwik.org/docs/tracking-api/reference/#toc-debugging-the-tracking-api-requests

comment:50 Changed 2 years ago by thivnor

Matt, you pointed me in the right direction. I think I have found a bug.

When the SimplePiwikTracker object is created, it sets the visitorID to a random string:

this.setVisitorId(this.md5(UUID.randomUUID().toString()).substring(0, 16));

This is never changed. I wrote up some quick code to examine the cookies and extract the visitorId:

/**
 * Sets the id of the requester based on the cookie in the request
 */
public final void setVisitorId(HttpServletRequest request) {
	
	Cookie[] cookies = request.getCookies();
	if (cookies == null || cookies.length == 0) { 
		LOG.error("Can't set setVisitorId from HttpServletRequest, there are no cookies");
		return;
	}
	for (Cookie cookie : cookies) {
		if (cookie.getName().startsWith("_pk_id")) {
			String[] parts = cookie.getValue().split("\\.");
			if (parts == null || parts.length == 0) {
				LOG.error("Can't set setVisitorId from HttpServletRequest, failed to parse cookie value: " + cookie.getValue());
			} else {
				this.setVisitorId(parts[0]);
				LOG.debug("Set visitorId to " + this.getVisitorId());
				return;
			}
		}
	}
	LOG.error("Can't set setVisitorId from HttpServletRequest, no cookies start with _pk_id");
}

After invoking this function, things seem to work.

Someone who is more familiar with the code should definitely review this, I'm not at all familiar with piwik internals. But the code as written seems to make two key mistakes:

  • Forwarding the cookie named "piwik_visitor" but not cookies like "_pk_id.6.1fff" and "_pk_ses.6.1fff"
  • Setting the visitorId to a random string.

I'm using @version 1.0.4. There is a comment "cookiesupport removed cause was not used" Perhaps this functionality used to be present but was removed by mistake?

comment:52 Changed 2 years ago by JulienM (JulienMoumne)

This constructor won't actually help you. It only looks for the piwik_visitor cookie.

With the PHP PiwikTracker, the functionality you are looking for is getVisitorId : http://dev.piwik.org/trac/browser/trunk/libs/PiwikTracker/PiwikTracker.php?desc=1#L570.

This method looks for cookies which contain the visitor id.

Have a look at the getVisitorId in the JAVA PiwikTracker : http://dev.piwik.org/trac/browser/trunk/libs/PiwikTracker/java/src/main/java/org/piwik/SimplePiwikTracker.java?desc=1#L115.

It doesn't look for cookies.

This method should be rewritten to provide the same level of functionality as the PHP PiwikTracker. Otherwise, users will be confused.

To further normalize both PiwikTrackers, there should also be only one JAVA constructor (which doesn't look for the 'piwik_visitor' cookie).

comment:53 Changed 2 years ago by thivnor

I compared the Java PiwikTracker class against the PHP version and see many differences. Crucially, the Java class uses a different cookie name, and lacks logic for extracting the visitor ID from the cookie. It appears this logic was simply not translated correctly in the initial check-in, and has not been corrected since. In short, visitor tracking seems totally broken.

The Java code lacks the convenient "doTrackPageView" method which is 1) Included in the PHP version 2) useful and 3) described in the original bug request.

The Java code adds an interface for the PiwikTracker class, but it is unused. I don't see a reason for it to exist.

I haven't tested this but strongly suspect the code will not play nicely behind a proxy server like the Apache HTTPd server. I suspect it will record the IP address of the proxy server not the end-user. It is fairly common for Java services to be implemented on a different server, and made available to end-users via proxy. That is certainly how I do things. To address this I would replace calls like getRemoteAddr(), which are not proxy-aware, with a proxy-aware equivalent.

I'm not sure what the overall process for this group is. This particular bug report has an empty "Owned By" field. The original submitter and author is exactt, but he hasn't posted here in over a year. So, I'm hesitant to spend time making fixes and submitting them here. But it feels to me that the code should be brought into line with the PHP version. I could do this work, but don't want to step on anyone's toes, and don't want to do the work if it won't get reviewed / integrated. FYI- I am a solid Java developer, over 10 years now. My PHP is not strong, but I can make sense of the existing code.

So ... opinions on next steps?

comment:54 Changed 2 years ago by matt (mattab)

  • Resolution set to fixed
  • Status changed from new to closed

thivnor, thanks for your message.

The existing code probably works for some use cases but it was created a while back, and we have regularly updated the PHP client since. However the php client is now stable and will stay so.. it is therefore a good time to revisit and finalize the Java client!

Would you be interested to work on it and submit a 1.0 release synced with PHP client in terms of API?

Ideally you would submit a patch, see: http://piwik.org/participate/development-process/#toc-how-to-submit-a-patch

If you are interested to maintain it, we could also give you SVN access, see the link for explanation of our dev process.

It would be great to have this code updated in trunk. Also it would be nice if we could auto generate the Java API doc similarly to our automated PHP doc.

I propose that you create a new ticket called "Java Tracking client 1.0" as this ticket is too long, and we can start fresh

comment:55 Changed 2 years ago by thivnor

OK I will try to find some time this work to create updated Java code based on the existing PHP code. When I get it working I will create a new ticket.

FYI- I get the core problem now ... the Java code just hasn't been kept up to date.

comment:56 Changed 2 years ago by klaus

I commited the patch mentioned above to the trunk. I am also interested in creating a better Java implementation of the PIWIK-Tracker to fix those issues you mentioned before. (tracking behind a proxy, ..)

Please share the link to the new ticket here when you create one.

comment:57 Changed 23 months ago by matt (mattab)

The new ticket is at: #3151

When you commit changes in the javaq code, please add in the commit message "Refs #3151" so that it creates a comment in the ticket automatically

comment:58 Changed 23 months ago by matt (mattab)

  • Priority changed from normal to major
  • Summary changed from PiwikTracker.java to Java Tracking Client V1 - PiwikTracker.java
Note: See TracTickets for help on using tickets.