Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include GeoIP in core after improvements #1823

Closed
mattab opened this issue Nov 16, 2010 · 90 comments
Closed

Include GeoIP in core after improvements #1823

mattab opened this issue Nov 16, 2010 · 90 comments
Assignees
Labels
Critical Indicates the severity of an issue is very critical and the issue has a very high priority. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone

Comments

@mattab
Copy link
Member

mattab commented Nov 16, 2010

See doc: Geo Locate visitors countries cities and regions.

GeoIP plugin #45 is one of the most popular plugins. For a web analytics tool, getting user countries as accurately as possible is critical, and Piwik should help users in this direction.

When the plugin is released in trunk, we should update the FAQ, website pages and wiki pages mentionning GeoIP, and mark as closed the GeoIP ticket #45. For Goals compatibility of GeoIP plugin, see #1434.

Please let us know in the comments your feedback. If you would like to participate... well you know what to do!

@gka
Copy link
Contributor

gka commented Dec 4, 2010

Just some input to clarify the terms "country" and "region". Refering to the list of administrative levels used in OpenStreetMaps a country would correspond to admin level 2 while regions would correspond to admin level 4.

@gka
Copy link
Contributor

gka commented Dec 4, 2010

Do we record lat/long for each visitor, or do we assume that other systems will know where to plot a given City

I think it is NOT necessary to record lat/long for each visitor. It is sufficient to record the city id. The GeoIP db would resolve each visitors location within the same city to the same lat/long anyway. In fact for each city there is only one lat/long stored in the GeoLite City DB (more precisely in the cityByCountry table).

As the number of available cities (= pairs of lat/long) differs between the different GeoIP databases, it makes no sense to put these information into other systems like the world map.

@anonymous-matomo-user
Copy link

if I might add to this. The maxmind db gives a city lookup. This does not work how people think it will.
Blocks of IP numbers are sold to to service providers who resell to end users. However, the IP issuing authority assign the city of the ISP address to all the IP numbers. At least that is how it works in the UK. Things may vary in different countries and ISPs don't reallocate city when they sell dedicated IP numbers to end users.
The result is that city lookup generally only gives the city of the ISP and not where the visitor is visting from. The ISP can be anywhere in the country and hundreds of miles from where the visitor is based. In other words, city lookup is useless except for giving the location of ISPs. This also means that lat long is useless too since it seems to be based on city lookup.
When IPV6 is rolled out and if, and only if, ISPs allocate city to users when they purchase a fixed IP then city lookup may become useful. But many ISPs still use dynamically allocated IPs so it wouldn't work in that case either.
In short the concept of providing city and/or lat/long of vistors is fundamentlly flawed.

@oparoz
Copy link

oparoz commented Dec 17, 2010

+1 for this, especially the Apache module detection routine. I get a few fatal errors in my logs because the plugin insists on loading the local files instead of getting the data from Apache.

@anonymous-matomo-user
Copy link

I have the commercial db of Maxmind. You can use it if you want for developing the new plugin. Let me know how I can contact you.

@mattab
Copy link
Member Author

mattab commented Feb 5, 2011

See a bug report in the forum in the php script to update past visits: http://forum.piwik.org/read.php?2,71587,page=2#msg-71784

and fix: http://forum.piwik.org/read.php?2,70989,page=1#msg-92434

@robocoder
Copy link
Contributor

I'll take this on, in conjunction with the ipv6 ticket.

@mattab
Copy link
Member Author

mattab commented Feb 9, 2011

  • New report: Users companies? I believe that we can get the 'Enterprise/Company' that Visitors connect from, with the GeoIP data. It would be interesting to propose this new report in Users > Countries page.

@anonymous-matomo-user
Copy link

Great idea.

@robocoder
Copy link
Contributor

In the existing GeoIP plugin, there's a misc/.htaccess file. We don't want this in the new plugin. Access to geoipUpdateRows.php (or equivalent) should be guarded via token_auth.

@anonymous-matomo-user
Copy link

Should I delete the .htaccess file in there?

@robocoder
Copy link
Contributor

Yes, you can remove the .htaccess file. After you've run it once, you shouldn't have to run it again.

@anonymous-matomo-user
Copy link

$ php geoipUpdateRows.php

Fatal error: Call to undefined function _parse_ini_file() in /home/kiplingw/avaneya.com/piwik/core/Config.php on line 373

???

I also removed the .htaccess file.

@robocoder
Copy link
Contributor

This should be fixed in the updated .zip that I attached to #45. Heres the patch so you don't have to redownload the .zip:

Index: geoipUpdateRows.php
===================================================================
--- geoipUpdateRows.php (revision 51)
+++ geoipUpdateRows.php (working copy)
@@ -20,8 +20,8 @@
        . PATH_SEPARATOR . PIWIK_INCLUDE_PATH . '/libs'
        . PATH_SEPARATOR . PIWIK_INCLUDE_PATH . '/plugins');

+require_once PIWIK_INCLUDE_PATH . '/libs/upgradephp/upgrade.php';
 require_once PIWIK_INCLUDE_PATH . '/core/testMinimumPhpVersion.php';
-
 require_once PIWIK_INCLUDE_PATH . '/core/Loader.php';

 $GLOBALS['PIWIK_TRACKER_DEBUG'] = false;

@anonymous-matomo-user
Copy link

Thanks. Applied. How can I test it?

@anonymous-matomo-user
Copy link

I ran

$ php geoipUpdateRows.php

It finished execution (no output), and I noticed the UserCountry_ thing is still there in the stats. Should I just ignore that for now and assume new stats will not have that?

@robocoder
Copy link
Contributor

yes

@anonymous-matomo-user
Copy link

Thank you =)

@mattab
Copy link
Member Author

mattab commented Apr 23, 2011

To answer questions in the ticket:

  • we should track regions in the log_visit table (new field, char(2) ?) as per specification in: http://www.maxmind.com/app/fips10_4
  • we should also track cities, and add a new field for cities.
    • the API for cities, would return the 'label' being the city name, and also one column for latitude, one column for longitude, since this is required to draw the Cities Maps Open Source SVG Map to show cities and regions #1652
    • this means, that we don't store lat/long in log_visit

@robocoder
Copy link
Contributor

Replying to matt:

  • this means, that we don't store lat/long in log_visit

I'm thinking of keeping lat/long because:

@robocoder
Copy link
Contributor

re: comment:13

I would like to propose:

  • rolling the provider plugin into the geolocation plugin
  • if the geolocation plugin can get the organization field, it populate location_provider
  • otherwise, fallback to the gethostbyaddr() method

@mattab
Copy link
Member Author

mattab commented May 4, 2011

I would like to propose:

  • rolling the provider plugin into the geolocation plugin
  • if the geolocation plugin can get the organization field, it populate location_provider
  • otherwise, fallback to the gethostbyaddr() method

great idea!

The only thing, is please make sure the few "Provider" special cases are still working. In particular, VisitorGenerator & proxy-piwik.php disable the Provider lookup because it is too slow

@robocoder
Copy link
Contributor

Ok. There are a couple of third party plugins (e.g., KSVisitorImport and TrackerSecondaryDb) that also disable the Provider plugin.

@mattab
Copy link
Member Author

mattab commented May 4, 2011

as a note, these plugins will be obsolete once we implement #134

@anonymous-matomo-user
Copy link

+1 vote for adding regions onto this as well. They are available in GeoLiteCity, so might as well use them. It would be great to include this into a regional map as well that the country map can drill down into.

@anonymous-matomo-user
Copy link

Copy of my coment to #5465 (sorry, I used wrong ticket, apparantly, I knew there was one specifically for integration of GeoIP into core):

This new plugin sounds promising. But I hope you are going to also keep the old browser language/country detection, maybe named as such. I personally consider that language display equally important as the IP location display.

Following scenario: I'm on a travel around the world, and have a travel blog. People accessing that blog are often people I have met on the trip, often still traveling. Now, when I see my Piwik logs, the IP location (which I currently check manually) is surely interesting, but what tells me more about a visitor is actually his browser language. If you check the IP address I am writing this from, you will see that it is Malaysian. How much do I have to do with Malaysia? Nothing. My browser language is German of Germany, which tells more. And the combination of the two IP location and browser country (i.e. the current detection) actually provides one more detail: the visitor is most likely a traveler or an expat. I can imagine website who interested in that marketing information.

You would not believe how many travelers roam the world this days. And I would say most of them use the often free WiFi (at their place of stay, bars and restaurants all over Southeast Asia) with their own devices: Laptops, Phones, Tablets, etc. It seems to be the new way of travel, with people sticking their noses into displays half of their time, with most of that time on Facebook.

P.S.: Since there are countries with several languages (Belguim, etc.), but also countries with common language (UK, US, etc.), maybe both, the browser country and its language could be shown (if provided by browser). Additionally to the IP location provided by this plugin.

@robocoder
Copy link
Contributor

jawsmith: #638

@anonymous-matomo-user
Copy link

+1 on jawsmith proposal on having a combined vision on location against visitor's preferred language.
As a belgian developer I can tell you that this kind of information can be of crucial interest in a country like Belgium, but in many others too. For example, usage of the spanish language in some regions of the US can be an important factor I think...

I imagine an ideal "Vistor countries" GeoIP plugin offering the current "Countries" split, clicking a country name would open a "Regions" list, clicking a region name would open (the currently available) "Cities" list. Then an additional button could be fit at the bottom, between the "Display simple table" and "Display a table with more metrics" that would "Display a table with languages". That table could have one additional column for each language that was detected...

@anonymous-matomo-user
Copy link

Replying to vipsoft:

jawsmith: #638

Thank you very much for the info on the browser language detection plugin! That just leaves the browser country detection, in case the GeoIP plugin replaces it in core. (E.g.: Is it a British or an American accessing my website from the Philippines?)

@gka
Copy link
Contributor

gka commented Oct 22, 2011

Do we recording regions as well as Countries?
Do we record Cities?

As the new world map widget will be able to display data for regions and cities, it would be amazing if Piwik would be able to record the data for regions and cities :)

Do we record lat/long for each visitor, or do we assume that other systems (eg. the world map) will know where to plot a given City (and maintain their own database)?

Nope, the world map doesn't store locations for every city. Instead, it will be able to plot any given lat/long onto the map.

@diosmosis
Copy link
Member

(In [7181]) Refs #1823, fade 'Done' in & out after successfully switched location providers.

@diosmosis
Copy link
Member

(In [7186]) Refs #1823, many changes including:

  • Add warning if old GeoIP plugin is used.
  • Display links to installation instructions for different providers if they are not installed.
  • Add report documentation for country, region, continent and city reports, round latitude/longitude.
  • Increase 'Done' timeout when switching providers
  • Display quick start instructions for GeoIP if no GeoIP provider is currently working.
  • Add script to geolocate old data.

@diosmosis
Copy link
Member

(In [7187]) Refs #1823, display informative note when Region + City reports have no location data, merge unknown rows in region & city reports and make sure latitude/longitude is rounded in API output.

@mattab
Copy link
Member Author

mattab commented Oct 15, 2012

(In [7203]) Refs #1823 Adding important-to-have target=_blank

@mattab
Copy link
Member Author

mattab commented Oct 15, 2012

(In [7204]) Minor text change & provider ordering Refs #1823

@mattab
Copy link
Member Author

mattab commented Oct 15, 2012

(In [7205]) remove test Refs #1823

@diosmosis
Copy link
Member

This bug is fixed. :) I created a ticket for improvements here: #3442

@mattab
Copy link
Member Author

mattab commented Oct 16, 2012

See doc: Geo Locate visitors countries cities and regions.

@diosmosis
Copy link
Member

(In [7234]) Refs #1823, add note to geoipUpdateRows.php that tells user to re-process their reports.

@diosmosis
Copy link
Member

(In [7236]) Refs #1823, add note to IP anonymization about geolocation accuracy.

@diosmosis
Copy link
Member

(In [7260]) Refs #1823, add alternative check for GEOIP_COUNTRY_CODE $_SERVER var to ServerBased GeoIP implementation.

@diosmosis
Copy link
Member

(In [7281]) Refs #1823, do broken check w/ both GEOIP_ADDR & GEOIP_COUNTRY_CODE.

@diosmosis
Copy link
Member

(In [7283]) Refs #1823, added redundant trusted hosts warning to general settings page & display help icon that links to faq in warning.

@diosmosis
Copy link
Member

[7628] refs this ticket, not #1253. (typo in commit msg)

@mattab
Copy link
Member Author

mattab commented Dec 27, 2012

We need YOUR help! We are running a crowd funding campaign to raise funds to implement the detailed Visitors Maps of Countries, Regions and Cities (for all countries)!

These maps will be beautiful, usable, and built using open standards SVG+JS. They will show detailed visitor count, conversion rates, by Country but also (New!) by city and region.

Pledge now at: http://crowdfunding.piwik.org/analytics-maps-world-country-city-region/

Piwik needs you!

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Critical Indicates the severity of an issue is very critical and the issue has a very high priority. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Development

No branches or pull requests

9 participants