Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPv6 support #1111

Closed
robocoder opened this issue Jan 16, 2010 · 25 comments
Closed

IPv6 support #1111

robocoder opened this issue Jan 16, 2010 · 25 comments
Assignees
Labels
Critical Indicates the severity of an issue is very critical and the issue has a very high priority. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone

Comments

@robocoder
Copy link
Contributor

This is a follow-up ticket to #1110.

To support IPv6 addresses will require data conversion in the database:

  • IPv6 address are 128-bit. IP addresses are stored in the MySQL tables as 8-byte (64-bit) BIGINT UNSIGNED, or 4-byte (32-bit) INT UNSIGNED (as of r2237).

In php.net/ip2long, there's an implementation of ip2long6 and long2ip6 that relies on the GNU MP (Multi-Precision) extension ('gmp'). But for greater cross-platform support, we might also consider comparable implementations using the BC Math ('bcmath') extension.

Above uses inet_ntop and inet_pton to aid in conversion to/from user readable and internal representation (in_addr). (Note: Windows support added in php 5.3.)

Issues/Considerations:

  • Exclude visits from IP (added in Piwik 0.6)
  • on Windows XP + IIS7, $_SERVER!['REMOTE_ADDR'] for "localhost" uses the deprecated form, "::1"
  • IP-to-country database/API support for ipv6
@dm8tbr
Copy link

dm8tbr commented May 6, 2010

Please add me to cc on this issue. Regular users don't seem to be able to do it themselves.

It's important for me to stay up to date on the ipv6 status of Piwik as my webservers all run dual-stack and disabling IPv6 is not an option.

I'd rather disable IPv4 but that's not yet an option. ;)

@robocoder
Copy link
Contributor Author

Other issues:

  • geoip support for ipv6
  • exclude visits from ip

Since this adds more runtime dependencies, and incurs performance and storage costs, I'd ideally like to implement this as a plugin.

@anonymous-matomo-user
Copy link

The future will be IPv6-only and IPv4 addresses are running out in the next couple of years (some even tell in the next 2 years) so I would think that IPv6 support would/should be a core feature.

@robocoder
Copy link
Contributor Author

Drupal's reverse proxy configuration has a setting for the IPs of trusted proxies.

@anonymous-matomo-user
Copy link

In order to store IPv6 adresses, converting them to long long int (128 bits integer) is perhaps not the best idea.

For IPv4 adresses, the difference between decimal and dotted-decimal format doesn't change much, for IPv6, storing a 40 characters string and a 128 bits decimal integer is a HUGEdifference. I'll suggest storing IPv6 adresses as string. Coding a ip2long6 function for PHP is no big deal, but the returned result will be a pain in the ass to treat/calculate/print/store.

Also, as ts77 noticed, I really think this should be a core feature : less than 300 days before exhaustion !

@anonymous-matomo-user
Copy link

Attachment: Basic IPv6 integration patch with Live Plugin
ipv6.patch

@anonymous-matomo-user
Copy link

My little contribution ;). Just a little patch in order to add IPv6 support to Live Plugin. This adds a new column in {prefix}_log_visit named location_ipv6 and a few line of code to fill it in the database and read it on the interface.

I've just done some very basic testing on it, so use with care.

Enjoy !

@halfdan
Copy link
Member

halfdan commented Oct 7, 2010

I suggest to raise the importance of that ticket. The german "Deutsche Telekom" has just announced (http://www.heise.de/newsticker/meldung/Deutsche-Telekom-konkretisiert-IPv6-Plaene-1102458.html) that they will migrate all users until the end of 2011 to a dual-stacked IPv4 + IPv6. This also affects all other services like DNS, E-Mail and Hosting.
Piwik has gained a very high popularity in Germany and not supporting IPv6 in the near future could very well result in a huge drop of interest.

@robocoder
Copy link
Contributor Author

nod

@mgc8
Copy link

mgc8 commented Oct 22, 2010

Replying to veretcle:

In order to store IPv6 adresses, converting them to long long int (128 bits integer) is perhaps not the best idea.
For IPv4 adresses, the difference between decimal and dotted-decimal format doesn't change much, for IPv6, storing a 40 characters string and a 128 bits decimal integer is a HUGEdifference. I'll suggest storing IPv6 adresses as string. Coding a ip2long6 function for PHP is no big deal, but the returned result will be a pain in the ass to treat/calculate/print/store.

Sorry, but this is wrong. Please do not store IPv6 addresses as strings! Reasons:

  1. A 40-char string actuall takes 320bits. An 128bit integer comparatively occupies 16 characters. Remember that a "char" is 8bit.
  2. An IPv6 address can have different representations as string -- for example "2001::1" or "2001:0000:0000:(etc):1", how are you going to handle that? The only sane thing to do is store them as what they are, and that is 128bit integers.
  3. Conversion functions form integer to string are trivial to implement.

On another note, IPv6 support is critically needed at this point. Please make this bug a priority!

@robocoder
Copy link
Contributor Author

Everyone: yes, the sky is falling. We understand the need for this feature, and if you're not contributing a patch, we would appreciate your patience in the meantime.

The proposal is for:

@mattab
Copy link
Member

mattab commented Nov 25, 2010

This sounds important, yet not clear how to deal with the new ipv6 notation...
Some random notes

  • What does running as a dual stack exactly mean? Looking at wikipedia, it seems there are many proposed implementations for the transition period. Will all these proposal expose the same 'REQUEST_ADDR' value?
  • What will 'REQUEST_ADDR' look like for a IPv6 client? I think it can have several notations like ::ffff:24.65.128.105 (ipv4 mapped addresses), ::1 seems to be the same as 127.0.01 (source), etc..
  • Should we use inet_pton for converting IPs?
  • Could inet_ntop be used for pretty printing?
  • We will need to check in requirements that ipv6 support is enabled in PHP (by default it is)

@mattab
Copy link
Member

mattab commented Jan 5, 2011

I was debugging #1939 and saw an ipv6 display bug in Live! (as expected since we dont support ipv6 yet)

we have $_SERVER
  ["HTTP_X_FORWARDED_FOR"]=>
  string(19) "1.b.c.d:28941"

config file has


[General]
proxy_client_headers[] = HTTP_X_FORWARDED_FOR

IP is saved as (int)0

to replicate, in piwik.php add at the top

$_SERVER['HTTP_X_FORWARDED_FOR'] = 'a3.b.c.d:28941';

In this case my IP was indeed a.b.c.d

@mattab
Copy link
Member

mattab commented Jan 5, 2011

(In [3622]) Refs #1111 Dealing with a.b.c.d:12345 IPs

@mattab
Copy link
Member

mattab commented Feb 6, 2011

Can we be compatible with GeoIP lookups on ipv6 addresses? A very quick search brought this thread: http://forum.maxmind.com/viewtopic.php?t=915 which sounds like geoip might be compatible with ipv6 lookups.

@robocoder
Copy link
Contributor Author

Should be possible.

@anonymous-matomo-user
Copy link

Replying to mgc8:

Sorry, but this is wrong. Please do not store IPv6 addresses as strings! Reasons:

  1. A 40-char string actuall takes 320bits. An 128bit integer comparatively occupies 16 characters. Remember that a "char" is 8bit.
  2. An IPv6 address can have different representations as string -- for example "2001::1" or "2001:0000:0000:(etc):1", how are you going to handle that? The only sane thing to do is store them as what they are, and that is 128bit integers.

This is effectively an issue : using VARBINARY(16) is much more interesting.

So I have done another (simplier) patch that allow IPv6 in visit log. The only issue is that Piwik_Common::getIp() returns an in_addr structure and not a decimal IPv4 address (this will probably lead to an issue if another module uses this function and expect an integer).

@anonymous-matomo-user
Copy link

Attachment:
ipv6_2.patch

@robocoder
Copy link
Contributor Author

@robocoder
Copy link
Contributor Author

(In [4533]) fixes #1111 - add support for IPv6 addresses (tracking, anonymization, and exclusion)
fixes #2095 - add new anonymization hook (pre-heuristics)
fixes #2055 - optional IP filter when multiple proxies present
fixes #1775 - SitesManager: supports CIDR notation for IP exclusion

Notes:

  • Installer no longer checks for IPv6, so the related messages should be deleted from translations
  • IPv4 mapped addresses (e.g., ::ffff:127.0.0.1) are no longer re-mapped into IPv4 space
  • users who to query IP addresses from MySQL directly, can use the following SQL, but inet_ntoa() is limited to IPv4 addresses:
select inet_ntoa(conv(hex(location_ip), 16, 10)) from piwik_log_visit;

@robocoder
Copy link
Contributor Author

(In [4534]) refs #1111 - debug off

@robocoder
Copy link
Contributor Author

(In [4535]) refs #1111 - fix some of the unit tests; will look at the failing integration tests later

@anonymous-matomo-user
Copy link

IPv6 support for the detailed visitor log does not seem to work. Visitors via IPv6 do not show up in the detailed log. Realtime visitor log works, though.

@robocoder
Copy link
Contributor Author

worksforme. please make sure you're viewing the correct date/period.

@anonymous-matomo-user
Copy link

ACK, WFM too, the date was wrongly chosen.

@robocoder robocoder added this to the 1.4 - Piwik 1.4 milestone Jul 8, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Critical Indicates the severity of an issue is very critical and the issue has a very high priority. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Development

No branches or pull requests

6 participants