IPv6 support

[vipsoft]

This is a follow-up ticket to #1110.

To support IPv6 addresses will require data conversion in the database:

• IPv6 address are 128-bit. IP addresses are stored in the MySQL tables as 8-byte (64-bit) BIGINT UNSIGNED, or 4-byte (32-bit) INT UNSIGNED (as of r2237).

In php.net/ip2long, there's an implementation of ip2long6 and long2ip6 that relies on the GNU MP (Multi-Precision) extension ('gmp'). But for greater cross-platform support, we might also consider comparable implementations using the BC Math ('bcmath') extension.

Above uses inet_ntop and inet_pton to aid in conversion to/from user readable and internal representation (in_addr). (Note: Windows support added in php 5.3.)

Issues/Considerations:

• Exclude visits from IP (added in Piwik 0.6)
• on Windows XP + IIS7, $_SERVER['REMOTE_ADDR'] for "localhost" uses the deprecated form, "::1"
• IP-to-country database/API support for ipv6

[dm8tbr]

Please add me to cc on this issue. Regular users don't seem to be able to do it themselves.

It's important for me to stay up to date on the ipv6 status of Piwik as my webservers all run dual-stack and disabling IPv6 is not an option.

I'd rather disable IPv4 but that's not yet an option. ;)

[vipsoft]

Other issues:

• geoip support for ipv6
• exclude visits from ip

Since this adds more runtime dependencies, and incurs performance and storage costs, I'd ideally like to implement this as a plugin.

[ts77]

The future will be IPv6-only and IPv4 addresses are running out in the next couple of years (some even tell in the next 2 years) so I would think that IPv6 support would/should be a core feature.

[vipsoft]

Drupal's reverse proxy configuration has a setting for the IPs of trusted proxies.

[veretcle]

In order to store IPv6 adresses, converting them to long long int (128 bits integer) is perhaps not the best idea.

For IPv4 adresses, the difference between decimal and dotted-decimal format doesn't change much, for IPv6, storing a 40 characters string and a 128 bits decimal integer is a HUGE difference. I'll suggest storing IPv6 adresses as string. Coding a ip2long6 function for PHP is no big deal, but the returned result will be a pain in the ass to treat/calculate/print/store.

Also, as ts77 noticed, I really think this should be a core feature : less than 300 days before exhaustion !

[veretcle]

Basic IPv6 integration patch with Live Plugin

[veretcle]

My little contribution ;). Just a little patch in order to add IPv6 support to Live Plugin. This adds a new column in {prefix}_log_visit named location_ipv6 and a few line of code to fill it in the database and read it on the interface.

I've just done some very basic testing on it, so use with care.

Enjoy !

[halfdan]

I suggest to raise the importance of that ticket. The german "Deutsche Telekom" has just announced ( http://www.heise.de/newsticker/meldung/Deutsche-Telekom-konkretisiert-IPv6-Plaene-1102458.html) that they will migrate all users until the end of 2011 to a dual-stacked IPv4 + IPv6. This also affects all other services like DNS, E-Mail and Hosting. Piwik has gained a very high popularity in Germany and not supporting IPv6 in the near future could very well result in a huge drop of interest.

[vipsoft]

*nod*

[mgc8]

Replying to veretcle:

In order to store IPv6 adresses, converting them to long long int (128 bits integer) is perhaps not the best idea. For IPv4 adresses, the difference between decimal and dotted-decimal format doesn't change much, for IPv6, storing a 40 characters string and a 128 bits decimal integer is a HUGE difference. I'll suggest storing IPv6 adresses as string. Coding a ip2long6 function for PHP is no big deal, but the returned result will be a pain in the ass to treat/calculate/print/store.

Sorry, but this is wrong. Please do not store IPv6 addresses as strings! Reasons: 1. A 40-char string actuall takes 320bits. An 128bit integer comparatively occupies 16 characters. Remember that a "char" is 8bit. 2. An IPv6 address can have different representations as string -- for example "2001::1" or "2001:0000:0000:(etc):1", how are you going to handle that? The only sane thing to do is store them as what they are, and that is 128bit integers. 3. Conversion functions form integer to string are trivial to implement.

On another note, IPv6 support is critically needed at this point. Please make this bug a priority!

[vipsoft]

Everyone: yes, the sky is falling. We understand the need for this feature, and if you're not contributing a patch, we would appreciate your patience in the meantime.

The proposal is for:

• IP addresses to be stored in a VARBINARY(16) field
• an pretty printing function should take care of printing IPv4 vs IPv6 addresses in a pleasing human read-able form
• data conversion of existing caller_ip and location_ip columns, from INT to VARBINARY; we may need some magic here to convert from machine-dependent to network byte order.
• impacted: Live, SitesManager (plus #1755), Tracker (plus #1553), Geolocation plugins (e.g., GeoIP), and changes introduced to getIp() for Piwik 1.1

[matt]

This sounds important, yet not clear how to deal with the new ipv6 notation... Some random notes

• What does running as a dual stack exactly mean? Looking at  wikipedia, it seems there are many proposed implementations for the transition period. Will all these proposal expose the same 'REQUEST_ADDR' value?
• What will 'REQUEST_ADDR' look like for a IPv6 client? I think it can have several notations like ::ffff:24.65.128.105 (ipv4 mapped addresses), ::1 seems to be the same as 127.0.01 ( source), etc..
• Should we use  inet_pton for converting IPs?
• Could  inet_ntop be used for pretty printing?
• We will need to check in requirements that ipv6 support is enabled in PHP (by default it is)

[matt]

I was debugging #1939 and saw an ipv6 display bug in Live! (as expected since we dont support ipv6 yet)

we have $_SERVER
  ["HTTP_X_FORWARDED_FOR"]=>
  string(19) "1.b.c.d:28941"

config file has

[General]
proxy_client_headers[] = HTTP_X_FORWARDED_FOR

IP is saved as (int)0

to replicate, in piwik.php add at the top

$_SERVER['HTTP_X_FORWARDED_FOR'] = 'a3.b.c.d:28941';

In this case my IP was indeed a.b.c.d

[matt]

(In [3622]) Refs #1111 Dealing with a.b.c.d:12345 IPs

[matt]

Can we be compatible with GeoIP lookups on ipv6 addresses? A very quick search brought this thread:  http://forum.maxmind.com/viewtopic.php?t=915 which sounds like geoip might be compatible with ipv6 lookups.

[vipsoft]

Should be possible.

[veretcle]

Replying to mgc8:

Sorry, but this is wrong. Please do not store IPv6 addresses as strings! Reasons: 1. A 40-char string actuall takes 320bits. An 128bit integer comparatively occupies 16 characters. Remember that a "char" is 8bit. 2. An IPv6 address can have different representations as string -- for example "2001::1" or "2001:0000:0000:(etc):1", how are you going to handle that? The only sane thing to do is store them as what they are, and that is 128bit integers.

This is effectively an issue : using VARBINARY(16) is much more interesting.

So I have done another (simplier) patch that allow IPv6 in visit log. The only issue is that Piwik_Common::getIp() returns an in_addr structure and not a decimal IPv4 address (this will probably lead to an issue if another module uses this function and expect an integer…).

[vipsoft]

Note to self:

•  https://wiki.ubuntu.com/IPv6
•  http://bgp.he.net/ipv6-progress-report.cgi

[vipsoft]

(In [4533]) fixes #1111 - add support for IPv6 addresses (tracking, anonymization, and exclusion) fixes #2095 - add new anonymization hook (pre-heuristics) fixes #2055 - optional IP filter when multiple proxies present fixes #1775 - SitesManager: supports CIDR notation for IP exclusion

Notes:

• Installer no longer checks for IPv6, so the related messages should be deleted from translations
• IPv4 mapped addresses (e.g., ::ffff:127.0.0.1) are no longer re-mapped into IPv4 space
• users who to query IP addresses from MySQL directly, can use the following SQL, but inet_ntoa() is limited to IPv4 addresses:

  select inet_ntoa(conv(hex(location_ip), 16, 10)) from piwik_log_visit;
  

• Windows: IPv6 inet_pton()/inet_ntop() not supported until php 5.3; see #2351

[vipsoft]

(In [4534]) refs #1111 - debug off

[vipsoft]

(In [4535]) refs #1111 - fix some of the unit tests; will look at the failing integration tests later

[absynth]

IPv6 support for the detailed visitor log does not seem to work. Visitors via IPv6 do not show up in the detailed log. Realtime visitor log works, though.

[vipsoft]

worksforme. please make sure you're viewing the correct date/period.

[absynth]

ACK, WFM too, the date was wrongly chosen.