Opened 4 years ago

Closed 3 years ago

Last modified 3 years ago

#1111 closed New feature (fixed)

IPv6 support

Reported by: vipsoft Owned by: vipsoft
Priority: critical Milestone: 1.4 - Piwik 1.4
Component: Core Keywords:
Cc: Sensitive: no

Description (last modified by vipsoft)

This is a follow-up ticket to #1110.

To support IPv6 addresses will require data conversion in the database:

  • IPv6 address are 128-bit. IP addresses are stored in the MySQL tables as 8-byte (64-bit) BIGINT UNSIGNED, or 4-byte (32-bit) INT UNSIGNED (as of r2237).

In php.net/ip2long, there's an implementation of ip2long6 and long2ip6 that relies on the GNU MP (Multi-Precision) extension ('gmp'). But for greater cross-platform support, we might also consider comparable implementations using the BC Math ('bcmath') extension.

Above uses inet_ntop and inet_pton to aid in conversion to/from user readable and internal representation (in_addr). (Note: Windows support added in php 5.3.)

Issues/Considerations:

  • Exclude visits from IP (added in Piwik 0.6)
  • on Windows XP + IIS7, $_SERVER['REMOTE_ADDR'] for "localhost" uses the deprecated form, "::1"
  • IP-to-country database/API support for ipv6

Attachments (2)

ipv6.patch (4.2 KB) - added by veretcle 4 years ago.
Basic IPv6 integration patch with Live Plugin
ipv6_2.patch (1.0 KB) - added by veretcle 3 years ago.

Download all attachments as: .zip

Change History (34)

comment:1 Changed 4 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:2 Changed 4 years ago by dm8tbr

Please add me to cc on this issue. Regular users don't seem to be able to do it themselves.

It's important for me to stay up to date on the ipv6 status of Piwik as my webservers all run dual-stack and disabling IPv6 is not an option.

I'd rather disable IPv4 but that's not yet an option. ;)

comment:3 Changed 4 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:4 Changed 4 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:5 Changed 4 years ago by vipsoft (robocoder)

Other issues:

  • geoip support for ipv6
  • exclude visits from ip

Since this adds more runtime dependencies, and incurs performance and storage costs, I'd ideally like to implement this as a plugin.

comment:6 Changed 4 years ago by ts77

The future will be IPv6-only and IPv4 addresses are running out in the next couple of years (some even tell in the next 2 years) so I would think that IPv6 support would/should be a core feature.

comment:7 Changed 4 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:8 Changed 4 years ago by vipsoft (robocoder)

  • Description modified (diff)

comment:9 Changed 4 years ago by vipsoft (robocoder)

Drupal's reverse proxy configuration has a setting for the IPs of trusted proxies.

comment:10 follow-up: Changed 4 years ago by veretcle

In order to store IPv6 adresses, converting them to long long int (128 bits integer) is perhaps not the best idea.

For IPv4 adresses, the difference between decimal and dotted-decimal format doesn't change much, for IPv6, storing a 40 characters string and a 128 bits decimal integer is a HUGE difference. I'll suggest storing IPv6 adresses as string. Coding a ip2long6 function for PHP is no big deal, but the returned result will be a pain in the ass to treat/calculate/print/store.

Also, as ts77 noticed, I really think this should be a core feature : less than 300 days before exhaustion !

Changed 4 years ago by veretcle

Basic IPv6 integration patch with Live Plugin

comment:11 Changed 4 years ago by veretcle

My little contribution ;). Just a little patch in order to add IPv6 support to Live Plugin. This adds a new column in {prefix}_log_visit named location_ipv6 and a few line of code to fill it in the database and read it on the interface.

I've just done some very basic testing on it, so use with care.

Enjoy !

comment:12 Changed 4 years ago by halfdan

I suggest to raise the importance of that ticket. The german "Deutsche Telekom" has just announced (http://www.heise.de/newsticker/meldung/Deutsche-Telekom-konkretisiert-IPv6-Plaene-1102458.html) that they will migrate all users until the end of 2011 to a dual-stacked IPv4 + IPv6. This also affects all other services like DNS, E-Mail and Hosting.
Piwik has gained a very high popularity in Germany and not supporting IPv6 in the near future could very well result in a huge drop of interest.

comment:13 Changed 4 years ago by vipsoft (robocoder)

  • Priority changed from normal to major

*nod*

comment:14 in reply to: ↑ 10 ; follow-up: Changed 3 years ago by mgc8

Replying to veretcle:

In order to store IPv6 adresses, converting them to long long int (128 bits integer) is perhaps not the best idea.
For IPv4 adresses, the difference between decimal and dotted-decimal format doesn't change much, for IPv6, storing a 40 characters string and a 128 bits decimal integer is a HUGE difference. I'll suggest storing IPv6 adresses as string. Coding a ip2long6 function for PHP is no big deal, but the returned result will be a pain in the ass to treat/calculate/print/store.

Sorry, but this is wrong. Please do not store IPv6 addresses as strings! Reasons:

  1. A 40-char string actuall takes 320bits. An 128bit integer comparatively occupies 16 characters. Remember that a "char" is 8bit.
  2. An IPv6 address can have different representations as string -- for example "2001::1" or "2001:0000:0000:(etc):1", how are you going to handle that? The only sane thing to do is store them as what they are, and that is 128bit integers.
  3. Conversion functions form integer to string are trivial to implement.

On another note, IPv6 support is critically needed at this point. Please make this bug a priority!

comment:15 Changed 3 years ago by vipsoft (robocoder)

  • Milestone changed from Features requests 1.x or 2.x to 1.2 - Piwik 1.2

Everyone: yes, the sky is falling. We understand the need for this feature, and if you're not contributing a patch, we would appreciate your patience in the meantime.

The proposal is for:

  • IP addresses to be stored in a VARBINARY(16) field
  • an pretty printing function should take care of printing IPv4 vs IPv6 addresses in a pleasing human read-able form
  • data conversion of existing caller_ip and location_ip columns, from INT to VARBINARY; we may need some magic here to convert from machine-dependent to network byte order.
  • impacted: Live, SitesManager (plus #1755), Tracker (plus #1553), Geolocation plugins (e.g., GeoIP), and changes introduced to getIp() for Piwik 1.1

comment:16 Changed 3 years ago by matt (mattab)

  • Priority changed from major to critical

comment:17 Changed 3 years ago by matt (mattab)

This sounds important, yet not clear how to deal with the new ipv6 notation...
Some random notes

  • What does running as a dual stack exactly mean? Looking at wikipedia, it seems there are many proposed implementations for the transition period. Will all these proposal expose the same 'REQUEST_ADDR' value?
  • What will 'REQUEST_ADDR' look like for a IPv6 client? I think it can have several notations like ::ffff:24.65.128.105 (ipv4 mapped addresses), ::1 seems to be the same as 127.0.01 (source), etc..
  • Should we use inet_pton for converting IPs?
  • Could inet_ntop be used for pretty printing?
  • We will need to check in requirements that ipv6 support is enabled in PHP (by default it is)

comment:18 Changed 3 years ago by vipsoft (robocoder)

  • Owner set to vipsoft

comment:19 Changed 3 years ago by matt (mattab)

I was debugging #1939 and saw an ipv6 display bug in Live! (as expected since we dont support ipv6 yet)

we have $_SERVER
  ["HTTP_X_FORWARDED_FOR"]=>
  string(19) "1.b.c.d:28941"

config file has

[General]
proxy_client_headers[] = HTTP_X_FORWARDED_FOR

IP is saved as (int)0

to replicate, in piwik.php add at the top

$_SERVER['HTTP_X_FORWARDED_FOR'] = 'a3.b.c.d:28941';

In this case my IP was indeed a.b.c.d

comment:20 Changed 3 years ago by matt (mattab)

(In [3622]) Refs #1111 Dealing with a.b.c.d:12345 IPs

comment:21 Changed 3 years ago by matt (mattab)

Can we be compatible with GeoIP lookups on ipv6 addresses? A very quick search brought this thread: http://forum.maxmind.com/viewtopic.php?t=915 which sounds like geoip might be compatible with ipv6 lookups.

comment:22 Changed 3 years ago by vipsoft (robocoder)

Should be possible.

comment:23 Changed 3 years ago by vipsoft (robocoder)

  • Milestone changed from 1.2 Piwik 1.2 to 1.3 - Piwik 1.3

comment:24 in reply to: ↑ 14 ; follow-up: Changed 3 years ago by veretcle

Replying to mgc8:

Sorry, but this is wrong. Please do not store IPv6 addresses as strings! Reasons:

  1. A 40-char string actuall takes 320bits. An 128bit integer comparatively occupies 16 characters. Remember that a "char" is 8bit.
  2. An IPv6 address can have different representations as string -- for example "2001::1" or "2001:0000:0000:(etc):1", how are you going to handle that? The only sane thing to do is store them as what they are, and that is 128bit integers.

This is effectively an issue : using VARBINARY(16) is much more interesting.

So I have done another (simplier) patch that allow IPv6 in visit log. The only issue is that Piwik_Common::getIp() returns an in_addr structure and not a decimal IPv4 address (this will probably lead to an issue if another module uses this function and expect an integer…).

Changed 3 years ago by veretcle

comment:26 in reply to: ↑ 24 Changed 3 years ago by sple007

comment:27 Changed 3 years ago by vipsoft (robocoder)

  • Resolution set to fixed
  • Status changed from new to closed

(In [4533]) fixes #1111 - add support for IPv6 addresses (tracking, anonymization, and exclusion)
fixes #2095 - add new anonymization hook (pre-heuristics)
fixes #2055 - optional IP filter when multiple proxies present
fixes #1775 - SitesManager: supports CIDR notation for IP exclusion

Notes:

  • Installer no longer checks for IPv6, so the related messages should be deleted from translations
  • IPv4 mapped addresses (e.g., ::ffff:127.0.0.1) are no longer re-mapped into IPv4 space
  • users who to query IP addresses from MySQL directly, can use the following SQL, but inet_ntoa() is limited to IPv4 addresses:
    select inet_ntoa(conv(hex(location_ip), 16, 10)) from piwik_log_visit;
    
  • Windows: IPv6 inet_pton()/inet_ntop() not supported until php 5.3; see #2351

comment:28 Changed 3 years ago by vipsoft (robocoder)

(In [4534]) refs #1111 - debug off

comment:29 Changed 3 years ago by vipsoft (robocoder)

(In [4535]) refs #1111 - fix some of the unit tests; will look at the failing integration tests later

comment:30 Changed 3 years ago by absynth

IPv6 support for the detailed visitor log does not seem to work. Visitors via IPv6 do not show up in the detailed log. Realtime visitor log works, though.

comment:31 Changed 3 years ago by vipsoft (robocoder)

worksforme. please make sure you're viewing the correct date/period.

comment:32 Changed 3 years ago by absynth

ACK, WFM too, the date was wrongly chosen.

Note: See TracTickets for help on using tickets.