Opened 18 months ago

Closed 17 months ago

Last modified 14 months ago

#3456 closed New feature (fixed)

Create auto-updater & auto-downloader for GeoIP databases

Reported by: capedfuzz Owned by: capedfuzz
Priority: critical Milestone: 1.10 - Piwik 1.10
Component: Core Keywords:
Cc: Sensitive: no

Description

One of the improvements in: #3442

The goal of this ticket is to create an admin function that will automatically download a GeoIP database after Maxmind updates it.

  • It should support free & commercial databases.
  • There should be a way to download a free database to get started using GeoIP as quickly as possible.
    • The download should display a progress bar.
  • This download should take place during installation if possible.

Change History (18)

comment:1 Changed 18 months ago by matt (mattab)

  • Milestone set to 1.9.2 - Piwik 1.9.2

comment:2 Changed 18 months ago by matt (mattab)

  • Owner set to capedfuzz
  • Priority changed from normal to major

comment:3 Changed 18 months ago by matt (mattab)

  • Priority changed from major to critical

comment:4 follow-up: Changed 18 months ago by halfdan

Can't that just be done via a Scheduled Task?

For the GeoLite City database the task would only have to run on the first Wednesday of each month. (The database is updated on the first Tuesday of each month - http://www.maxmind.com/en/geolite)

That of course leaves out the commercial database.

On a related note: Maxmind offers a Web API for Geolocation that could potentially be added as a provider (http://www.maxmind.com/en/web_services), although I think it would hurt the performance of tracking when calling the Maxmind API for each request.

comment:5 in reply to: ↑ 4 Changed 17 months ago by hpvd

Replying to halfdan:

On a related note: Maxmind offers a Web API for Geolocation that could potentially be added as a provider (http://www.maxmind.com/en/web_services), although I think it would hurt the performance of tracking when calling the Maxmind API for each request.

yes using web api would be very interesting. Some words on this topic are also posted in ticket #3442

Last edited 17 months ago by hpvd (previous) (diff)

comment:6 Changed 17 months ago by capedfuzz (diosmosis)

  • Resolution set to fixed
  • Status changed from new to closed

(In [7550]) Fixes #3456, added updating scheduled task for GeoIP databases w/ manager UI and easy-install for GeoLiteCity.

Notes:

  • Added new PEAR lib Archive_Tar and new Unzip implmentations for .tar.gz, .tar.bz2 & .gz files.
  • Modified Http class to allow use of Range HTTP header.
  • Added ability to download file in chunks to Http class.
  • Moved GeoIP admin page styles to separate CSS file.
  • Allowed monthly scheduled tasks to specify day of week to run on.

comment:7 Changed 17 months ago by capedfuzz (diosmosis)

(In [7576]) Refs #3456, add extra security for GeoIP auto-updater/downloader & fix a couple bugs in the code. Also made the UX a tiny bit more robust.

comment:8 Changed 17 months ago by capedfuzz (diosmosis)

(In [7577]) Refs #3456, various UX improvements to GeoIP updater/auto-downloader.

comment:9 Changed 17 months ago by capedfuzz (diosmosis)

(In [7578]) Refs #3456, remove downloaded files on all exceptions, even sanity check.

comment:10 Changed 14 months ago by capedfuzz (diosmosis)

In 4169d90b648d695d1ce40a8345cff2c5c8e7b2fd:

Refs #3456, add redundant checks of GeoIP database wellness to GeoIP auto updater.

comment:11 Changed 14 months ago by capedfuzz (diosmosis)

In 91098c7f1b817b5ca9442445bb4cdaca4185ab20:

Refs #3456, only save one broken file when doing redundant checks & add unit test for checks.

comment:12 Changed 14 months ago by diosmosis <benaka.moorthi@…>

comment:13 Changed 14 months ago by diosmosis <benaka.moorthi@…>

comment:14 Changed 14 months ago by mattab <matthieu.aubry@…>

In 5aa918f8549c3a64318d88cbccc29aa560567827:

this should help the build refs #3456

comment:15 Changed 14 months ago by Fabian Becker

comment:16 Changed 14 months ago by Fabian Becker

comment:17 Changed 14 months ago by Fabian Becker

In 7f9332dd486c57c3a720732d3d5e794406ad0e3e:

this should help the build refs #3456

comment:18 Changed 14 months ago by capedfuzz (diosmosis)

In 34d65f2442c8a3d929da1092bb506b0520ab16de:

Refs #3456, if geoip updater has not been run, display inline help saying so in geoip updater setup.

Note: See TracTickets for help on using tickets.