This was mentioned in #1514.

This should be implemented in the next few weeks.

So, I will start the map development in the next days. There are a couple of things to do:

• porting the map preparation script to Python, reading shapefiles, projection, simplification, AMF export
• extracting all countries sub-regions from shapefiles found at http://gadm.org (seem to be complete for all countries)
• updating the frontend SWF

However, there are some open problems.

• How do we map the geo-locations to the country regions. I think this should be done once every time the GeoIP database gets an update. There are two possible ways to do this:
  • by checking point-in-polygon for each lat/lng with every region polygon and storing the result (lat/lng > country_region). This is almost accurate, but some cities may lay directly on a region border and thus may be wrongly mapped.
  • another way could be to use the data we get from the GeoIP database. At least for some countries there should by some information about the region, e.g. the state id in USA. In any case we need to map the country region ids (which are defined in the shapefiles) to the country regions ids in the GeoIP database. However, i'm not sure if this information is available for every country.
• In both ways, this would require some install script to run every time the user updates his GeoIP database. Do you think this is possible?

Attachment: A mockup of the new city view
cities2.png

Attachment: A mockup of the new region view
regions1.png

Note, in this image you can see an overview about all available region outlines. Think this is pretty much complete.

http://gadm.org/img/gadm_v1_level1_high.png

Greg, great news that you will resume work on this :)

GADM data looks good and there for all countries, this sounds fantastic, nice find!

In both ways, this would require some install script to run every time the user updates his GeoIP database. Do you think this is possible?

It is possible to run a script on every GeoIP update, but not a good solution for ease of use. Here is a proposal: Maybe, we could run the script once before Piwik release (and commit the file to SVN after checking it's OK). We would use the latest GeoIP database for this. Then, if users use an older version (or newer version) compared to the pre-generated DB, maybe they could run the script manually themselves?

But, I would like to ask about this script, what exactly will it map?

My current understanding is that Piwik will report:

• top cities + lat/long: visits, pages, etc.
• top regions (as per this specification): visits, pages, etc.

Is the algorithm designed to map "regions" according to GeoIP, to "regions" in the Piwik map?

I guess that, for each GeoIP region, we could give one lat/long of a city belonging to this region. If so, do we need a database for this, maybe the SWF could map in real time the lat/long to the pixel inside the region?

Thanks for claryfying

Replying to matt:

It is possible to run a script on every GeoIP update, but not a good solution for ease of use. Here is a proposal: Maybe, we could run the script once before Piwik release (and commit the file to SVN after checking it's OK). We would use the latest GeoIP database for this. Then, if users use an older version (or newer version) compared to the pre-generated DB, maybe they could run the script manually themselves?

My current understanding is that Piwik will report:

• top cities + lat/long: visits, pages, etc.
• top regions (as per this specification): visits, pages, etc.

Is the algorithm designed to map "regions" according to GeoIP, to "regions" in the Piwik map?

I didn't knew that the GeoIP DB contains a complete list of fips regions. So Piwik already knows how many visitors each region had? In this case, there's no need for another update script. Now, all I have to assure is that the map region ids are the same as the GeoIP/FIPS region ids.

Just to make sure I understand everything, here's a sample request flow:

• At first, the client loads the Piwik Map SWF
• the Piwik Map SWF will then request the vistors per country via Piwik API and display the world map
• now the user can click on a continent to zoom in
• now the user can click on a country
• the map SWF will now load the detailed map data for this country (and it's regions)
• at the same time, the map SWF will request the visitors per region (of this single country)
• once the SWF received all data, it will display the regions map as shown in regions1.png
• now the user can switch to the city mode
• the map SWF will now request he visitors per city for this country. The API will serve a list of cities including their lat/lng coords with the number of visitors for each city.

Did I got it right?

Thanks

Yes it sounds good!

One other feedback, would be to give labels to the icons to switch to city/region view, and to the button to zoom in/zoom out, since these buttons are very important and must be easy to reach.

I think that we must check that GeoIP data will output the regions as we expect them, ie. that all visitors are indeed assigned in one of the regions listed in: http://www.maxmind.com/app/fips10_4

I will double check this and confirm here.

Is there any other open question appart from this one?

Greg, will the Flash maps know about all cities in the GeoIP Database?

Or, will the flash map expect a list of cities & lat/long, and plot them "blindly" in the flash map (projecting from lat/long to pixels, and drawing City name based on Piwik API as input) ?

The flash map doesn't know anything about the cities. Instead the map is able to project lat/lng coordinates to the exact pixels. I'm not sure if it will display all cities "blindly", maybe there will be some simple clustering of cities that are very close.

One thing I think of, is that sometimes GeoIP will not return region/city info (or just, not city some other time). So, maybe we can plan to display the "Unknown" on the map somewhere, discreetly?

Because, I guess the % displayed, will take into account the % for the Unknown region/city?

You mean that for some visitors GeoIP only knows the country but not the city and/or region? We can add a "Plus X unlocatable visitors" text somewhere to display this data. Good point.

Yes, we better expect the worst with geoip free edition, all use cases can happen. I think Piwik will return, for a country's region, the 'Unknown' (or 'Other') row that will contain these. simple text "X visits couldn't be located" sounds good!

Sorry, I'm late joining this discussion.

I'm going to add a FIPS data file as part of my work in #1823 to convert the region names into the more compact FIPS 10-4 code when storing it in the log_visit table.

• Do the region names in MaxMind's FIPS file correspond 1-to-1 with the region names used by gadm.org, or does the data need to be massaged?
• Would it help if I provide a reverse lookup, i.e., FIPS code to region name?

re: the MaxMind FIPS file

• names are (English) localized without any special characters
• there are almost 4000 entries which means we shouldn't expect translations
• the codes are used inconsistently, i.e., the data switches to ISO for US states and Canadian provinces

Partial feedback:

names are (English) localized without any special characters
Region names are in English charset, but they are not translated in English. For example french regions are written in french. Maybe this is not true for all countries.

Would it help if I provide a reverse lookup, i.e., FIPS code to region name?
I think, that the API output for getRegions (eg.) should contain the region code, and the full region name, like we do for Country API output (which includes the country code and full name, icon path, etc.)

Just wanted to let you know that I've got plenty of stuff to do right now. Hope to be able to continue working on this feature soon. Sorry for the delay..

I have just seen an interesting project: jVectorMap for canvas+JS Map library.

Greg, what do you think about this work?
hope your work load is getting better :)

Hi Matt,

jvectormap looks quite nice. I'm currently thinking a lot about JS/SVG based mapping myself, even developed some early prototypes while working on other projects. Still, our biggest challenge is how to build the map data files for every country (including admin level2 regions).

Thus, one of my next steps is to setup a Mapnik server and let it export clean SVG projections of shapefiles. My goal is to do that by writing as less code as possible. Mapnik is such a powerful library, so it should be almost only a matter of configuration.

After we created the map files, we still have the choice to either use one of the currently emerging JS/SVG mapping libraries (like jVectorMaps) or to develop a mini library especially for Piwik by ourself.

Besides of that, I fully agree on JS/SVG instead of Flash. Quite a challenge, but possible.

and, yes, my work load starts getting better. :)

Greg, I'm happy to hear you are keen on SVG for your own work. As you may have seen in the blog recently, we are now using canvas graphs only, and Timo from the team has contributed many patches to jqplot to make it work nicely in our use case. Hopefully, an existing library can meet our needs in terms of performance, maintainability, features, licensing.

In any case this work will be reused, not only in Piwik but I'm sure in hundreds of other projects in the future, since we are building the first truly open source world mapping with region details for all countries.

With shapefiles, I hope you can find a format that is of low size and with good shape quality, that must be quite a challenge.

Once the Maps show cities and region, we definitely have to show the maps in the actual Country report: full awesomeness! see #1821

See also the blog post by greg: http://vis4.net/blog/posts/piwik-maps-2/

I now proceed with this task. As a first step, I looked at the shapefiles provided by gadm.org. Since some people seem to be very interested in my work (I got plenty of mails after finishing the first version of the map), I decided to blog about my progress. I will post the links in here.

Part 1: Finding a map data source:
http://vis4.net/blog/en/posts/recreating-the-piwik-map

Greg; excellent first step, looking forward to the next part!

Regarding the simplification of shape files, it is obvious that we don't need any detail around Chile, a rough outline of the coast would do perfectly (we can't affort all the little islands ;).

Also is it possible to add a test not to plot islands less than 10 square km, or something similar?

It is really key for user experience to have the smallest file size possible to ensure fast downloading and JS parsing / CPU usage. :)

I just looked into a world shapefile and computed the areas of all 3761 polygons (a country may have multiple polygons for islands etc). Filtering every polygon smaller than 10 square km would remove 434 polygons (=11%). I checked the names of the countries which the removed polygons belong to and found many island states among them (as expected).

However, I think a hard cut at 10 sq km has some problems:

• it removes some countries entirely (like the Maldives Islands)
• it still keeps very many small islands (starting from 11sqkm)

See sample rendering.

I tried a different rule: remove all polygons smaller than 5% of the maximum polygon area of that country. Since the maximum area of the Maldives is 9sqkm, all islands are kept, while all small islands of the USA are removed. Also this halved the resulting SVG file size keeping the map correct in terms of includedness of countries.

See sample rendering.

However, removing every polygon smaller than 5% of the maximum per country is not satisfactory as well. Big and also well-known islands like Hawaii or Novaya Zemlya shouldn't be removed in my opinion.

Also there is a problem with those tiny islands countries like the Maldives, which is obvious if you look at the zoomed view on the Maldives. The islands are way too small to be rendered in a meaningful way.

This leads to some important questions: Does the map needs to include all countries or is it acceptable to ignore some countries, like the Maldives Islands?

In the old map widget this wasn't important because there was no country level view.

I think would like to keep islands and outlying regions in the country-level views. For instance, I could try to generate composed maps that mix different projections to fit the complete country into the map.

Like in this example for the United States.

One note regarding the projection used for the country-level views. I would like to simplify the whole thing by using the same projection for every country, but with different parameters (namely the projection center).

One of the simpler projections is the orthographic projection, which looks the same as if looking onto a 3D globe. This gives quite good and less-distorted views for almost every country. The only country that looks quite distorted is Russia. Because of it's huge area, Russia takes too much space on the globe.

You can have a look at distorted Russia here.

My opinion is that this distortion is acceptable given the simplicity of rendering maps in a single projection. In an ideal mapping world, one would use specialized projection for every country, like the New Zealand Map Grid for New Zealand etc.

Any more opinions?

The next design decision has to do with the aspect ratio of the maps. At some point in the map generation process I need to crop the country-level maps to a bounding box. The idea is to fit the countries as big as possible into the views while also showing a bit of their neighbour countries for navigation.

Now the question arises to which aspect ratio the maps should be cropped. As far as I see do we have to options:

1. A fixed aspect ratio (presumably some wide-screenish format) would be the simplest solution. The complete cropping could be done in the preprocessing stage, which would reduce complexity of map rendering. However, the obvious drawback of this solution is that the maps won't look as good as they could when displaying countries in the "opposite" aspect ratio. For some portrait countries (like Germany), this should be acceptable. For other, more extreme aspect ratios (my favourite example is Chile) don't.
2. In the latter cases, a dynamic aspect ratio would make much more sense. It would be perfect if we could present the Chilean Piwik users (I assume there are some) with a portrait view of their country. Dynamic aspect ratios could be done either by choosing a fixed ratio per country or by cropping the maps at rendering time. Choosing a fixed ratio per country has the drawback that those countries could not fit a landscape ratio in fullscreen mode. In contrast, cropping at runtime may take more CPU. Also, I don't know if the current dashboard supports dynamic resizing of widgets, but this may be easy to implement.

I tried a different rule: remove all polygons smaller than 5% of the maximum polygon area of that country.

Sounds good

This leads to some important questions: Does the map needs to include all countries or is it acceptable to ignore some countries, like the Maldives Islands?

Removing countries all together is I think not a good idea. Maybe we could still display ALL countries, but leave the rule of 5% for all other countries. (eg. Maldives would be displayed with at least the main islands (if possible..), while canada would still lose many big islands.

I think would like to keep islands and outlying regions in the country-level views. For instance, I could try to generate composed maps that mix different projections to fit the complete country into the map.
Like in this example for the United States.

If you can do that (at least for some selected countries?), it would be pretty cool!

My opinion is that this distortion is acceptable given the simplicity of rendering maps in a single projection. In an ideal mapping world, one would use specialized projection for every country, like the New Zealand Map Grid for New Zealand etc.

Simple is always better, even if the shape is not perfect. We could improve this later anyway. What would it look like for NZ? ;-)

The idea is to fit the countries as big as possible into the views while also showing a bit of their neighbour countries for navigation.

Also, maybe it would useful if hovering over the countries next to the zoomed country, would display a little tooltip with the Country metrics. Maybe it could be displayed next or below to the main country metrics (which would always be displayed?).

fixed aspect ratio (presumably some wide-screenish format) would be the simplest solution.

Agreed. I think it is expected that all country zooms have the same dimensions. dashboard doesn't support dynamic size widgets at present (not planned).
Chile looks good in the map, even with a lot of sea. We can use the space for legends, metrics, etc.

The dynamic aspect ratio could be implemented later in the lib for static country-specific maps...

Thanks for posting your thoughts and log here, cheers!

Just added

• three-level-zoom
• coloring for world and continent maps (based on the existing API).
• selector for countries and contintents

http://piwik.vis4.net

Added fake display for regions and cities (100 randomly chosen). Also the widget now remembers the country/continent and the view mode (region/city).

Test it live:
http://piwik.vis4.net/index.php?module=CoreHome&action=index&idSite=14&period=day&date=yesterday

Beautiful & Amazing work!! :-)

after 1 hour testing I found some bugs and have a few questions:

• when clicking on Netherlands, Belgium is not clickable
• UK has too many regions, but you mentionned the other viewe with only 6 regions which might be too small -- ist here anything in between?
  • Also can we really choose different region mapping, will it work with GeoIP DB? Or will you do point in polygon or something similar? I'm still unsure how it all works haha ;)
• some countries, venezuela, russia, thailand, do not display the tooltip on hover on regions, also it seems imposible to click on neighbouring countries
• in NZ specifically, there is no region "Wellington" but there should be sincec it's the capital city - ws the region removed because too small or simply absent from original dataset? I'm curious if similar regions missing could happen in other countries as well.
• when looking at panama, the country on the left (Costa rica) is not clickable

Here is my UX review & feedback:

• Could a click on ocean/sea unzoom the map? this would make it more user friendly as one wouldn't have to aim for the icons in the footer
• Updates to the tooltip
  • On a continent view, the tooltip for each country could say "France - 145 visits - 5% of all visits"
  • On a country view, for a particular region or city, the tooltip could say "Poitou Charentes (France) - 15 visits - 10% of visits from France - 0.5% of all visits"
  • When a country has no visit, could the tooltip say "Panama - No visit" rather than not display the tooltip which could be less easier to understand
• Unknown visitors could be displayed on the world / continent views, for example bottom right in grey?
  • Above Unknown, on the World view (or continent view) we could also display "All visits: 5432 - Unknown visitors location: 433 (15%)" so that users always know
  • When on a county region or city zoom, this bottom right legend could display "All visits: 5432 - Visits from France: 145 (2.7%)"
• In the footer, could we display "Regions" next to the icon, and "Cities", similarly to: http://piwik.org/wp-content/uploads/2011/05/Ecommerce-Best-products-report.png - there might be some space missing, could we display the text only for the currently selected icon, and on hover on the icon?
  • To save much space, please set eg. the SELECT for metrics and country with font-size: 11px; then it should be enough space to write the Region/City legend?
• During the time anything is loading, could we have low opacity on the map, and have the rolling wheel in the middle 100% opacity? It would feel smoother I think, currently there are some screen jumps/flashing elements as they are loading.

I believe with all these improvements the usability and ease to understand the map will be much improved. Did you have other ideas as well and/or does it sound good?

(In [7173]) filter_pattern= will now work on metadata columns. This will be useful for example, if we wanted to select only a number of "countries", from the UserCountry.getCity API, we could do the following: &filter_column=country&filter_pattern=de|fr|es|it|nl
It will filter and only return only the Cities, which belong to the country specified. This will work ,because rows have a metadata <country>
Refs #1652

(In [7498]) Fixing some tests, also adding <city>xx</city> to tell unknown cities from others, as requested by Greg Refs #1652

In #3585 I added a new parameter &showRawMetrics that will return all raw metrics in the API metadata getProcessedReport call.

(In [7643]) deleting the map plugin not working and will revert to previous version Refs #1652

We are starting crowd funding for this feature! Participate in this fantastic project by donating at: http://crowdfunding.piwik.org/analytics-maps-world-country-city-region/

(In [7644]) Refs #1652 Maps

We need YOUR help! We are running a crowd funding campaign to raise funds to implement the detailed Visitors Maps of Countries, Regions and Cities (for all countries)!

These maps will be beautiful, usable, and built using open standards SVG+JS. They will show detailed visitor count, conversion rates, by Country but also (New!) by city and region.

Pledge now at: http://crowdfunding.piwik.org/analytics-maps-world-country-city-region/

Piwik needs you!

In b9c192c: Refs #1652 Adding Continent+ Country codes to Live API output for easier plotting

In 7bf551f: Fixes Integration test after changes to display Continent+Country Code

refs #1652

In e54bb8d: Fixes Integration test after changes to display Continent+Country Code

refs #1652

js/vendor/kartograph.is has conflicting licenses GPL and LGPL.

js/vendor/kmeans.js is missing any sort of attribution/license.

PiwikMap references in LEGALNOTICE should be removed.

Remove swfoject per #3666.

@vipsoft, all fixed now!

In 62402bd: Refs #1652 Encoding the JS output for making map work in languages with single quotes in strings, eg. French

Real time Visitor Map and aggregated World map + Countries + Regions + Cities were released in Piwik 1.11!