New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve CSV/TSV Export #2714
Comments
|
|
|
Here's an example: Actual views:
Full report:
If we export it like this, it's easier to read for humans and might be useful in some cases. But if you want to sum the pageviews, the sum is 8, which is incorrect. If we export without inner nodes, it looks like this:
The sum is 5, which is correct.
|
I finally understand, nothing beats a good example ;) Maybe the parameter could be called "hideExpandedRows" ? |
"hideExpandedRows" sounds a lot like "hideRowsWithSubtable". The name should make clear that only the inner nodes of the tree (i.e. the rows with aggregated values) but not their children are hidden. I couldn't come up with anything better than "includeInnerNodes"... |
In general, the default for includeInnerNodes should be 0, since this is the version of the output that is most useful for further processing. But what about the export icon in the Piwik UI? Should it add |
(In [5334]) refs #2714 improved csv/tsv export |
There are columns in the export that are not used in the UI. Therefore, they are not documented (and translated) in the metadata. I added generic translations to the CSV-Renderer (Piwik_DataTable_Renderer_Csv::translateColumnNames), which reduces the impact of this ticket on other parts of Piwik. Please review and consider alternative options: we could add some of the metrics to Piwik_API_API::getDefaultProcessedMetrics or extend the metadata output of all plugins. |
On the build server, a unit test is failing that passes on my machine. When exporting with The comma in the number also breaks the CSV format where commas are used to separate columns. |
(In [5335]) refs #2714 changed expected number format of csv export according to behavior of build server |
(In [5336]) refs #2714 using API_datatable_default_limit config for export icons, upping default filter limit to 100 (which was the value previously hardcoded in the datatable template) |
(In [5345]) Refs #2714 remove echo |
Cool! Feedback
|
A general problem with CSV/TSV is, that the download starts right away and the URL can't be modified afterwards (e.g. to change includeInnerNodes). Since the possible parameters are getting more and more complex, it might be a good idea to introduce an export dialog. It could look like this:
This way, normal users don't have to read the API documentation (which will probably be too hard to read for them) and still can use all the cool features. What do you think? |
Sounds interesting, but I think a bit too complicated. The simple approach of putting expanded=1&translateColumnNames=1 in the URL is good. Also as proposed, the includeInnerNodes would always be set to 0 (when expanded=1) but this is done in the code, not in the URL. Sane defaults are better than offering too much control ;) |
Concerning comment:15, how about creating a new ticket/place holder on the subject of defining an advanced export page in Piwik ? |
(In [5382]) Refs #2714 Remove idsubtable metadata column from output being tested since it has different value in build server and local dev station |
At the moment, there's a list of translations in I'm not happy with this solution because the renderer should not care about the concrete translations. Also, when we add more "invisible" columns, we'll have to add them to the renderer, which is not intuitive at all. I think, we should include the invlisible columns in the meta data output. Adding them to "metrics" could be problematic because "metrics" should only contain columns that are relevant for the UI. How about adding a new array? We already have "metrics" and "processedMetrics", maybe "additionalMetrics"? |
I agree that column labels in renderer is not ideal... but adding data in the output, that we don't yet use is maybe not better (performance overhead). As a quick fix, maybe you can move the "Core/translation" logic out of the renderer into a public static function in API/API.php (in the plugin class rather than public API) - so that at least all logic is in the same place. I think it'd enough? |
(In [5458]) refs #2714 removed outdated comment |
(In [5463]) refs #2714 column translations for html and rss export, general fine tuning of export column translations |
As far as I'm concerned, we can close this ticket as soon as #2765 is fixed. |
Otherwise, excellent changes! It feels good to see very old features being polished :) |
(In [5468]) Refs #2714 Prevent warning when cookie not init |
I still think we might need both options (includeInnerNodes=(0|1)). Leaving the inner nodes out is better for further processing because the sums are right. On the other hand, some reports contain valueable information on inner nodes only. For example, the search engines report has conversion metrics for the search engines but not for the keywords. If we leave inner nodes out, only the keywords are exported and we don't have the metrics. At the moment includeInnerNodes is 1 when the export icon is clicked but the API default is 0. I did it this way because it's more likely that an auto-fetched report is processed further. Still, I'm not entirely happy...
|
Thx for clarifications.
Sounds good to me... There is no known problem with that, we will add a FAQ if users ask about it. It is the easiest so the way to go maybe since we want to KISS... |
(In [5477]) fixes #2714 includeInnerNodes=1 by default, nice export filename for API.get report |
The realted ticket #2765 is still open. If it was closed, CSV export would be polished completely. Matt, did you work on it yet? |
No and I will probably not fix it for the next release I'm afraid... |
(In [5495]) refs #2714 use default language of user for export file name |
(In [5496]) refs #2714 export filename compatible with firefox |
(In [5838]) Refs #2714 |
(In [6181]) refs #2714, refs #3073
|
The improvements include the following API extensions
expanded=1
should work for CSV/TSV exports. Labels have to be put together in a path.includeInnerNodes
to choose whether inner nodes (e.g. folders in the pages report) should be exported as well. If set to 1, they are present and the report is easier to read for humans. If 0, they are left out and it's easier to process the report, e.g. calculate sums.translateColumnNames
to get localized names of metrics.The export link in Piwik should be adjusted
expanded=1
by defaulttranslateColumnNames=1
by default for CSV and TSV formatAPI_datatable_default_limit
configuration for export linksThe text was updated successfully, but these errors were encountered: