Context Navigation

← Previous Ticket
Next Ticket →

Ticket #703 (reopened New feature)

Opened 3 years ago

Last modified 35 hours ago

Piwik an alternative to AWStats and Urchin, build server log import script

Reported by:	bharatkalyan	Owned by:
Priority:	critical	Milestone:	1.7.2 - Piwik 1.7.2
Component:	Core	Keywords:
Cc:	Cyril	Sensitive:	no

Description (last modified by matt) (diff)

Urchin Alternative: Import your server logs in Piwik, the Free web analytics platform!

See blog post Piwik alternative to Urchin for more information.

Piwik is the Urchin alternative but also Webalyzer and AWStats alternative: with a Python script, you can now import webserver logs (apache, iis, and more) in Piwik, instead of using the javascript tracking.

Description A Python script available in piwik/misc/log-analytics/ will parse server logs efficiently and automatically call the Piwik Tracking API to inject the visits/pageviews/downloads in Piwik.

How to install / how to use

Requires Piwik >= 1.7.2-rc2. Download the latest version from http://builds.piwik.org/?C=N;O=D
Requires at least Python 2.6
Requires one or many server log files, typically called access.log in Apache for example. These log files will be imported into Piwik.
You can also create a "test website" in Piwik to import all data into, rather than importing into your existing websites. Then, use the command --idsite=X to force all info from the log files to be imported into this idsite
You can use --dry-run command to have a test run and make sure you will not track data or create new websites

How you can help?

please use the script and report your feedback and bugs here
if you are a hacker yourself, please review the code and consider submitting performance optimization, or improvements.
If you are a webhost or web agency and wish to offer Piwik to hundreds of your customers, please contact us
review the doc at Server log analytics

Tasks to do before final release

Test, test and test
Setup on demo.piwik.org in a new website
Check all code review feedback managed
Review Import Logs in Piwik doc page.
decomission apache2piwik (update blog post)

Feature requests for V2 or later

detect that a given log file has already been imported and don't import again
When CTRL+C, message is displayed. Could we also display the --skip=X value to resume where we cancelled the script?
make it easy to delete logs for one day only.
- This would be a new option to the python script. It would reuse the code from the Log Delete feature, but would only delete one day. The python script would call the CoreAdmin API for example, deleting this single day for a given website. This would allow to easily re-import data that didn't work the first time or was bogus.

PERFORMANCE improvement ideas Performance improvements ideas

How to debug performance? First of all, you can run the script with --dry-run to see how many log lines per second are parsed. It typically should be between 2,000 and 5,000. When you don't do a dry run, it will insert new pageviews and visits calling Piwik API.

Ideas to improve log import performance:

Enable persistent connections
- In mysqli: In the config.ini.php, host = "p:localhost".
Use keep alive connections
- This requires a new library to use in Python
- Only implement if gain is significant: let's do performance testing first and then see.
Bulk load requests to Piwik via POST see #3134
- Once we have bulk loading, we could even improve and optimize the PHP code. For example, we could issue less SELECT query since we know the last status of a visit in memory

Change History

Changed 14 months ago by matt

status changed from closed to reopened
description modified (diff)
component changed from UI (templates, javascript) to Core
summary changed from HOW TO USE PIWIK LIKE AWSTATS WITH WEBSERVER LOGFILES to Piwik as an alternative to AWStats
priority changed from critical to major
sensitive unset
milestone changed from RobotRock to 1.x - Piwik 1.x
resolution invalid deleted

Changed 2 months ago by oliverhumpage

Performance-wise: I've set up piwik in its own jail now, turned off unnecessary PHP extensions, tweaked apache, and enabled APC. If I use --recorders=48 I get good import speeds (at least at first) without the load average going too high. However, something odd happens, and some way through importing a log file the recorders drop off (I can see fewer and fewer apache processes too, so clearly it's just not being hit as much):

2846 lines parsed, 233 lines recorded, 233 records/sec
4372 lines parsed, 506 lines recorded, 273 records/sec
[...]
8300 lines parsed, 7570 lines recorded, 9 records/sec
8300 lines parsed, 7579 lines recorded, 9 records/sec
8300 lines parsed, 7588 lines recorded, 9 records/sec
8300 lines parsed, 7598 lines recorded, 10 records/sec

I don't think I have any weird throttling going on - any ideas what might be up? There's nothing else being output during the processing even with debugging on. The drop-off seems to start roughly half way through any given logfile.

follow-up: ↓ 36 Changed 2 months ago by Cyril

oliverhumpage: 48 is almost certainly too high, unless you have a 48-core machines. You shouldn't have to exceed the number of cores in your system, even a bit lower (as the import script and MySQL will run at the same time).

As for why your performance decreases over time, I don't know. What does a 'top' say? You'd have to find the bottleneck. It may be Apache, PHP, MySQL. On my system, I have a sustained 300 req/s for more than 3 hours.

Regarding the static files excluded, we'll add an option to include those (disabled by default). I'm sure the whole importing process will get better over time, it's only the beginning :)

in reply to: ↑ 30 Changed 2 months ago by oliverhumpage

Replying to Cyril:

oliverhumpage: 48 is almost certainly too high, unless you have a 48-core machines. You shouldn't have to exceed the number of cores in your system, even a bit lower (as the import script and MySQL will run at the same time).

I did quite a few experiments, and eventually found that 40 is about right. This is a VM running on a high powered Dell R710, so although the OS only thinks it has 4 CPUs I don't know how things actually pan out. All I know is that the number of records/sec increases pretty much linearly with --recorders up until 40. E.g. if I run at 32, I get more like 200r/sec rather than 250+r/sec. A single recorder manages around 6-7r/sec. After 40 the benefits tail off.

I also tried a few experiments to see where the bottleneck might lie, for instance I stuck in a mod_rewrite to send the importer to a basic PHP file that just returned the .gif without doing any processing, but weirdly the performance was about the same. However, running with --dry-run (or just removing the line which actually calls the script) means the python script runs at around 4000r/sec, so I can only conclude the limit is in apache/php (putting in APC definitely helped). I also tried hacking the script to run a PHP wrapper script that called piwik.php directly on the command line, but it went horribly slowly, presumably because of the lag in loading up PHP.

Anyway, I'm happy with 250-300r/sec. I may set up a separate VM with a tweaked kernel and optimised apache to deal with log imports anyway, so I'm sure I can improve on that figure.

Regarding the steady tailing-off, what I'm wondering is: when you specify lots of recorders, do they each grab an equal number of log lines at the start then work through them? That would explain why some finish earlier than others (if e.g. one gets a lot with non-loggable lines it'd finish sooner). I notice the number of apache processes starts tailing off around half to 2/3 of the way through the log, and then just steadily decline until only 1 recorder is left.

Regarding the static files excluded, we'll add an option to include those (disabled by default). I'm sure the whole importing process will get better over time, it's only the beginning :)

That'd be brilliant, thank you. Thanks to you all for being so responsive in general too.

Oliver.

Changed 2 months ago by matt

oliverhumpag, thanks for your comments it's very interesting! Since you seem keen, maybe you can consider running XHProf, the facebook php profiler: http://pecl.php.net/package/xhprof

I haven't run that for a long time and never under high load such as 300 req/s so it would be very interesting. If you install it, i would love to see the reports generated! The last time we ran XHPRof on Piwik we found 2-3 quick fixes that made things a lot better. I'm sure we can make tracker faster in many ways.

It would also be good to know the % of consumption of Apache/php VS mysql (not sure the best way to do this however?).

Changed 2 months ago by oliverhumpage

Actually, I do have one small request for piwik itself.

Would it be possible to choose on the fly between multiple database options: you see, I'm using one physical install of piwik at 2 different URLs - one for JS-based sites, and one for log-based, and therefore also 2 different sets of db tables so that --add-sites-new-hosts on the log-based system doesn't interfere with the JS websites (they'd have the same URLs). What I've done atm is set an environment var in apache and patch core/Config.php to set $config->database to either $config->database_weblog or $config->database_js depending on that env var.

However, being able to define a constant like DATABASE_CONFIG_SECTION_NAME in bootstrap.php, which Config.php then used to work out which section of the config file to use, would be much easier and more robust. I could of course just have 2 different installs of piwik, but then I have to update it twice with each release. Probably not worth enlarging the codebase just for my weird setup, but thought I'd ask - I can easily submit a patch if you're interested.

Changed 7 weeks ago by oliverhumpage

Cyril:

Have tested using - instead of /dev/stdin, seems to work fine.

Re the regex, I think that's explained in the comments: because I want it to pick up hostnames that are subsites and so have slashes (e.g. I want the hostname 'domain.com/subsite' to be picked up and created with that name in piwik), I needed to amend the normal vhost regex to allow "/" in the host character class. It's also a very good example of what and how to escape shell special characters in apache log pipes :)

(I spent a fun morning with a test apache installation and a perl script testing each special character in turn until I got it working... then a fun afternoon wondering why it wasn't working with import_logs.py, until I realised there wasn't a .compile for the custom regex!)

I did originally put things like "domain.com.subsite" in the hostname so the standard regex would work, but it looks ugly and non-user-friendly in piwik.

follow-up: ↓ 92 Changed 7 weeks ago by tiouk

The script doesn't parse IIS6 or IIS7 log files (not tried IIS8). I tried the following regex that matches the log lines in kiki but no luck with the script. Any pointers?

Also minor change to line 1068

' the --format option'

needs updating to

' either the --log-format-name or --log-format-regex option'

_IIS6_FORMAT = (
    '(?P<date>^\d+[-\d+]+ [\d+:]+) '
    '\S+ \S+ [\d*.]+ \S+ '
    '(?P<path>\S+) '
    '\S+ \d+ \S+ '
    '(?P<ip>[\d*.]*) '
    '\S+ '
    '(?P<user_agent>\S+) '
    '\S+ '
    '(?P<referrer>\S+) '
    '\S+ '
    '(?P<status>\d+) '
    '\S+ \S+ '
    '(?P<length>\S+)'
)

Changed 6 weeks ago by schrefel

for some years I was looking for an alternative to awstats and with your import script I think I've found it - great work so far.

But I've troubles with the log file. We use a Lotus Notes clusters and for each server in the cluster we've a seperate log file per day. The import is working but the result isn't ok and I think it because of the log file format.

It looks like this: 192.168.1.1 bene.com - [01/Apr/2012:00:00:01 +0200] "GET /mobiliario-de-oficina/news-filo-design-preis-2009.html HTTP/1.1" 200 20719 "" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@…)" 1453 "" "D:/Notes/Data/benecom/cont_es.nsf"

in awstats I can describe the log file format like this: LogFormat=%host %virtualname %lognamequot %time1 %methodurl %code %bytesd %refererquot %uaquot %other %other %other

and http error and redirects are also not found:

17758 requests imported successfully 542 requests were downloads 0 requests ignored:

0 invalid log lines 0 requests done by bots, search engines, ... 0 HTTP errors 0 HTTP redirects 0 requests to static resources (css, js, ...) 0 requests did not match any known site 0 requests did not match any requested hostname

See more log at: http://pastebin.com/zSMXqEpu

Changed 6 weeks ago by matt

Also the log format Websphere is not detected:

example 20 lines log: http://pastebin.com/eY7N5weT

Error is

Traceback (most recent call last):
  File "C:\Python26\lib\threading.py", line 522, in __bootstrap_inner
    self.run()
  File "C:\Python26\lib\threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
  File "c:\wamp\www\piwik\misc\log-analytics\import-logs.py", line 756, in _run
    self._record_hit(hit)
  File "c:\wamp\www\piwik\misc\log-analytics\import-logs.py", line 794, in _reco
rd_hit
    'url': main_url + hit.path[:1024],
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 148: ordinal not in range(128)

Changed 5 weeks ago by guardian

I'm having problems with the script which enters an infinite loop

6846 lines parsed, 1919 lines recorded, 0 records/sec
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/share/www/piwik/misc/log-analytics/import_logs.py", line 860, in _run
    self._record_hit(hit)
  File "/usr/local/share/www/piwik/misc/log-analytics/import_logs.py", line 924, in _record_hit
    headers={'User-Agent' : hit.user_agent},
  File "/usr/local/share/www/piwik/misc/log-analytics/import_logs.py", line 694, in call
    return self._call_wrapper(self._call, expected_content, path, args, headers)
  File "/usr/local/share/www/piwik/misc/log-analytics/import_logs.py", line 676, in _call_wrapper
    response = func(*args, **kwargs)
  File "/usr/local/share/www/piwik/misc/log-analytics/import_logs.py", line 625, in _call
    response = urllib2.urlopen(request)
  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1178, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.6/urllib2.py", line 1143, in do_open
    r = h.getresponse()
  File "/usr/lib/python2.6/httplib.py", line 990, in getresponse
    response.begin()
  File "/usr/lib/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.6/httplib.py", line 355, in _read_status
    raise BadStatusLine(line)
BadStatusLine

6846 lines parsed, 1919 lines recorded, 0 records/sec
6846 lines parsed, 1919 lines recorded, 0 records/sec
6846 lines parsed, 1919 lines recorded, 0 records/sec
6846 lines parsed, 1919 lines recorded, 0 records/sec
6846 lines parsed, 1919 lines recorded, 0 records/sec
...

and it keep repeating

6846 lines parsed, 1919 lines recorded, 0 records/sec

The offending line 1920 in my nginx log is:

62.83.238.31 - - [17/Apr/2012:12:53:35 +0200] "-" 400 0 "-" "-"

Changed 5 weeks ago by julianito

Hi all,

I have big problems with the import, I'm only importing 15-20 lines per second. I need to import more than 20Milions of lines. Really I need more than 10 days to import these logs?

I tried to use the command with --dry-run, but this command not insert the lines in the database, it's just to check if the comand works well.

How can I improve the import speed? (I tried also with recorders=16, but don't work well. I only have 2 cpus -4 cores-)

Do you have any guide or something like that to follow it and improve the import speed?

Thanks in advance for your feedback.

I'm using this command: python /var/www/piwik/misc/log-analytics/import_logs.py --url= http://localhost/piwik access_log.0 --idsite=2 --recorders=4 --enable-http-errors --enable-http-redirects --enable-static --enable-reverse-dns --enable-bots

Changed 5 weeks ago by guardian

6846 lines parsed, 215 lines recorded, 2 records/sec
6846 lines parsed, 215 lines recorded, 0 records/sec
6846 lines parsed, 215 lines recorded, 0 records/sec
6846 lines parsed, 215 lines recorded, 0 records/sec
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/share/www/piwik/misc/log-analytics/import_logs.py", line 865, in _run
    self._record_hit(hit)
  File "/usr/local/share/www/piwik/misc/log-analytics/import_logs.py", line 929, in _record_hit
    headers={'User-Agent' : hit.user_agent},
  File "/usr/local/share/www/piwik/misc/log-analytics/import_logs.py", line 699, in call
    return self._call_wrapper(self._call, expected_content, path, args, headers)
  File "/usr/local/share/www/piwik/misc/log-analytics/import_logs.py", line 691, in _call_wrapper
    message = e.reason
AttributeError: 'HTTPError' object has no attribute 'reason'

It's the very same log file, which tells it's not an invalid line causing that problem. So far I don't know why it returns an empty reply, nginx and php-fpm logs remain empty about that.

Changed 3 weeks ago by oliverhumpage

Just wanted to add that I get really bad speeds unless I ramp up the reporters way beyond core number: I may have said above, I get to about 40 (on a 4 core virtual machine) before noticing no further improvements. There is definitely a lag somewhere in the http requests or mysql connections on some setups. There is also the issue of the "tail end", noted above, where the number of recorders slowly drops off as IPs run out (I did try altering the script to give new IPs to the recorder with the lowest workload, but it didn't make much odds).

Having a PHP script that talked to the piwik system directly, instead of via http requests, would likely speed things up hugely for all users.

Changed 3 weeks ago by matt

From email report:

I try to use the module import_logs.py for my apache logs, I have a problem because I use HAProxy and the script does not seem to consider my rgex for my log formatfollowing:

  LogFormat "% v% {X-Forwarded-For} i% l% u% t \"% r \ "%> s% b \"% {Referer} i \"\"% {User-Agent} i \ ""vhost_combined


X.com 90.28.198.22 - - [11/Apr/2012:20:52:12 +0200] "GET /index.php HTTP/1.1" 200 - "http://www.X.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0) Gecko/20100101 Firefox/10.0"

I lunch the script :  
python /var/www/piwik/misc/log-analytics/import_logs.py --url=http://127.0.0.1/piwik /var/log/apache2/access_webmail.log --log-format-regex "%v %{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" --add-sites-new-hosts --recorders=1 

0 lines parsed, 0 lines recorded, 0 records/sec
Parsing log /var/log/apache2/access_webmail.log...

Is it the same problem as the IIS log, or maybe a different problem? or is the --log-format-regex wrong maybe?

in reply to: ↑ 121 Changed 3 weeks ago by matt

description modified (diff)

Replying to Cyril:

asterixcapri: what's your import speed? What's the limiting factor? Python or PHP? How large are your log files? The Python import script can easily max out 8 Piwik PHP processes on my machines, so I doubt the HTTP requests represent a large overhead. Besides, an easy way to reduce that overhead would be to aggregate hits in a single request, which is already planned.

I created a ticket for this specific performance improvement: #3134

Changed 11 days ago by oliver

Hi, I tried to import apache logs into an empty Piwik database. The log-file has a lot of different hosts, that should be added automatically.

./import_logs.py --login=user --password=pwd --add-sites-new-hosts --url=http://site/piwik ./apache.log

However, the scripts shows the following error and won't exit.

Purging Piwik archives for dates: 2012-05-10
Traceback (most recent call last):
  File "./import_logs.py", line 1221, in <module>
    main()
  File "./import_logs.py", line 1197, in main
    Recorder.invalidate_reports()
  File "./import_logs.py", line 949, in invalidate_reports
    idSites=','.join(stats.piwik_sites),
TypeError: sequence item 2: expected string or Unicode, int found

in reply to: ↑ 138 Changed 9 days ago by tiouk

Replying to matt:

Can you please confirm you tested your patch on the 3 IIS log formats?

I have tested the iis6_w3c_all on real logs. I think there maybe an issue with the IIS7/7.5, as the although they work, the logs I tested aginst appear to have extra options selected and I can't find a definitive default format on the MS site apart from for IIS6. I have some tweaked format regex, will post when tested.

Can anyone using IIS please post the first 5 lines from a few logs and state whether you have added any non default options to the logging.

IIS logs seperate the query string from the base URL, I haven't addressed that.

in reply to: ↑ 142 Changed 8 days ago by tiouk

Replying to Cyril:

tiouk: thanks. Unless matt insists on doing so, I'd like to commit your patch myself as I'd like to refactor it a bit. Can you provide some log lines for each format? I'll put them in the tests/logs directory which provides automatic testing for each log format.

Will see if I can do tomorrow. There's an updated file on the same URL as the old patch, it has a bit of work to skip lines in an IIS log with --check-iis-logs-format and displays the log options line in --debug along with updated regexs. Cheers Mike

Changed 7 days ago by tiouk

IIS logs the cs-uri-stem & cs-uri-query separately, do they need concatenating?

Replace \S+ after the <path> match with '(?P<querystr>\S+) ' in the regex's

line 1199 
            if config.options.check_iis_log_format:
                hit = Hit(
                    filename=filename,
                    lineno=lineno,
                    status=match.group('status'),
                    full_path=match.group('path') + config.options.query_string_delimiter + match.group('querystr'),
                    is_download=False,
                    is_robot=False,
                    is_error=False,
                    is_redirect=False,
                )
            else:
                hit = Hit(
                    filename=filename,
                    lineno=lineno,
                    status=match.group('status'),
                    full_path=match.group('path'),
                    is_download=False,
                    is_robot=False,
                    is_error=False,
                    is_redirect=False,
                )

follow-up: ↓ 158 Changed 7 days ago by aspectra

Hi, I got following error during importing log files (apache):

419375 lines parsed, 6906 lines recorded, 57 records/sec
419703 lines parsed, 6960 lines recorded, 54 records/sec
419896 lines parsed, 6976 lines recorded, 16 records/sec
419896 lines parsed, 6976 lines recorded, 0 records/sec
419896 lines parsed, 6976 lines recorded, 0 records/sec
419896 lines parsed, 6976 lines recorded, 0 records/sec
Fatal error: didn't receive the expected response. Response was <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
        <title>Piwik &rsaquo; Error</title>
        <meta http-equiv="Conte..
You can restart the import of "/opt/piwiktests/logfile.gz" from the point it failed by specifying --skip=296312 on the command line.

with debug:

2012-05-13 12:17:59,837: [DEBUG] Error when connecting to Piwik: <urlopen error didn't receive the expected response. Response was <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   "SAME URL as above">
<html>
<head>
        <title>Piwik &rsaquo; Error</title>
        <meta http-equiv="Conte.. >

The end of the log file shows:

5190432 lines parsed, 759119 lines recorded, 28 records/sec
5190432 lines parsed, 759147 lines recorded, 28 records/sec
Purging Piwik archives for dates: 2012-05-12
2012-05-13 13:49:58,052: [DEBUG] Error when connecting to Piwik: <urlopen error Piwik returned an invalid response:
<div style='word-wrap: break-word; border: 3px solid red; padding:4px; width:70%; background-color:#FFFF96;'>
            <strong>There is an error. Please report the message and full backtrace in the <a href='?module=Proxy&action=redirect&url=http://forum.piwik.org' target='_blank'>Piwik forums</a> (plea>

I'm running the script on a 12 Core (24 with HT) server with this command:

python import_logs.py --url=http://piwiktest.aspectra.com/piwiklog/ --idsite=6 --recorders=12 --output=logtest_20120513.log --skip=358639 -d -d /opt/piwiktests/logfile.gz &

Sometimes the script recovers from the errors and continuous recording, sometimes it stops working and shows the --skip= option. As far as I can see it seems that the script stops working if the error occurs 4 times in a row. Is this some kind of a timeout and can it be set?

Best regards, André

follow-up: ↓ 151 Changed 6 days ago by tiouk

Diff for the auto detecting of IIS logs based on log header line 4, this should be able to decode any IIS log file whatever the options selected. One previso, it does need both the date and time options which can be de-selected. However without these options it is not possible to generate stats unless you don't mind all site visits to occur at the same time!

http://mike.org.uk/import_logs_py_diff_2.txt

Ready patched file based on version 6170 for people to test on different IIS formats, let me know how you get on. http://mike.org.uk/import_logs_py.txt (rename to .py)

Changed 6 days ago by tiouk

Patch 6213 applied. Error generated by +íå+íàøëîñü+ôîðìû+äëÿ+îòïðàâêè; in iis log.

  File "/usr/lib64/python2.6/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xed in position 154: invalid continuation byte

2012-05-03 04:29:30 W3SVC230092276 SERVER1 172.16.65.25 GET /Discussions/tabid/56/forumid/1/postid/36/scope/posts/language/en-GB/Default.aspx+Result:+íå+íàøëîñü+ôîðìû+äëÿ+îòïðàâêè; - 80 - 94.153.71.194 HTTP/1.0 Mozilla/0.6+Beta+(Windows) - http://xxx-xxxxxx.net/Discussions/tabid/56/forumid/1/postid/36/scope/posts/language/en-GB/Default.aspx+Result:+%ED%E5+%ED%E0%F8%EB%EE%F1%FC+%F4%EE%F0%EC%FB+%E4%EB%FF+%EE%F2%EF%F0%E0%E2%EA%E8; xxx-xxxxxx.net 404 0 2 1814 715 62

Changed 5 days ago by tiouk

Version 6260 with IIS patch.

Previously decoded line generates error when posting to Piwik.

2012-05-16 14:15:08,670: [DEBUG] Error when connecting to Piwik: 'ascii' codec can't encode characters in position 126-128: ordinal not in range(128)

Raw log line:
2010-09-29 07:00:25 W3SVC3 172.16.65.22 GET /xxxxxxxxx/xxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxxxx/tabid/129/language/en-US/Default.aspx+Result:+ýòî+íå+ôîðóì+/+ãîñòåâàÿ+êíèãà+(ëèáî+îòñóòñòâóåò+ïîäêëþ÷åíèå+ê+èíòåðíåòó) 

args from def _call_wrapper(self, func, expected_response, *args, **kwargs):
('/piwik.php', {'cdt': '2010-09-29 07:00:25', '_cvar': u'{"1":["Not-Bot","Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+Deepnet+Explorer+1.5.0;+.NET+CLR+1.0.3705)"]}', 'apiv': '1', 'cip': u'173.75.247.80', 'urlref': '', 'token_auth': u'b13583ea29ddb846f48d1ea1721a8eac', 'idsite': '3', 'url': u'http://xxxxxxxxxxx.xxx.xx/xxxxxxxxx/xxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxxxx/tabid/129/language/en-US/Default.aspx+Result:+\xfd\xf2\xee+\xed\xe5+\xf4\xee\xf0\xf3\xec+/+\xe3\xee\xf1\xf2\xe5\xe2\xe0\xff+\xea\xed\xe8\xe3\xe0+(\xeb\xe8\xe1\xee+\xee\xf2\xf1\xf3\xf2\xf1\xf2\xe2\xf3\xe5\xf2+\xef\xee\xe4\xea\xeb\xfe\xf7\xe5\xed\xe8\xe5+\xea+\xe8\xed\xf2\xe5\xf0\xed\xe5\xf2\xf3)?-', 'rec': '1', 'dp': '0'}, {'Content-type': 'application/x-www-form-urlencoded', 'User-Agent': u'Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+Deepnet+Explorer+1.5.0;+.NET+CLR+1.0.3705)'})

Changed 3 days ago by mlrxnet

Hi, we have some issues with the latest build of the import on our IIS 7.5 logs. The following messages are shown and the import stopped:

...
2012-05-17 22:07:07,918: [DEBUG] Error when connecting to Piwik: 'ascii' codec c
an't encode character u'\xe4' in position 32: ordinal not in range(128)
Fatal error: 'ascii' codec can't encode character u'\xe4' in position 32: ordina
l not in range(128)
You can restart the import of "u_ex120513.log" from the point it failed by specifying --skip=214 on the command l
ine.

Attached are the following Lines 213-215 with the field specification for an review:

Fields:

#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken

Line 213:

2012-05-13 01:26:04 W3SVC7 XNW-WEB01 46.4.192.150 GET / - 443 - 213.133.113.84 HTTP/1.1 Hetzner+System+Monitoring - www.wolke.com 302 0 0 677 91 31

Line 214:

2012-05-13 01:26:36 W3SVC7 XNW-WEB01 46.4.192.150 GET /imagefilm/deutsch.swf - 80 - 216.246.45.86 HTTP/1.0 gsa-crawler+(Enterprise;+S5-HKQ2CJT3FSJJT;+info@tobaccopeople.com) - www.m600.com 200 0 0 34601 236 468

Line 215:

2012-05-13 01:27:03 W3SVC7 XNW-WEB01 46.4.192.150 GET /de/kontakt/anfahrt - 80 - 95.211.139.1 HTTP/1.0 Mozilla/5.0+(compatible;+AcoonBot/4.10.8;++http://www.acoon.de/robot.asp) - www.wolke.com 200 0 0 21376 193 124

Command Line:

import_logs.py  --debug --url=http://statistik.mlr-xnet.de u_ex120513.log --idsite=8 --recorders=4 --enable-http-errors --enable-http-redirects --enable-static --enable-bots --enable-reverse-dns

Changed 3 days ago by matt

Thanks for all your work and feedack.

AS per the last comments it seems IIS log parsing is now fully working which was the last critical open bug.

To all users listening here, is your log format now recognized as expected?

AS per comments in this thread the last remaining changes to make are: Todo per comments

Change constants

PIWIK_MAX_ATTEMPTS = 9 PIWIK_DELAY_AFTER_FAILURE = 5 The errors are still occurring but the script is able to reconnect and does not exit.

recognize static files not ending with the filetype but can have cache buster parameter strings x.js?ts=319743413

I checked the logfile and looked in the importer code and found out, that many static-files of - at least - Typo3-Websites are not recognized, as long as they are suffixed with ?timestamp by Typo3 and the importer-regex just checks the end of the filesname (e.g. typo3temp/javascript_0b12553063.js?1283017207).

These sound like easy changes. Otherwise looks like the script is ready for prime time?

Note: See TracTickets for help on using tickets.

Download in other formats:

Context Navigation

Ticket #703 (reopened New feature)

Piwik an alternative to AWStats and Urchin, build server log import script

Description (last modified by matt) (diff)

Change History

Changed 3 years ago by vipsoft

Changed 3 years ago by matt

Changed 14 months ago by matt

Changed 13 months ago by matt

Changed 8 months ago by matt

Changed 3 months ago by matt

Changed 3 months ago by matt

Changed 3 months ago by matt

Changed 2 months ago by matt

follow-up: ↓ 11 Changed 2 months ago by matt

in reply to: ↑ 10 Changed 2 months ago by Cyril

Changed 2 months ago by matt

Changed 2 months ago by vipsoft

Changed 2 months ago by oliverhumpage

Changed 2 months ago by Cyril

Changed 2 months ago by MichaelP08

follow-up: ↓ 18 Changed 2 months ago by Cyril

in reply to: ↑ 17 Changed 2 months ago by MichaelP08

follow-up: ↓ 20 Changed 2 months ago by Cyril

in reply to: ↑ 19 Changed 2 months ago by MichaelP08

Changed 2 months ago by oliverhumpage

Changed 2 months ago by matt

Changed 2 months ago by Cyril

Changed 2 months ago by hhamano

follow-up: ↓ 29 Changed 2 months ago by matt

Changed 2 months ago by oliverhumpage

Changed 2 months ago by oliverhumpage

Changed 2 months ago by oliverhumpage

in reply to: ↑ 25 ; follow-up: ↓ 32 Changed 2 months ago by hhamano

follow-up: ↓ 36 Changed 2 months ago by Cyril

Changed 2 months ago by matt

in reply to: ↑ 29 ; follow-up: ↓ 34 Changed 2 months ago by matt

Changed 2 months ago by matt

in reply to: ↑ 32 Changed 2 months ago by hhamano

Changed 2 months ago by matt

in reply to: ↑ 30 Changed 2 months ago by oliverhumpage

Changed 2 months ago by matt

Changed 2 months ago by matt

follow-up: ↓ 40 Changed 2 months ago by Cyril

in reply to: ↑ 39 Changed 2 months ago by oliverhumpage

Changed 2 months ago by Cyril

Changed 2 months ago by oliverhumpage

Changed 2 months ago by oliverhumpage

Changed 2 months ago by matt

Changed 2 months ago by Cyril

Changed 2 months ago by Cyril

Changed 2 months ago by Cyril

Changed 2 months ago by matt

Changed 2 months ago by matt

follow-up: ↓ 52 Changed 2 months ago by matt

Changed 2 months ago by oliverhumpage

in reply to: ↑ 50 ; follow-up: ↓ 54 Changed 2 months ago by Cyril

Changed 2 months ago by Cyril

in reply to: ↑ 52 ; follow-up: ↓ 57 Changed 2 months ago by matt

follow-up: ↓ 58 Changed 2 months ago by matt

Changed 2 months ago by matt

in reply to: ↑ 54 Changed 2 months ago by Cyril

in reply to: ↑ 55 Changed 2 months ago by Cyril

Changed 8 weeks ago by guardian

follow-up: ↓ 61 Changed 8 weeks ago by matt

in reply to: ↑ 60 ; follow-up: ↓ 62 Changed 8 weeks ago by guardian

in reply to: ↑ 61 Changed 8 weeks ago by guardian

Changed 8 weeks ago by guardian

Changed 8 weeks ago by oliverhumpage

Changed 8 weeks ago by oliverhumpage

Changed 8 weeks ago by oliverhumpage

Changed 7 weeks ago by Cyril

Changed 7 weeks ago by Cyril

Changed 7 weeks ago by Cyril

Changed 7 weeks ago by Cyril

Changed 7 weeks ago by Cyril

Changed 7 weeks ago by Cyril

Changed 7 weeks ago by Cyril

Changed 7 weeks ago by Cyril

Changed 7 weeks ago by matt