Customised Log Analysis for bitz.co.nz

Writen by Paul Bourke
Version 1, August 1995. Version 2, March 1997.


This software was primarily written to analayse the WWW logs from the Apache WWW server (CERN format) generated in an environment with many virtual servers running from the same machine. In addition the statistics have information separated in ways of interest to New Zealand WWW sites.

The search is done based on the directory containing the pages. Multiple log files from the virtual domains are combined in the search for matches to the log string.

The program in version 2 is disk based and can thus analyse sites with millions of hits.

The program can be executed on the UNIX command line or through a WWW page. The later is illustrated below.


WWW log analysis

Prepared for kiwipages/

Covers the period from 1-April-1997 to 20-April-1997


Summary statistics

                                 Accesses     KBytes
                         Total       5047    22267.1
                 International        749     3295.2 Peak (8am - 9pm)
                                      530     2158.7 Offpeak
                   New Zealand       3768    16813.2

                 Rate per hour        9.8       43.3 (covers 514.80 hours)
                  Rate per day      235.3     1038.1 (covers 21.45 days)

Address summary

554 unique addresses

Address                                            Access   KBytes
celebrian.otago.ac.nz                                 825   3013.0
cantso.canterbury.ac.nz                               419   1568.2
wyvern.xtra.co.nz                                     158    728.0
proxy.xtra.co.nz                                      134    844.2
kauri.lincoln.ac.nz                                   105    303.8
totara.its.vuw.ac.nz                                  102    603.8
pegasus.mhri.edu.au                                    70    421.4
wheel.wave.co.nz                                       67    270.4
zombie.xtra.co.nz                                      65    548.4
clw232056.gte.net                                      61    308.9
ppp0-13.thenet.co.nz                                   59     35.0
mhri-1.mhri.edu.au                                     59    175.2
rdynam-21.tpnet.co.nz                                  56    111.3
proxy8.es.co.nz                                        53    210.5
ad2.twp.ac.nz                                          52    111.4
204.131.47.100                                         51    209.4
ppp37.tipnet.co.nz                                     51    251.3
wanggate.wang.co.nz                                    45    165.4
mail.unitec.ac.nz                                      42    142.0
168-225-244.ipt.aol.com                                42    179.6
galore-bbn.infoseek.com                                40     85.4
Cut

Domain summary

33 unique country domains

  Domain     Access       KBytes     Description 
      nz       3657      16171.5     New Zealand
     com        358       1504.1     US Commercial
 n.n.n.n        306       1187.3     Numeric  
     net        235       1110.9     Network  
      au        212        983.3     Australia
    arpa        110        641.7     Old style Arpanet
      uk         31        160.1     United Kingdom
     edu         28         99.5     US Educational
      de         17         89.0     Germany  
      ca         15         52.8     Canada   
      dk         11         36.5     Denmark  
      jp         10         32.7     Japan    
     org         10         33.2     Non-Profit
      se          7         17.1     Sweden   
      ch          5         23.3     Switzerland
      gr          4          4.2     Greece   
      sg          4         10.1     Singapore
      no          4          4.2     Norway   
      be          3          9.3     Belgium  
      il          3         21.3     Israel   
      bg          2          8.6     Bulgaria 
      fr          2          5.4     France   
     gov          2         11.7     US Government
      br          2          9.5     Brazil   
      nl          1          4.3     Netherlands
      it          1          2.8     Italy    
      es          1          2.8     Spain    
      ru          1          2.8     Russian Federation
      ae          1          4.3     United Arab Emirates
      my          1          2.8     Malaysia 
      at          1         12.3     Austria  
      kw          1          4.6     Kuwait   
      hk          1          2.8     Hong Kong

File summary

152 files

Filename                                                     Access   KBytes
/kiwipages/index.html                                           623   2654.4
/kiwipages/listem.cgi                                           341   1274.4
/kiwipages/frost.html                                           131   1609.9
/kiwipages/kimball.gif                                          114     48.0
/kiwipages/josh.html                                            100    350.8
/kiwipages/hellchick.html                                        92    180.4
/kiwipages/azariah.html                                          90    411.9
/kiwipages/line.gif                                              88     29.0
/kiwipages/bkgrnd.gif                                            88    109.5
/kiwipages/josh4.jpg                                             85     23.0
/kiwipages/rule18.gif                                            83    267.1
/kiwipages/hell2.gif                                             79    616.9
/kiwipages/ang_bec.gif                                           78   1969.0
/kiwipages/bullet5.gif                                           78     60.9
/kiwipages/cflylogo.gif                                          74    113.1
/kiwipages/helen.html                                            61     35.1
/kiwipages/coolness.html                                         60    281.1
/kiwipages/robots.txt                                            59      5.3
/kiwipages/cactus.html                                           58    164.0
/kiwipages/jane.html                                             57     99.3
/kiwipages/jester.html                                           56    115.2
/kiwipages/ahac.html                                             54     86.1
/kiwipages/lambda.html                                           45    304.6
/kiwipages/ahac1.gif                                             44    293.9
/kiwipages/cis.html                                              44     69.6
/kiwipages/ahac4.gif                                             43     57.7
/kiwipages/ahac2.gif                                             43     11.8
/kiwipages/jestback.gif                                          43     68.0
/kiwipages/ahac3.gif                                             43     14.5
/kiwipages/jestcnst.gif                                          43     11.7
/kiwipages/bazza.html                                            42     60.0
/kiwipages/jestbutt.gif                                          42     12.7
/kiwipages/cts.html                                              41     53.2
/kiwipages/kim.html                                              41    132.6
/kiwipages/jason.html                                            40    129.0
/kiwipages/cts1.gif                                              40    468.0
/kiwipages/josh2.gif                                             39     22.5
/kiwipages/ahac6.gif                                             39    103.3
/kiwipages/cts2.gif                                              39      9.5
/kiwipages/spca.html                                             38     32.7
/kiwipages/josh3.gif                                             38     26.2
/kiwipages/shaun_malone.html                                     38    380.8
Cut

Date summary

Date            Access   KBytes
 1-Apr-1997        281   1206.3
 2-Apr-1997        297   1202.6
 3-Apr-1997        166   1102.3
 4-Apr-1997        133    931.6
 5-Apr-1997        205    924.6
 6-Apr-1997        106    617.0
 7-Apr-1997        200    872.3
 8-Apr-1997        130    448.0
 9-Apr-1997        146    751.1
10-Apr-1997        226   1032.7
11-Apr-1997        241   1202.8
12-Apr-1997        327   1293.0
13-Apr-1997        232    512.5
14-Apr-1997        452   1933.9
15-Apr-1997        244    990.7
16-Apr-1997        359   1376.8
17-Apr-1997        280   1038.5
18-Apr-1997        217   1042.1
19-Apr-1997        183    609.4
20-Apr-1997        171    964.9
21-Apr-1997        259   1443.6
22-Apr-1997        192    770.3

Hour summary

Hour            Access   KBytes
Hour            Access   KBytes
Midnight           144    574.5
 1 am               73    361.5
 2 am               59    200.9
 3 am              131    533.2
 4 am               51    165.1
 5 am               41     95.9
 6 am               97    456.0
 7 am               82    591.7
 8 am               89    429.4
 9 am              203    927.8
10 am              229   1123.7
11 am              326   1381.0
Noon               321   1403.5
 1 pm              316   1642.4
 2 pm              391   1628.0
 3 pm              215    969.4
 4 pm              252   1090.4
 5 pm              294   1700.9
 6 pm              234    837.6
 7 pm              251   1003.3
 8 pm              471   2032.4
 9 pm              287   1041.8
10 pm              273   1199.1
11 pm              217    877.5