RE: Excessive logging by the webserver

[Mario Goulart <mario@proxy.furg.br>]

Hi Charles

On Tue, 20 Mar 2001, Charles Hall wrote:

> Mario, can you post a few dozen sample log entries? Off hand I can't
> imagine why webglimpse should be generating anything special in the way
> of log activity.

I'm pasting some lines of the access_log file from the only machine I have
access. Unfortunately there are just a few HTML files.

Note: Webglimpse is scanning the same machine where it is installed, but
through the http server, not a "raw" directory scan.

< From access_log >

200.19.254.126 - - [20/Mar/2001:11:27:40 -0300] "GET /proxy/ie4/ie4.html
HTTP/1.0" 200 1216
200.19.254.126 - - [20/Mar/2001:11:27:40 -0300] "GET
/proxy/netscape/netscape.html HTTP/1.0" 200 2102
200.19.254.126 - - [20/Mar/2001:11:27:54 -0300] "GET /robots.txt
HTTP/1.0" 404 274
200.19.254.126 - - [20/Mar/2001:11:27:54 -0300] "GET / HTTP/1.0" 200 2566
200.19.254.126 - - [20/Mar/2001:11:27:54 -0300] "GET /proxy/ie4/ie4.html
HTTP/1.0" 200 1216
200.19.254.126 - - [20/Mar/2001:11:27:54 -0300] "GET
/proxy/netscape/netscape.html HTTP/1.0" 200 2102
200.19.254.126 - - [20/Mar/2001:11:28:53 -0300] "GET /robots.txt
HTTP/1.0" 404 274
200.19.254.126 - - [20/Mar/2001:11:28:53 -0300] "GET / HTTP/1.0" 200 2566
200.19.254.126 - - [20/Mar/2001:11:28:53 -0300] "GET /proxy/ie4/ie4.html
HTTP/1.0" 200 1216
200.19.254.126 - - [20/Mar/2001:11:28:53 -0300] "GET
/proxy/netscape/netscape.html HTTP/1.0" 200 2102

< end >

Entries like that full up the log files in the scanned machines. They are
normal access logs, I know, but I was wondering if there is a way to avoid
logging accesses from the machine running webglimpse. That's because the
other servers have a lot of HTML files, so the access generate a lot of
information in the log files.

That's the screen output when I run wgreindex: 

proxy:/usr/local/apache/logs# /usr/local/apache/htdocs/wgreindex 
No search boxes used
Getting local links by hops, 2 hops...
Neighborhood will be 1 hops.
Got url http://proxy.furg.br into file
/usr/local/apache/htdocs/.remote/1.br.
Traversing 2 hops...
Url http://proxy.furg.br/proxy/ie4/ie4.html is remote...
Getting remote url: http://proxy.furg.br/proxy/ie4/ie4.html
Got url http://proxy.furg.br/proxy/ie4/ie4.html into file
/usr/local/apache/htdocs/.remote/2.html.
Url http://proxy.furg.br/proxy/netscape/netscape.html is remote...
Getting remote url: http://proxy.furg.br/proxy/netscape/netscape.html
Got url http://proxy.furg.br/proxy/netscape/netscape.html into file
/usr/local/apache/htdocs/.remote/3.html.
No more links to traverse.
Got url http://www.ecomp.furg.br into file
/usr/local/apache/htdocs/.remote/4.br.
Traversing 2 hops...

Regards
Mario