GO IT WORLD | IT TECH | IT NEWS

Foreword

In my previous article “Using Nginx to enhance the speed of site visits,” introduced Nginx the HTTP server and how to accelerate through its Web site access.  In the actual Web site operators, we often need to understand that the site visit, for example, the number of IP day visit, PV is the number, which URL to visit the largest, most users browser which is what way to know this site and the number of users access to an error and so on, through mastering the information to improve the user experience, thereby improving the quality of the site.  Generally we can visit free of charge through a number of statistics to sites such as Google Analytics or the information.   But the inadequacies of such sites is only an analysis of the page does not include static documents; There may be a lot of regulators do not want to use such tools to expose their own data, all sorts of these factors make regulators want to analyze access logs.  And awstats is enough to meet all these requirements.

Awstats in SourceForge to develop quickly a Perl of WEB-based log analysis tool, a full analysis of the log so that Awstats shows you the following information:

1. Visits, the number of unique visitors,
2. Access time and the last visit,
3. User authentication, the recent certification visit
4. Weekly peak time (the number of pages, click-through rate per hour and week kilobytes),
5. Name / country hosts visitors (pages, click-through rate, byte, 269 domains / countries detected, geoip detection),
6. Host list of recently visited and did not resolve the IP address list
7. Most have read the entry and exit pages,
8. , File types,
9. Site compression tables (mod_gzip or mod_deflate),
10. Operating system (one for each operating system, the number of pages, click-through rate, byte, 35 OS detected),
11. Using a browser,
12. Robot visits (319 robots detected),
13. Worm attacks (5 worm family),
14. Search engines, use keyword search to find your address,
15. HTTP protocol error (the most recent inspection did not find the page),
16. Other reports based on the personalized URL, link parameters, involving the field of integrated marketing purpose.
17. Your site by adding “favorite bookmarks.” Views.
18. Screen size (in the index page of the need to add some HTML tags).
19. The proportion of browser support: Java, Flash, RealG2 reader, Quicktime reader, WMA reader, PDF reader.
20. The ratio of load-balancing server cluster report.

Awstats operating environment requires PERL support awstats documents from view, its support for Apache HTTP Server is a very perfect, and when we Nginx replaced after the Web server to run awstats become very troublesome.  First Nginx support Perl itself is relatively weak, and even the official does not recommend the use of; another format in the log there is a need to revised in order to run.

This paper mainly introduces awstats by allowing the outcome of the survey on the log to generate a static page, and then through Nginx statistical output in order to achieve the effect of Nginx access logs, including how to make automatic cutting Nginx log files.

Nginx configure automatic log cutting

With the Apache HTTP Server (hereinafter referred to Apache) The difference is, Apache will log the output of the way through the pipeline re-orientation, and so to automatically log cutting. In the current version of Nginx could not, like Apache, through parameters such as% YY in batches by date to create the log, but nginx process through to send a specific signal, can regenerate nginx log files.  We can implement a Shell script to switch the log, rename or transfer, the specific script is as follows:

# mv  /opt/nginx/logs/access.log /opt/nginx/logs/access_`date +%Y%m%d`.log
# killall –s USR1 nginx

The above script will be saved as a file name content logcron.sh depositors to self-directory, for example, / opt / nginx / sbin / logcron.sh

Crontab allow the use of script in a day 23:59 self-executing, you can do so by the day to create the log.

To install and configure Awstats

Before installing the need to confirm your server Perl environment already in place.

perl –version See the current environment is a Perl version of the command perl-version

We also need to log format Nginx be small changes will not be able to otherwise awstats statistics.

Examples are as follows (bold part):

# vi /opt/nginx/conf/nginx.conf

server {
listen       80;
server_name  localhost;

location ~ ^/web/ {
root   /data/web;
index  index.html;
error_log off;
charset utf-8;
}

log_format  new_log
'$remote_addr - $remote_user [$time_local] $request '
        '"$status" $body_bytes_sent "$http_referer" '
        '"$http_user_agent" "$http_x_forwarded_for"';
        access_log  logs/access.log new_log;
}

Download the latest version of awstats package, download the address, see the end, the article references. To download the tar package extract to any directory, for example: /usr/local/awstats. And then the implementation of tools directory awstats_configure.pl Configuration Wizard, create a new statistics.

-----> Check for web server install

Enter full config file path of your Web server.
Example: /etc/httpd/httpd.conf
Example: /usr/local/apache2/conf/httpd.conf
Example: c:\Program files\apache group\apache\conf\httpd.conf
Config file path ('none' to skip web server setup):
#> none

Enter

Your web server config file(s) could not be found.
You will need to setup your web server manually to declare AWStats
script as a CGI, if you want to build reports dynamically.
See AWStats setup documentation (file docs/index.html)

-----> Update model config file '/usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf'
  File awstats.model.conf updated.

-----> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ?
#> y

Enter

-----> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
#> www.moabc.net
www.moabc.net

Enter

-----> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):
#>

Enter directly using the default, then there will be the following tips

----> Add update process inside a scheduler
Sorry, configure.pl does not support automatic add to cron yet.
You can do it manually by adding the following command to your cron:
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.moabc.net
Or if you have several config files and prefer having only one command:
/usr/local/awstats/tools/awstats_updateall.pl now

A SIMPLE config file has been created: /etc/awstats/awstats.www.moabc.net.conf
You should have a look inside to check and change manually main parameters.
You can then manually update your statistics for 'www.moabc.net' with command:
> perl awstats.pl -update -config=www.moabc.net
You can also build static report pages for 'www.moabc.net' with command:
> perl awstats.pl -output=pagetype -config=www.moabc.net

Press ENTER to finish...

Enter the completion of the wizard, then modify configuration www.moabc.net statistics

#vi /etc/awstats/awstats.www.moabc.net.conf
Statistics of the log file to find the path

LogFile=”/var/log/httpd/mylog.log”
Changed
LogFile=”/opt/nginx/logs/access_%YYYY-0%MM-0%DD-0.log

Nginx above the corresponding log cutting procedures generated directory storage structure, attention should be paid to the date Awstats format Nginx with different wording. We are now the order of the implementation of statistics are:

Nginx generated log -> Log Cutting -> Nginx continue to produce the log -> Save cutting logs -> by Awstats Statistics -> generate results

In this article, the statistics Awstats logs, have been cut down that part. Can also reverse the order of a further cut before the end statistics.  But this is easier statistical omission.  Configuration modifications are completed, save quit.  Then we can begin to try manually.

1. First implementation of the log cutting logcron.sh script to log Nginx cut.
2. Awstats log and then update the implementation of the beginning of statistical analysis.

# /opt/nginx/sbin/logcron.sh
# /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.moabc.net

Create/Update database for config "/etc/awstats/awstats.www.moabc.net.conf"
        by AWStats version 6.7 (build 1.892)
From data in log file "/opt/nginx/logs/access_20080804.log"...
Phase 1 : First bypass old records, searching new record...
Direct access after last parsed record (after line 450421)
Jumped lines in file: 450421
 Found 450421 already parsed records.
Parsed lines in file: 120
 Found 0 dropped records,
 Found 0 corrupted records,
 Found 0 old records,
 Found 120 new qualified records.

See above shows that the log cutting and running Awstats have been correct. Statistical analysis is completed, the results also Awstats database.  In Apache, you can directly open a Perl program page view statistics.   However, the beginning of this article has already been mentioned, Nginx support of Perl is not good, so we need a change in methods, the use of the tool will awstats the outcome of the survey to generate static documents, concrete steps are as follows:

• First of all, in the webroot directory, create a folder.
• Then let Awstats to generate static pages to the directory

# mkdir  /data/webroot/awstats

# /usr/local/awstats/tools/awstats_buildstaticpages.pl -update  \
-config=www.moabc.net -lang=en -dir=/data/admin_web/awstats  \
-awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats.pl

The specific meaning of the order is as follows:

• /usr/local/awstats/tools/awstats_buildstaticpages.pl  Awstats static page generation tool
• -update-config = www.moabc.net update the configuration item
• -lang = en language is English
• -dir=/data/admin_web/awstats  statistical results output directory
• -awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats.pl  Awstats log update path.

Next, just nginx.conf in the directory can be configured up.  Examples are as follows: (bold part):

server {
listen       80;
server_name  localhost;

location ~ ^/web/ {
root   /data/web;
index  index.html;
error_log off;
charset utf-8;
}

location ~ ^/awstats/ {
        root   /data/webroot/awstats;
        index  index.html;
        access_log off;
        error_log off;
        charset utf-8;
}

location ~ ^/icon/ {
        root   /usr/local/awstats/wwwroot;
        index  index.html;
        access_log off;
        error_log off;
        charset utf-8;
        }
}

Use your browser to view the detailed results of the statistical http://youhostname/awstats/awstats.www.moabc.net.html

At this point, the use of awstats have been able to fully support the Nginx log statistics.

Configured to run automatically Awstats

In order for the entire statistical process log auto-complete, we need to set up crontab scheduled tasks, so that Nginx Log Awstats automatic cutting and running, from time to time to generate results page.

#vi /etc/crontab

11 59 * * * /opt/nginx/sbin/logcron.sh
00 1 * * * /usr/local/awstats/tools/awstats_buildstaticpages.pl  \
-update -config=www.moabc.net -lang=cn -dir=/data/admin_web/awstats \
-awstatsprog=/usr/local/awstats/wwwroot/cgi-bin/awstats.pl  

#00:01  Awstats analysis log

#crontab /etc/crontab

The protection of the log results page

Regulators are generally reluctant to make people aware of their station not the real traffic, so results should Awstats password-protected pages. Nginx using Apache with the same password in encrypted format, where the need to bring their own tools apache used htpasswd.

If you default on this machine equipped with Apache, which you only in its directory run

For example:

#/usr/local/apache2/bin/htpasswd -c admin.pass admin

New password:
Re-type new password:
Adding password for user admin

server {
 	listen       80;
 	server_name  localhost;
 	location ~ ^/web/ {
 	root   /data/web;
 	index  index.html;
 	error_log off;
 	charset utf-8;
 	} 

 	location ~ ^/awstats/ {
        root   /data/admin_web;
        index  index.html;
        access_log off;
        error_log off;
        charset utf-8;
        auth_basic     "admin";
     /opt/ngx/conf/admin.pass;
        }

        location ~ ^/icon/ {
        root   /usr/local/awstats/wwwroot;
        index  index.html;
        access_log off;
        error_log off;
        charset utf-8;
        }
}

Aggregate

Although with the Apache HTTP Server comparison, Nginx function is relatively weak, but we can still use some techniques to circumvent these weaknesses, Nginx designers certainly give full consideration to this issue.  Now a growing number of third-party development in a gradual expansion module Nginx function.  But from the perspective of the application itself, Nginx more inclined to superior performance, rather than function, which in a number of additional functions, we can not too high on the requirements.

Reference resources

• Nginx English site: http://www.nginx.net
• Awstats Web site: http://awstats.sourceforge.net