Parsing Nginx Access Logs with GoAccess
Learn how to parse the nginx-access.log file with GoAccess to gather information on your visitors and referral traffic.
Contributors: Albert Causing, Sarah German.
Discuss in our Forum Discuss in SlackPantheon runs nginx web servers for optimal performance. Your site's nginx access logs record web server events and activities that can help you identify potential issues and gather information about users.
Note
Requests served by the Pantheon Global CDN will not hit the nginx web server and will not be logged in nginx-access.log
.
GoAccess is a free, open source utility that creates reports by parsing nginx-access.log
files. Use it to quickly identify the most used browsers and operating systems, visitor IPs, or most frequent 404s — all from the command line.
Before You Begin
Be sure that you have:
- Terminus
- GoAccess
- Mac OS X: Install via Homebrew (
brew install goaccess
) - Windows: Use Windows Subsystem for Linux
- Mac OS X: Install via Homebrew (
This guide is written for the latest stable release of GoAccess as of this writing, which is version 1.5 (release notes).
Edit GoAccess Configuration
To parse your nginx-access.log
files with GoAccess, you'll need to configure GoAccess to read Pantheon's log formats.
- Use the command
goaccess --dcf
to check where your configuration file is located. - Copy this configuration file to your home directory. For example, if you installed GoAccess with Homebrew, your command might look like this:
cp /opt/homebrew/Cellar/goaccess/1.5.4/etc/goaccess/goaccess.conf ~/.goaccessrc
- Add the following lines to the configuration file:
time-format %H:%M:%S
date-format %d/%b/%Y
log-format %^ - %^ [%d:%t %^] "%r" %s %b "%R" "%u" %T "%h,%^"
Note that when providing configuration from your home directory, the file needs to be named .goaccessrc
. If you're storing this file elsewhere, it should be named goaccess.conf
. Read more about the configuration file.
Create a report
Download your nginx log files from Pantheon via SFTP.
From the directory containing your
nginx-access.log
file, run GoAccess:goaccess nginx-access.log
You can use the arrow keys on your keyboard to scroll down to view more of the report, or hit
q
to exit.Alternatively, you can generate an HTML report:
goaccess nginx-access.log > report.html
View the report in your browser by opening
report.html
. For MacOS:open report.html
For Linux:
xdg-open report.html
Troubleshooting "goaccess.conf Not Found"
In certain MacOS Homebrew installations of GoAccess versions 1.3 and earlier, goaccess.conf
is not found by the binary.
To resolve, update your local packages, or to update the GoAccess package specifically:
brew upgrade goaccess
Automate GoAccess Reports
Copy the general log retrieval script from Automate Downloading Logs, and use this to download logs from all application containers on the desired environment.
Add the following to either
collect-logs.sh
or a separate file:# Unpack archived log files (optional). gunzip */nginx-access.log-* # Create a GoAccess report and open it in a browser. goaccess */nginx-access.log* > goaccess.html && open goaccess.html # Or xdg-open for Linux
Alternatives to GoAccess
You can navigate the nginx-access.log
file using the CLI, without GoAccess. The following commands are a great starting point for navigatiing the nginx-access.log
file:
Locate the most frequent client IP addresses
cat nginx-access.log | awk -F '\"' '{ print $8 }' | awk -F ',' '{print $1}' | sort | uniq -c | sort -frn | head -n 25
Locate the most frequent URLs
cat nginx-access.log | awk -F '\"' '{print $2}' | sort | uniq -c | sort -nr | head
Identify the most frequent User Agents
cat nginx-access.log | awk -F '\"' '{print $6}' | sort | uniq -c | sort -nr | head