Logs are the valuable resource for debugging. When we have a website, we also often look into Nginx logs to see what happened with our website. But they are often cluttered by the visit of search bots, which make us difficult to find the noteworthy lines. So how to tell Nginx to log the search bot activities to another file, to make our access log cleaner?
To do that, first, create an file to help Nginx distinguish who is search bot. Create a file bot_definition.conf in /etc/nginx/conf.d folder, with this content:
Now, update your virtual host config. For example, with my website, the file is in /etc/nginx/sites-available/quanweb.conf. Add the
include to the top of file then find the
access_log directive and change to this:
include conf.d/bot_definition.conf; # Other lines access_log /var/log/nginx/quanweb/access.log combined if=$is_not_bot; access_log /var/log/nginx/quanweb/bot_access.log combined if=$is_bot;
Change the log file path to your actual setup. What is added is the
combined if=.... Because Nginx does not support the
if=!$some_var syntax, we have to define two variables,
After saving the file, remember to test if your new config breaks Nginx:
If it says Ok, you can then apply the config:
Now, check you main access log file. It no longer contains the visit of search bot. Note the the list of patterns to recognize bot above is not complete. You can extend it if you find something new.