---
title: How to split Nginx logs for bot visits
date: 2023-07-26 07:30:56.181229 UTC
---
Logs are the valuable resource for debugging. When we have a website, we also often look into Nginx logs to see what happened with our website. But they are often cluttered by the visit of search bots, which make us difficult to find the noteworthy lines. So how to tell Nginx to log the search bot activities to another file, to make our access log cleaner?

To do that, first, create an file to help Nginx distinguish who is search bot. Create a file _bot_definition.conf_ in _/etc/nginx/conf.d_ folder, with this content:

```nginx
map $http_user_agent $is_bot {
        ~Pingdom 1;
        ~Appenlight 1;

        ~Googlebot 1;
        ~Baiduspider 1;
        ~facebookexternalhit 1;
        ~MJ12bot 1;
        ~YandexBot 1;
        ~AhrefsBot 1;
        ~coccoc 1;

        default 0;
}

map $is_bot $is_not_bot {
        0 1;
        1 0;
}
```

Now, update your virtual host config. For example, with my website, the file is in _/etc/nginx/sites-available/quanweb.conf_. Add the `include` to the top of file then find the `access_log` directive and change to this:

```nginx
include conf.d/bot_definition.conf;

# Other lines

access_log /var/log/nginx/quanweb/access.log combined if=$is_not_bot;
access_log /var/log/nginx/quanweb/bot_access.log combined if=$is_bot;
```

Change the log file path to your actual setup. What is added is the `combined if=...`. Because Nginx does not support the `if=!$some_var` syntax, we have to define two variables, `$is_bot` and `$is_not_bot`.

After saving the file, remember to test if your new config breaks Nginx:

```sh
$ sudo service nginx testconfig
```
If it says Ok, you can then apply the config:

```sh
$ sudo systemctl reload nginx
```

Now, check you main access log file. It no longer contains the visit of search bot. Note the the list of patterns to recognize bot above is not complete. You can extend it if you find something new.