Filter out bot visits from Gunicorn log

Our web is often written in Python, and to run the web app on production, we often use Gunicorn. Its log is also a resource for incident investigation. But the log of bot visits is so noisy. How to exclude them?

When running Gunicorn, we often have a config file for Gunicorn. We often name it, with content like this:

proc_name = 'awesome-web'
workers = 6
worker_tmp_dir = '/dev/shm/'

# Make short log line. Some info is discarded, because it is shown by journalctl already.
logconfig_dict = {
    'formatters': {
        'generic': {
            'format': '[%(levelname)s] %(message)s',
    'loggers': {
        'gunicorn.error': {
            'level': 'INFO',
            'handlers': ['error_console'],
            'propagate': False,
            'qualname': 'gunicorn.error',
        'gunicorn.access': {
            'level': 'INFO',
            'handlers': ['console'],
            'propagate': False,
            'qualname': 'gunicorn.access',

To tell Nginx not to log visits of bots, we will manipulate Gunicorn logger object. First, define a function to identify bots (search bots and crawling bots) and a logger filter class:

import logging
from logging import LogRecord

def is_bot(user_agent: str):
    bot_ids = (
        'Sogou web spider',
    return any(s in user_agent for s in bot_ids)

class BotIgnoreFilter(logging.Filter):
    def filter(self, record: LogRecord) -> bool:
        passed = super().filter(record)
        # Ref:
        user_agent = record.args['a']
        from_bot = is_bot(user_agent)
        return passed and not from_bot

Then, we inject the code of setting logger into Gunicorn's on_starting hook:

def on_starting(server):

Done. If you let Gunicorn controlled by systemd, you can use systemctl to tell Gunicorn to re-read new config (given that your systemd unit file is our-web.service):

$ sudo systemctl reload our-web.service

Gunicorn's way of using Python script for configuration looks weird as first. But in some situation, like this case, it is an advantage.