Log bot requests in nginx - Rocketeers app

  [ Rocketeers ](/)   

[Login](https://rocketeersapp.com/login) 

 On this page

 Knowledge
---------

Log bot requests in nginx
=========================

### [\#Nginx](https://rocketeersapp.com/knowledge/nginx)

If you want to know when and how often bots visit your website, you can easily track this using the following configuration in nginx.

 Published by [Mark van Eijk](https://rocketeersapp.com/author/mark-van-eijk) on November 18, 2024 
Updated on November 18, 2024 · 1 minute read

1. [Detect bots using nginx](#content-detect-bots-using-nginx)
2. [Readble log format](#content-readble-log-format)
3. [Log requests when it's a bot](#content-log-requests-when-its-a-bot)
4. [Log files per bot](#content-log-files-per-bot)

[\#](#content-detect-bots-using-nginx "Permalink")Detect bots using nginx
-------------------------------------------------------------------------

First we need to detect if the current visitor's user agent indicates it is a bot. We can do this with the `map` directive in nginx:

 ```
map $http_user_agent $bot {
    default "";
    "~*googlebot" "google";
    "~*bingbot" "bing";
    "~*slurp" "yahoo";
    "~*duckduckbot" "duckduckgo";
    "~*baiduspider" "baidu";
    "~*yandexbot" "yandex";
    "~*sogou" "sogou";
    "~*exabot" "exabot";
    "~*applebot" "apple";
    "~*twitterbot" "twitter";
}

```

This checks the user agent string for matches for known bot names and then it maps it to a specific name that is set to variable `$bot`.

[\#](#content-readble-log-format "Permalink")Readble log format
---------------------------------------------------------------

To make the entries readable, you can optionally choose to define a specific `log_format` for the bot requests:

 ```
log_format bots "$time_local: $request_method $scheme://$host$request_uri [$status] $bytes_sent @ $request_time ($http_referer)";

```

[\#](#content-log-requests-when-its-a-bot "Permalink")Log requests when it's a bot
----------------------------------------------------------------------------------

Now we can log these bot requests by creating a specific `bots.log` file using the `access_log` directive that logs requests only if `$bot` is filled and set the `log_format` to the newly created `bots` format.

 ```
access_log /var/www/logs/bots.log bots if=$bot;

```

[\#](#content-log-files-per-bot "Permalink")Log files per bot
-------------------------------------------------------------

If you prefer separating the logs per bot, so you can more easily see how many times specifically the Googlebot has come by your website, you can define a variable per bot:

 ```
if ($bot = "google") {
    set $google "1";
}
if ($bot = "bing") {
    set $bing "1";
}
if ($bot = "yahoo") {
    set $yahoo "1";
}
if ($bot = "duckduckgo") {
    set $duckduckgo "1";
}
if ($bot = "baidu") {
    set $baidu "1";
}
if ($bot = "yandex") {
    set $yandex "1";
}
if ($bot = "sogou") {
    set $sogou "1";
}
if ($bot = "exabot") {
    set $exabot "1";
}
if ($bot = "apple") {
    set $apple "1";
}
if ($bot = "twitter") {
    set $twitter "1";
}

```

And therefore setup log files per bot:

 ```
access_log /var/www/logs/bots/google.log bots if=$google;
access_log /var/www/logs/bots/bing.log bots if=$bing;
access_log /var/www/logs/bots/yahoo.log bots if=$yahoo;
access_log /var/www/logs/bots/duckduckgo.log bots if=$duckduckgo;
access_log /var/www/logs/bots/baidu.log bots if=$baidu;
access_log /var/www/logs/bots/yandex.log bots if=$yandex;
access_log /var/www/logs/bots/sogou.log bots if=$sogou;
access_log /var/www/logs/bots/exabot.log bots if=$exabot;
access_log /var/www/logs/bots/apple.log bots if=$apple;
access_log /var/www/logs/bots/twitter.log bots if=$twitter;

```

### Subscribe to our newsletter

Do you want to receive regular updates with fresh and exclusive content to learn more about web development, hosting, security and performance? Subscribe now!

  Fill in your email address to receive updates  Subscribe 

#### More in [\#Nginx](https://rocketeersapp.com/knowledge/nginx)

- [Server Side Includes (SSI) in nginx](https://rocketeersapp.com/knowledge/server-side-includes-ssi-nginx)
- [Configure Content Security Policy with nonce using nginx](https://rocketeersapp.com/knowledge/content-security-policy)
- [nginx rewrite or internal redirection cycle](https://rocketeersapp.com/knowledge/nginx-rewrite-or-internal-redirection-cycle)
- [nginx: upstream sent too big header](https://rocketeersapp.com/knowledge/nginx-upstream-sent-too-big-header)
- [How to install Nginx on Ubuntu](https://rocketeersapp.com/knowledge/how-to-install-nginx)
- [Detect Googlebot visits using nginx](https://rocketeersapp.com/knowledge/detect-googlebot-nginx)

 [View all 7 articles →](https://rocketeersapp.com/knowledge/nginx)