Monitor MongoDB

What you'll know

• MongoDB is accepting connections. A port check, sent from inside your network by the agent, or from outside if the port is open to the internet.
• The server answers, and its replica set has a primary. A script asks MongoDB itself. No primary means writes are failing, and that's a problem. A member that is neither primary nor secondary, such as one still recovering, is a warning.

The script uses MongoDB's hello command, which every server answers before anyone signs in. So this needs no MongoDB user and no password, even with access control turned on.

What this can't tell you: how far behind a secondary is, or whether queries are slow. Those need a user with the clusterMonitor role, beyond what this guide sets up. For backups, see Make sure last night's backup ran. It works the same with mongodump.

Before you start

• The MongoDB machine added as a host in Gryphon, and the Gryphon agent on it.
• mongosh, MongoDB's shell, on the same machine. It comes with MongoDB's packages, or on its own as mongodb-mongosh.

The example assumes MongoDB listens on 127.0.0.1:27017.

Steps

1Check the port

Open the host, go to Manage Services, choose Add service, and pick TCP port (agent).

TCP port does the same from outside with no agent, but only if 27017 is open to the internet. It almost never should be.

2Turn on script checks

The agent only runs programs from a folder you name, and the folder must belong to root with nobody else able to write to it:

sudo install -d -o root -g root -m 0755 /etc/gryphon/scripts

GWC_SCRIPTS_DIR=/etc/gryphon/scripts

sudo systemctl restart gryphon-agent

3Add the script

Save this as /etc/gryphon/scripts/mongodb-state and make it executable with sudo chmod 755 /etc/gryphon/scripts/mongodb-state.

#!/bin/sh
# Whether MongoDB answers, and whether its replica set has a primary.
# Asks with "hello", which every MongoDB answers before anyone signs in.
export MONGO_URI="mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=4000"
export HOME=/tmp    # mongosh keeps a log in the home directory; the agent has none

exec mongosh --nodb --quiet --norc --eval '
try {
    const h = connect(process.env.MONGO_URI).hello();
    if (!h.setName) {
        print("OK: MongoDB answers (standalone)");
        quit(0);
    }
    const role = h.isWritablePrimary ? "primary" : h.secondary ? "secondary" : "neither primary nor secondary";
    if (!h.primary) {
        print("CRITICAL: replica set " + h.setName + " has no primary; this member is " + role);
        quit(2);
    }
    if (!h.isWritablePrimary && !h.secondary) {
        print("WARNING: this member of " + h.setName + " is " + role + "; the primary is " + h.primary);
        quit(1);
    }
    print("OK: " + h.setName + " primary is " + h.primary + "; this member is " + role);
    quit(0);
} catch (e) {
    print("CRITICAL: MongoDB did not answer: " + e.message);
    quit(2);
}'

The exit code is the status: 0 healthy, 1 warning, 2 problem. The first line it prints is the message you'll see in Gryphon and in the alert. The agent runs scripts as its own user, with no home folder. Without the HOME line, mongosh's complaint about that would become the message.

Run it the way the agent does, to see it work:

sudo systemd-run --pipe --wait -q -p DynamicUser=yes -p PrivateTmp=yes /etc/gryphon/scripts/mongodb-state

4Add the script check

In a replica set, install the agent and the script on every member, and add both checks to each member's host. Each member reports its own view: whether it answers, its own role, and whether it can see a primary.

Test it

1. Use Check now on the script check. In a replica set, the message names the set, its primary, and this member's role.
2. On a test machine, stop MongoDB with sudo systemctl stop mongod. Both checks become problems within a few minutes, and the alert goes out. The script's message says MongoDB didn't answer.
3. Start it again. Both checks recover, and the recovery is announced like any other.

Variations

A server that requires TLS

Add MongoDB's TLS options to the address in the script, for example &tls=true&tlsCAFile=/etc/gryphon/mongodb-ca.pem, with a copy of the certificate authority the agent can read. The hello command still needs no user.

Another address or port

Change 127.0.0.1:27017 in MONGO_URI. Keep directConnection=true: it keeps the script talking to this member only, so the answer is this member's own view.

Running in Docker

If MongoDB is a container on the agent's machine, add a Docker: container check as well. It sees a container that's crashed or keeps restarting. The script still runs on the machine itself, pointed at the port the container publishes, so mongosh is needed there.