Check any service with send/expect

What you'll know

• The service on the port is the one you think it is, and it's answering. Gryphon connects, optionally says something, and waits for text that only that service would send back.
• It isn't just an open socket. A plain port check passes as soon as the connection opens. The operating system can accept the connection for a program that has hung, so a stuck mail server still looks fine. Send/expect needs the program itself to reply.

What this can't tell you: anything past the first exchange. It doesn't sign in, send a message or run a query. For that, use a script check, which can do anything a command can.

How it works

Each time the check runs, Gryphon:

1. connects to the port, and completes a TLS handshake first if you ask it to;
2. writes what's in Send, if anything;
3. reads the reply, for up to 10 seconds and up to the first 4 KB, looking for what's in Expect.

If the text arrives, the check is healthy. A wrong reply, silence, a closed or refused connection, or a failed handshake is a problem. The message says what came back, which is the quickest way to fix a wrong setting.

Services come in two kinds, and the kind decides what goes in Send:

• Services that speak first greet every connection before it says anything: SMTP, SSH, FTP, IMAP, POP3 and NATS. Leave Send empty and expect the greeting.
• Services that wait to be asked say nothing until the client does: Memcached, ZooKeeper, Beanstalkd, RabbitMQ and MQTT brokers. Put a harmless request in Send and expect its answer.

Before you start

• The machine, added as a host in Gryphon, and the port the service listens on.
• If the port is open to the internet, the check needs nothing else: it's TCP send/expect. If it's only reachable inside your network, install the Gryphon agent there and use TCP send/expect (agent), which has a Host field for the address the agent should connect to.
• A terminal on any machine that can reach the port, to see what the service says.

If your service is in the ready-made settings below, you can skip to step 3.

Steps

1See what the service says

Connect by hand and look. On Linux or a Mac, nc shows a service that speaks first. Give it a few seconds, and it prints the greeting:

nc -w 3 mail.example.com 25

220 mail.example.com ESMTP Postfix

For a service that waits, send the request and see the answer. \r\n ends the line:

printf 'version\r\n' | nc -w 3 cache.internal 11211

VERSION 1.6.45

For a port that speaks TLS from the first byte, such as 993 or 995, use openssl instead:

openssl s_client -quiet -connect mail.example.com:993 < /dev/null

On Windows, PowerShell can do the same. Change the address and port, and the request if it needs one:

$c = New-Object Net.Sockets.TcpClient('cache.internal', 11211)
$s = $c.GetStream(); $s.ReadTimeout = 3000
$send = [Text.Encoding]::ASCII.GetBytes('version' + [char]13 + [char]10)
$s.Write($send, 0, $send.Length)
$b = New-Object byte[] 4096
[Text.Encoding]::ASCII.GetString($b, 0, $s.Read($b, 0, $b.Length))
$c.Close()

For a service that speaks first, leave out the two lines that build and write $send.

2Pick the text to expect

Choose something short that this service always says, and nothing else would:

• Good: the start of the greeting or the answer, such as 220, SSH-, * OK or VERSION.
• Avoid version numbers and host names. VERSION 1.6.45 stops matching on the next upgrade, and the check turns into a problem for no reason.
• Match the case. ssh- doesn't match SSH-2.0-OpenSSH.

3Add the check

Open the host, go to Manage Services, choose Add service, and pick TCP send/expect. This one watches SSH from outside:

This one watches Memcached on a private address, through the agent. Pick TCP send/expect (agent) instead:

The fields, one by one

• TLS:
  • none is plain TCP.
  • tls completes a TLS handshake first and requires a certificate that's trusted and issued for the address, as a browser would.
  • tls-no-verify completes the handshake but accepts any certificate, for an internal service with a self-signed one.
  Choose tls only for a port that speaks TLS from the very first byte. A port that starts in plain text and upgrades later, such as SMTP on 25 or 587, is none.
• Send is written once, as soon as the connection is open. Write special characters as escapes. Spaces at the start or end of either field are dropped, so write a space there as \x20.
• Expect takes the same escapes. Leave out the line ending: the text only has to appear somewhere in the reply.

Ready-made settings

Every row was tested against a real server. Use the service's default port unless yours listens elsewhere.

• ZooKeeper answers ruok only when it's allowed: 4lw.commands.whitelist=ruok in zoo.cfg, or ZOO_4LW_COMMANDS_WHITELIST=ruok for the Docker image. Otherwise it replies that ruok is not in the whitelist, and closes the connection.
• RabbitMQ: the send is the start of an AMQP 0-9-1 connection, and RabbitMQ names itself in its reply. It logs a warning about a client that closed its connection each time. The RabbitMQ guide goes further.
• MQTT: the send is a minimal MQTT 3.1.1 connect request, and \x20\x02 is the start of the broker's reply. A broker that refuses anonymous clients still replies before it closes the connection, so this works either way.
• Mail over TLS: with a self-signed certificate, choose tls-no-verify. The mail server guide covers the certificates and the MX record too.

Test it

1. Use Check now on the new check. A healthy result names the text it found and how long the reply took, such as answered with "SSH-" in 13ms.
2. Stop the service, or point the check at a port nothing listens on. It becomes a problem. Gryphon rechecks every minute once a check fails, and alerts when three results in a row agree, so warn whoever gets the alerts first.

When it goes wrong

The message quotes what the service sent back, so most fixes are copied straight out of it.

• did not answer with "VERSION" within 10s; got nothing. The service is waiting for the rest of your request. Add \r\n to the end of Send.
• did not answer with "* OK" within 10s; got nothing, on a port such as 993, 995 or 465. The port speaks TLS and the check doesn't. Set TLS to tls.
• did not answer with "ssh-" within 10s; got "SSH-2.0-OpenSSH_10.3\r\n". It answered, but not with the text you expected. Copy the right text from the reply.
• TLS handshake failed. The certificate isn't trusted or isn't issued for the address. Use tls-no-verify for a self-signed certificate, and watch its expiry with an SSL Certificate check on the same port.
• closed the connection without answering. The service turned the request down, and what it got back usually says why: ZooKeeper replies ruok is not executed because it is not in the whitelist.
• connection refused. Nothing is listening on that port, or a firewall is rejecting Gryphon. Checks from outside come from the addresses on the check locations page.

When to use something else

• Websites and APIs: Monitor web page, or HTTP (agent) and HTTPS (agent). They speak HTTP properly, follow redirects and judge status codes. See health endpoints.
• PostgreSQL, MariaDB, MySQL and Redis: Gryphon has checks of their own, which need no hand-written handshake.
• Signing in, running a command, or reading a number: a script check.
• Only whether the port is open: TCP port, which connects and nothing more.
• Services on UDP: these checks speak TCP only. DNS has its own check; for anything else, use a script.