Logging Strategy: Syslog and Webhooks

The first rule of automation logging is small: reach for the logging module, never print(). A print() call writes one unlabelled string to standard output with no severity, no filtering, and no way to fan the same line to a second place; logging gives you all three, as configuration rather than code you rewrite per environment (Python logging HOWTO^[1]).

The record's journey

Every log.info("device provisioned") call builds a LogRecord — an object holding the message, its level, the timestamp, and the logger name. The record then travels a fixed path:

1. The logger compares the record's level to its own effective level and drops anything below it. Python's levels are ordered DEBUG (10) < INFO (20) < WARNING (30) < ERROR (40) < CRITICAL (50), and a logger emits records at or above its configured level and discards the rest (logging levels^[2]).
2. Surviving records are handed to every handler attached to that logger, and by default to the handlers of its ancestor loggers too.
3. Each handler applies its own level and formatter, then ships the record to a destination.

Because the second step fans one record to many handlers, a single log call can reach several places at once. The standard-library handlers cover the destinations this exam cares about: SysLogHandler sends to a syslog daemon, HTTPHandler sends to an HTTP server with GET or POST, StreamHandler writes to a stream such as stdout, and FileHandler writes to a file (logging.handlers^[3]). Attaching several of them to one logger is how the same event lands in your SIEM (security information and event management platform), your chat tool, and a local file without duplicating the call.

Why not print()

print() participates in none of this: no level threshold, so debug noise ships to production; no handler list, so exactly one destination; and no structured record, so downstream tools must scrape free text. Prefer the logging module precisely because per-level filtering, structured output, and multiple simultaneous destinations then become settings you change without editing the code that emits the event (logging.handlers^[3]).

A grep across a million lines of free-form log text is a losing game; a query over named fields is not. That is the whole case for structured logging: emit each event as a machine-parsable object — JSON (JavaScript Object Notation) — carrying explicit fields such as timestamp, level, event, device, and correlation_id, so a collector can filter and aggregate on device or level without brittle regular expressions (Python logging HOWTO^[1]).

Emit JSON, do not build it by hand

You do not concatenate JSON strings in the log call. Attach a JSON formatter to a handler and every LogRecord serialises to one JSON line on its way out; the widely used python-json-logger formatter does exactly this, and a custom logging.Formatter subclass can too (logging cookbook^[4]). The call stays ordinary — log.info("config pushed", extra={"device": "rtr1"}) — while the formatter decides the wire shape. Keep one event per line (newline-delimited JSON) so a log shipper can read the stream incrementally.

Correlation IDs tie a run together

One automation run touches many devices, and its lines interleave with every other run on the collector. Mint a correlation ID once per run (a unique value, sometimes called a trace id), stamp it onto every line that run emits, and you can later pull the whole workflow back out with a single correlation_id == "..." filter, across devices and services (logging cookbook^[4]). The clean way to inject it is a LoggerAdapter or a logging.Filter that adds the field to each record, rather than threading the id through every function argument (logging cookbook^[4]).

Read one exam rule before anything else: on the syslog scale a lower number is more severe. Severities run from 0 (Emergency) to 7 (Debug), so 0 is the worst thing that can happen and 7 is chatty debugging (RFC 5424^[5]). The number-feels-backwards instinct — reading 7 as "highest priority" — is the classic trap. On Cisco IOS the command logging trap <level> sets the threshold sent to the syslog server, and it forwards that level and everything more severe (numerically lower): logging trap 4 sends severities 0 through 4 and suppresses 5 through 7 (Cisco System Message Logging^[6]).

Facility and the PRI value

Every syslog message also carries a facility (0 to 23) naming the source subsystem — the local-use facilities local0 to local7 are 16 to 23, for instance. Facility and severity are combined into one wire PRI (Priority) value as PRI = facility * 8 + severity (RFC 5424^[5]). A message from local7 (23) at Informational (6) therefore carries PRI = 23 * 8 + 6 = 190.

Severity	Name	Meaning
0	Emergency	system is unusable
1	Alert	act immediately
2	Critical	critical condition
3	Error	error condition
4	Warning	warning condition
5	Notice	normal but significant
6	Informational	informational message
7	Debug	debug-level message

RFC 5424 versus the legacy BSD format

Modern syslog uses the RFC 5424 structured format: a header of PRI, VERSION, TIMESTAMP, HOSTNAME, APP-NAME, PROCID, and MSGID, then optional STRUCTURED-DATA, then the free-text MSG (RFC 5424^[5]). Do not confuse it with the older RFC 3164 BSD format, whose looser, less-parseable layout RFC 5424 replaces (RFC 3164^[7]); when a question contrasts the two, RFC 5424 is the one with the named header fields and machine-readable STRUCTURED-DATA.

Getting it off the box reliably

By default syslog ships over UDP port 514 — fire-and-forget, and therefore lossy: a dropped datagram is simply gone (RFC 5424^[5]). For an automation logging strategy that must not lose events, send syslog over TCP port 6514 with TLS (Transport Layer Security), the reliable, encrypted mapping defined in RFC 5425 (RFC 5425^[8]). So the default is convenient but unreliable; the deliberate choice for durable delivery is TCP/6514 over TLS.

A webhook is just an outbound HTTP request your automation makes when something happens: an HTTP POST whose body is a JSON document, sent to an HTTPS endpoint the receiver controls. In Python it is one line — requests.post(url, json=payload) — which serialises the dict and sets Content-Type: application/json for you (requests quickstart^[9]). It is a POST carrying a body, never a GET; a webhook that "fires" with a bare GET and no payload is the wrong mental model.

Prove who sent it

An open POST endpoint will happily accept a forged event, so receivers authenticate the sender. Two patterns dominate. The sender computes an HMAC (hash-based message authentication code) over the raw body with a shared secret and puts it in a header — GitHub's X-Hub-Signature is the canonical example — and the receiver recomputes the same HMAC to confirm the body is untampered and came from someone holding the secret (RFC 2104^[10]). Or the sender presents a bearer token in the Authorization: Bearer <token> header, which the receiver checks against its issued tokens (RFC 6750^[11]). Either way, a POST that fails the check is rejected before it is processed.

Assume the POST can fail

Networks and receivers are unreliable: the endpoint may be down or return an HTTP 5xx. A robust delivery treats any non-2xx response as a failure and retries with backoff, instead of firing once and hoping. With requests, mount an HTTPAdapter whose max_retries is a urllib3 Retry configured with a backoff_factor and a status_forcelist of retryable codes:

from requests import Session
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retries = Retry(total=5, backoff_factor=0.5,
                status_forcelist=[502, 503, 504],
                allowed_methods={"POST"})
session = Session()
session.mount("https://", HTTPAdapter(max_retries=retries))
session.post("https://hooks.example.com/events", json=payload, timeout=5)

The backoff_factor spaces the attempts out so a struggling receiver is not hammered (requests advanced usage^[12]). Once the retry budget is spent, surface the failure — alert, or park the event on a dead-letter queue — rather than dropping it silently.

Pick the destination by who reads it downstream, not by whatever the last script happened to use. The three handlers from the first section map cleanly onto three consumers: a syslog collector or SIEM wants line-oriented, severity-tagged text, so stream to it with SysLogHandler; an application endpoint — a ChatOps room such as Webex, or a ticketing API — wants a structured JSON event, so push a webhook with an HTTP POST; and a container platform whose log shipper tails standard output wants newline-delimited JSON on stdout, so use a StreamHandler with a JSON formatter. These are not exclusive: because one logger fans a record to every attached handler (the pipeline from the first section), the same event can go to all three at once, and you choose per downstream consumer rather than once for the whole program.

The rule with no exceptions

Whatever the destination, never log secrets. Tokens, passwords, private keys, and the Authorization header itself must be redacted or masked before the record is written, because logs are copied, forwarded, and retained far more widely than the systems they describe (OWASP Logging Cheat Sheet^[13]). The tempting failure mode is logging a full HTTP request — headers and all — while debugging a webhook; that one line quietly writes a live bearer token into every downstream collector. Mask sensitive fields at the formatter or filter stage so the redaction holds no matter which handler ships the record, and pair it with the transport protections from the syslog and webhook sections (TLS on the wire, an HMAC signature or bearer token at the receiver) so a log line is protected end to end.