The silent failure: nothing ran

A backup that DID NOT START and a job that DID NOT FIRE produce NO ERROR AT ALL. Watching for ABSENCE, which nothing does by default. Every monitor in an estate watches for things going wrong, and the most dangerous failures are things not happening. A BACKUP THAT DID NOT RUN PRODUCES NO FAILURE. The job did not start — a scheduler that was not running, a host rebooted, an entry somebody removed — so there is nothing to alert on. Monitoring that watches for backup failures reports a clean month across four weeks with no backups. THAT IS THE SHARPEST CASE BECAUSE IT COMBINES WITH MODULE 5 LESSON 2: NOBODY TESTS RESTORES EITHER, SO THE ABSENCE IS DISCOVERED THE DAY ONE IS NEEDED. THE SAME SHAPE COVERS EVERY SCHEDULED THING IN THE ESTATE. A nightly interface that did not fire. A purge that stopped running months ago — Module…

In this module: Module 13 · Monitoring, and knowing before the client does

1. 1What is worth watching
2. 2Checking availability honestly
3. 3The check that runs on the box it watches
4. 4Polling intervals, and what they cannot see
5. 5Alerting somebody who will act
6. 6Alert fatigue as a design failure
7. 7The silent failure: nothing ran
8. 8Reporting to somebody non-technical
9. 9What breaks: the outage nobody was told about