Watchdog for the night run

The Night Shift Runner starts the Christmas card project at night, but it has a weak spot. Sometimes InDesign stops responding in the middle of a run: no error, nothing in the log, it just hangs. The runner can’t do anything about it, because it’s waiting on the DoScript line for the main script to finish, which never happens. So you come in at 9 a.m. and find that the batch stopped at 3:05, after one class.

Finding and removing the cause is the real fix (see When InDesign stops responding). But some causes, like a network drive that drops out for a minute, can’t be fully removed. So I’m adding a watchdog: a safety net that notices the hang, restarts InDesign and carries on. This page describes how it works.

The idea

A watchdog is a second, small program that runs next to InDesign during the night run. It doesn’t do any of the work. It only checks, every minute, that the work is still moving. If InDesign has been stuck for too long, it kills InDesign and starts the run again.

It has to be a separate program. The runner can’t watch InDesign while it’s sitting on DoScript, so the runner starts the job in a separate process and does the watching itself.

The heartbeat

How does the watchdog know that InDesign is stuck and not just busy? The main script updates a small heartbeat file regularly: at the start of every class, after every PDF export. The watchdog looks at the file’s “last modified” time:

updated recently — InDesign is working, leave it alone;
not updated for, say, 20 minutes — InDesign is stuck, because no single class takes that long.

Why not the log file? Usually the script logs errors only, and on a good night it hardly writes anything. A separate heartbeat file is more reliable, and it can also hold the current school and class, so the watchdog knows where it got stuck.

When InDesign is stuck

Write a line to the runner’s log, for example: “03:47 — no progress for 20 min, last step: Year 2 / Mugs, restarting (attempt 1 of 3)”.
Kill InDesign.exe, and any EXCEL.EXE left hanging from reading the spreadsheet.
Clean up after the crash: delete the stale lock files (*.idlk) next to the intermediate InDesign documents and InDesign’s recovery data. Otherwise, the next InDesign may refuse to save over a “locked” document or offer to recover documents — and a dialog at 3 a.m. blocks the run just as well as a hang.
Start InDesign and run the main script again with the same saved settings.

Carrying on where it stopped

The nice part is that the main script hardly needs any changes for this. With the Overwrite existing PDFs option off, it already skips every PDF that exists and isn’t suspiciously small. So the second run quickly passes the schools and classes that are done and continues from where the first one froze. A PDF that was half-written when InDesign was killed is too small, so it’s made again. The files that are always rewritten (the checklists, the certificate, the school’s magnet and mug sticker sheets) are simply rebuilt, which is harmless.

Hangs that come back

If InDesign freezes in the same place every time — a broken scan, a bad template, a network drive that’s gone for the night — restarting forever just repeats the freeze. So the watchdog has limits:

after three hangs on the same school, the school goes onto a skip list, which the main script reads, and the next restart moves on to the next school;
after a fixed number of restarts in total, the watchdog gives up and writes a clear message.

Either way, in the morning you have all the other schools done and one line saying which school needs attention.

What changes

Main.jsx: writes the heartbeat and reads the skip list.
NightShiftRunner.vbs: starts the job in a separate process, watches the heartbeat, and does the kill, clean-up, restart and the retry limits.

Nothing changes in the products, the templates or the spreadsheet. Ideally, the user doesn’t notice the watchdog at all: the PDFs are just there in the morning. If a hang happened, the runner’s log shows when, where and that it recovered. A hang at 3 a.m. costs twenty minutes instead of the whole night.