============================================================================== pull_nws_obs.py -- MAINTAINER README ============================================================================== Last reviewed against script version v0.105 / v0.106. Audience: whoever inherits this script. Assumes general comfort with Python and Linux/cron, but NOT with this particular pipeline. Read this before you change anything. 1. WHAT IT DOES ------------------------------------------------------------------------------ Pulls the latest surface observation (temp, dewpoint, RH, wind dir/speed/gust, ob time) for a curated list of stations from the National Weather Service public API (api.weather.gov) and writes them into one fixed-width text file that the wxnw2 website reads and displays. It is the "fallback"/simple obs source -- it does not depend on MesoWest/Synoptic. On each run it ALSO drops a timestamped copy of that file into an archive folder and deletes archive copies older than 49 hours, giving a rolling ~2-day history. 2. FILES IT TOUCHES ------------------------------------------------------------------------------ All paths are under OUTDIR, set at the top of the script. Default: /var/www/html/weather/text_products/observations weather_obs_stations.txt INPUT. The station list. One station per line as "STID, Name". Blank lines and lines starting with '#' are ignored. STID must be a valid NWS station id -- for airports that is the ICAO id (KSLE, KMFR, KPDX...). This file is hand-maintained; the script never edits it. nws_obs_latest.txt OUTPUT. The live product the website reads. Overwritten every run. archive/ OUTPUT. Rolling history. Files are named nws_obs_YYYYMMDD_HHMM.txt using a UTC timestamp. (With ROUND_MINUTES=60 the minutes are always 00, e.g. nws_obs_20260811_1700.txt.) raw_obs.txt OPTIONAL DEBUG. Only written when SHOW_RAW=True. Full JSON dump of every station's API response. Overwritten every run. Leave SHOW_RAW=False for production. No external Python packages are required -- standard library only. It does need Python 3.9+ for the zoneinfo module (see gotchas). 3. HOW IT RUNS ------------------------------------------------------------------------------ From cron, twice an hour: 5,15 * * * * /usr/bin/python3 /full/path/pull_nws_obs.py >> \ /full/path/observations/pull_nws_obs.log 2>&1 The :05 and :15 runs BOTH write to the same hourly archive file. The :15 run overwrites the :05 copy on purpose -- it exists to pick up stations that reported late. This two-run pattern only makes sense while ROUND_MINUTES=60; see gotcha (d). Run it by hand exactly the same way to test. It is chatty on stdout: version banner first, then a "wrote N stations" line, then "archived copy: ...", "pruned N archived files...", and elapsed time. That output is what lands in the log. 4. CONFIG KNOBS (top of file) ------------------------------------------------------------------------------ OUTDIR / OUTFILE / STATIONS_FILE / ARCHIVE_DIR paths ARCHIVE_HOURS 49 prune archive files older than this (by mtime) ROUND_MINUTES 60 interval the archive-filename UTC time is rounded to SHOW_RAW False debug JSON dump toggle TITLE header line written into the output file USER_AGENT sent to api.weather.gov -- see gotcha (a), IMPORTANT Nothing else should need routine editing. The station list lives in its own file, not here. 5. HOW THE ARCHIVE + PRUNE WORKS (and why to leave it alone) ------------------------------------------------------------------------------ After the main file is written, archive_output() runs: - makes archive/ if missing - takes the current UTC time, rounds it with round_dt() to ROUND_MINUTES, and copies nws_obs_latest.txt to archive/nws_obs_.txt - deletes any file in archive/ whose name matches ARCHIVE_RE ( ^nws_obs_\d{8}_\d{4}\.txt$ ) AND whose mtime is older than ARCHIVE_HOURS. The regex is the safety net: the prune can ONLY delete files that match this exact name shape. Anything else in archive/ is untouchable. See gotcha (c) before you ever change the naming scheme. The whole archive+prune block is wrapped in try/except in main(). If it fails (disk full, permissions, etc.) it prints a warning to stderr and the run still produces nws_obs_latest.txt. The website product is never held hostage to the archive step. That is by design -- keep it that way. 6. GOTCHAS ------------------------------------------------------------------------------ (a) USER-AGENT IS MANDATORY. api.weather.gov rejects generic/blank User-Agent strings and can return 403. The USER_AGENT constant must stay descriptive and contain a working contact. If every station suddenly comes back empty (all "." columns), suspect this first. (b) UTC IN THE FILENAME, PACIFIC INSIDE THE FILE. Archive filenames use UTC on purpose (clean sorting, no DST fall-back collision). The TIME column and the "Updated at" line INSIDE the file are Pacific local. This mismatch is intentional -- do not "fix" it. (c) DO NOT CASUALLY RENAME THE ARCHIVE FILES. Retention depends on ARCHIVE_RE matching the names the script writes. If you change the naming scheme but not the regex, the prune silently stops matching and archive/ grows forever. If you LOOSEN the regex, you risk deleting files you did not mean to. Change both together, deliberately, and test in a scratch folder first. (d) ROUND_MINUTES AND THE TWO-RUN SCHEDULE ARE LINKED. At 60, the :05 and :15 runs share one hourly file (:15 overwrites :05). If you drop ROUND_MINUTES back to something small (say 5), those two runs will produce TWO separate files per hour and the "late reporter" overwrite trick stops working. Also note round_dt() rounds to NEAREST, so at 60 a run past :30 would round UP to the next hour; the :05/:15 schedule never trips this, but a different schedule might. If you want strict "top of the current hour" always, use a floor instead (replace the stamp line with a .replace(minute=0, second=0, microsecond=0) truncation). (e) RETENTION IS mtime-BASED. A file's age is its filesystem modification time, which equals when the script wrote it. If archive files ever get resync'd, copied, or `touch`ed, their mtime resets and the 49-hour math is thrown off (old files look new and linger, or vice versa). Don't run maintenance jobs that rewrite mtimes on this folder. (f) ASCII-ONLY SOURCE. Keep the .py pure ASCII -- no smart quotes, em-dashes, or other non-ASCII bytes. Text-mode file transfers (WinSCP) transcode those and corrupt the script. Same rule applies to this README. (g) VERSION LINE. Bump VERSION by 0.001 each edit and keep the changelog block current. The version prints at the top of every run's output, so the log tells you which copy actually ran -- your defense against editing one copy and running a stale one. 7. LIKELY FAILURE POINTS / TRIAGE ------------------------------------------------------------------------------ SYMPTOM: file writes, but many/all data columns are "." - Per-station API errors are caught and rendered as "." so one bad station can't sink the run. A FEW dots = those stations didn't report or the id is stale; check/repair weather_obs_stations.txt. - ALL dots across every station = systemic: NWS API outage, rate limiting, or a broken/blocked USER_AGENT (gotcha a). Set SHOW_RAW=True, run by hand, and read raw_obs.txt for the actual error. You can also hit https://api.weather.gov/stations/KSLE/observations/latest in a browser. SYMPTOM: script crashes immediately with a ZoneInfo / tzdata error - The system timezone database is missing. Install the OS tz data (or the Python 'tzdata' package). zoneinfo needs it for "America/Los_Angeles". SYMPTOM: "archive step failed" on stderr, but the product still updates - Working as designed. Usually a permissions or disk-space problem on archive/. Check that the cron user can write there and the disk isn't full. The website keeps working meanwhile; fix at leisure. SYMPTOM: archive/ grows without bound / old files never deleted - Naming vs regex drift (gotcha c), or mtimes got rewritten (gotcha e). SYMPTOM: nothing runs at all - Cron path/interpreter (gotcha h), or the cron user lacks write permission to OUTDIR. Check the log file and the system cron log. GENERAL FIRST STEP for anything: run it by hand the same way cron does and read the console output. It tells you the version, station count, archive path, and prune count every time. ==============================================================================