[time-nuts] can of worms: time-of-day in a community radio station

[Eric Scace]

   I fear that I am developing a reputation for bringing to the list rather oddball questions. In my rôle as agent provocateur, therefore, here is another such problem.

   Questions for you are at the end. Thanks for your thoughts,.

— Eric

Issue:
   A community broadcast radio station with multiple studio locations wishes to improve the display of time-of-day throughout the station’s operating environments. Its current legacy approaches (described below) cause problems such as:
on-air presenters fail to smoothly segue into scheduled program feeds (e.g., BBC news) because the studio clock is “a little off”… and because the studio clock is digital. [Quick: how many more seconds can you speak before the top of the hour when a digital clock shows 4:59:42? Watching an analog stepping second hand is much easier in this situation.]
computers that automatically capture audio programs to files in storage sometimes truncate the start or end of the program because the computer’s idea of time-of-day disagrees with that of the program source.
desktop computers throughout the station, including in production studios, all render slightly different versions of time-of-day to their users.
servers (e.g, for streaming, for archiving shows) are similarly in cheerful disagreement as to time of day.
wall clocks studios in one city show a different time to their engineers than the studios in another city, rendering handoffs more complicated. Ditto for remote broadcast sites, and even between studios in the same site.
requires manual intervention to bring the most egregious systems back to some semblance of reality

Background & existing situation:
   Commercial broadcast stations have more money and technology to solve these problems. In contrast, “community radio” stations have limited funds and are largely staffed by volunteers (like me!).

   In this case, the existing systems are a hodgepodge:
mostly Windows OS PCs, with a couple of Macs
Linux servers
mash-mash of wall clocks, the best of which is a LaCrosse WWVB digital in the primary on-air studio. The WWVB signal is more than adequate but the LaCross display format is sub-optimal for studio use.

Goals: (first pass)
minimum accuracy requirement: time-of-day displayed within ±0.1 second of UTC timescale. (Two clocks both falling outside this range will cause program handoffs to be uncomfortably tight or loose.)
no manual intervention required for summer/winter time transitions
no manual intervention required for leap seconds
leap second:
no smearing (minimum requirement)
accurate leap second display (desirable but unlikely to be achievable)
desirable consistency goal: time-of-day displayed within ±0.025 second throughout each site. At this level, two adjacent clock displays will not be perceived as out of step by a person.
presentation goals: studio/remote broadcast control point time-of-day displayed in both analog (with stepped seconds hands) and digital form (preferred H:MM).
If digital form includes seconds, the seconds digits should be visually separated; e.g..smaller. A presenter can then, at a glance and without confusion, announce the time (“four twenty-three”) from the digital display.
Date in form “Oct 17 Thu” available.
medium-term desirable: displays continue within specs for accuracy/consistency across power cutovers (to/from generator) and public Internet outages.
maintenance goals:
"eschew emergencies”: no one should have to rush to the station in the middle of the night, nor drop what their doing during the day, because time-of-day display has failed or gone out-of-spec.
standardize on the same hardware/software everywhere
identical hardware allows more cost-effective 1:N sparing
identical hardware/software means less confusion and less training for volunteers. (There are a large number of volunteers who use these systems, and most contribute time less than once per week. Consistency and straightforwardness in the toolkit improves the quality of on-air results.)
not too costly
try to avoid stringing a lot of new cabling around… but such cabling needs are recognized as a one-time investment, so this preference does not eliminate a cable-based solution that has other operational/maintenance advantages

Potential approaches:
potential short-term:
desktops and servers: better NTP configuration
NTP checks:
at least hourly?
verify NTP utility employs reasonable averaging of multiple NTP servers
use time.nist.gov <http://time.nist.gov/> and similar higher-quality NTP servers
some systems are on in-house Ethernet; others on in-house wi-fi
clocks:
replace with refurb iPads running a dedicated time-of-day display app providing both analog and digital form. A refurb wall- or desk-mounted 13” iPad in locking frame runs about $100. Smaller sizes can be mounted immediately adjacent to or atop a studio mixing board.
remote field broadcast site: iPad over local wi-fi or cellular data?
IRIG displays could be awesome, but seem far more costly per display and require pulling coax — and integrated analog + digital displays appear to be less common. Two displays (one analog, the other digital) in each studio provide some redundancy but costs go up fast.
NTP-clocks powered over ethernet could similarly be awesome, but are similarly more costly and few integrated analog + digital displays exist.
medium term:
add in-house NTP server (with GPS and reasonable holdover plus battery backup) on each city’s on-site LAN/wi-fi networks as primary source, with remote public NTP servers as secondary. Many suitable models appear to exist, including the ESE E-911 units mentioned recently. Only a few are needed: one for each site plus sparing.
potentially increase the frequency of NTP synchronization for each computer/display clock when a local NTP server has been installed.

Questions:
Are there other approaches that should be considered?
What NTP software should be used on Windows OS machines? Linux servers?
Mac OS allows one choice of NTP server but does not seem to provide for choice of NTP update frequency. Is there a 3rd party software solution, or some other parameter within MacOS that an admin can change to (a) establish a primary and secondary NTP server, and (b) set the frequency of NTP updates?
If one were to use an iPad to display time-of-day, what iPad apps provide the needed display formats, frequency of NTP checks, primary/secondary NTP sources?
For example, Atomic Clock Gorgy updates hourly and allows the user to choose one NTP server group (e.g., time.nist.gov <http://time.nist.gov/>). It has an analog seconds display format (circle of dots), digital H:MM (optional :SS), and MMM D DDD — although one could quibble over the typography. However, the display shows “SYNC” for a couple of seconds each hour while the NTP update occurs, which would be disconcerting if it happens when a presenter/engineer is in the midst of joining/cutting away from external program sources.
What have we overlooked?