[time-nuts] irregular data and Stable32

[Tom Van Baak]

Demetrios,

 > In my personal software I don’t deal with outliers or data gaps by 
interpolating,
 > spline-fitting, or any other fancier things.
 > Instead I compute the standard statistical measures by simply 
ignoring terms
 > in the summation that hit upon a data gap or outlier.

For critical data, consider doing it both ways and comparing the 
results. If the ADEV plots come out the same then it doesn't matter 
which one you choose. If the plots are different then pick the right 
one, which could be the interpolated method, or it could be the gap method.

I say this because I use Stable32 for mechanical clock data as well as 
quartz or atomic data. In these cases one must never ignore "bad" data 
or it will ruin any FFT / PSD / ACF processing. The periodic effects are 
large and often interesting in mechanical clock so playing games with 
even a single data point can ruin your day.

 > I don’t think Stable32 does that, so I’d like to pose this challenge 
to anyone

Actually, Stable32 already has code to handle gaps. It's buried in the 
manual somewhere. Any zero data is automatically treated as a gap and, 
like your method, it is excluded from statistical measures. It applies 
to both phase and frequency data.

You can see this in action for yourself. Take these 10 random points:

   0.516603
  -0.327618
   0.415049
   0.138100
   0.0
   0.730480
  -0.762049
   1.586196
   0.648489
  -1.586448

and import them as phase and then click plot. You'll notice that 
Stable32 does not shift the data down to cover the gap. Gaps are visible 
by a break in the plot line. And they are not part of the statistics. 
The same is true if you import them as frequency and click plot. Note if 
you convert phase->freq or freq->phase the double break.

All this seems carefully thought out in Stable32. It must have been a 
pain to implement.

Note that this automatic gap logic is awkward if you have legitimate 
phase 0 or frequency 0 data. One trick is to replace 0's by 1e-99 in 
your data so that Stable32 doesn't treat them as gaps.

/tvb