[time-nuts] irregular data and Stable32
Demetrios Matsakis
dnmyiasou at yahoo.com
Wed Jan 20 19:02:21 UTC 2021
I think I deleted the duplicate posts relevant to data gaps in deviation statistics, and apologize if I took anybody’s out.
In my personal software I don’t deal with outliers or data gaps by interpolating, spline-fitting, or any other fancier things. Instead I compute the standard statistical measures by simply ignoring terms in the summation that hit upon a data gap or outlier.
I don’t think Stable32 does that, so I’d like to pose this challenge to anyone interested in serving the community: Assuming IEEE will make the source code available, can the approach I use be made an option? And yes, I know that the uncertainties of the deviation points get hard to compute, but such issues can be warned about in the documentation, if not addressed.
> --------------------------------------------
>
> Message: 1
> Date: Tue, 19 Jan 2021 08:49:13 -0500
> From: Bob kb8tq <kb8tq at n1k.org>
> To: Discussion of precise time and frequency measurement
> <time-nuts at lists.febo.com>
> Subject: Re: [time-nuts] Long Wave Radio-Frequency standard testing
> Message-ID: <30728187-DBFE-4110-A595-D1497B5F4074 at n1k.org>
> Content-Type: text/plain; charset=utf-8
>
> Hi
>
> The normal approach to filling a gap is to put in a point that is the average
> of the two adjacent points. The assumption is that this is a ?safe? value that
> will not blow up the result. That?s probably ok if it is done rarely. The risk is
> that you are running a filter process (averaging is a low pass filter).
>
> If you pull out a *lot* of outliers and replace them, you are doing a lot of filtering.
> Since you are measuring noise, filtering is very likely to improve the result.
> The question becomes - how representative is the result after a lot of this or
> that has been done?
>
> Obviously the answer to all this depends on what you are trying to do. If you
> are running a control loop and the output improves, that?s fine. If you are
> trying to provide an accurate measure of noise ?. maybe not so much :)
>
> Bob
>
>> On Jan 19, 2021, at 2:15 AM, Gilles Clement <clemgill at gmail.com> wrote:
>>
>> Hi,
>> Yes outliers removal creates gap in Stable32.
>> The ? fill ? function can fills gaps with interpolated values.
>> It does not change much the graphs, except in the low Tau area (see attached).
>> Do you know a discussion of impact of outliers removal ?
>> Gilles.
>>
>>
>>
>>> Le 18 janv. 2021 ? 22:06, Bob kb8tq <kb8tq at n1k.org> a ?crit :
>>>
>>> Hi
>>>
>>> As you throw away samples that are far off the mean, you reduce the sample
>>> rate ( or at least create gaps in the record). Dealing with that could be difficult.
>>>
>>> Bob
>>>
>>>> On Jan 18, 2021, at 1:33 PM, Gilles Clement <clemgill at gmail.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> Very cool !!!
>>>>>
>>>>> The red trace is obviously the one to focus on. Some sort of digital loop that
>>>>> only operates under the ?known good? conditions would seem to make sense.
>>>>>
>>>>> Thanks for sharing
>>>>>
>>>>> Bob
>>>>
>>>> Hi,
>>>> I tried something with the idea to consider night records fluctuations as ? outliers ? as compared to day records.
>>>> Indeed the 3 days record mean value is flat and the histogram quite gaussian.
>>>> So I processed the 3 days record (green trace) with Stable32?s ? Check Function ?,
>>>> while removing outliers with decreasing values of the Sigma Factor. The graph below shows the outcome.
>>>> The graph with Sigma=0.8 (blue trace) connects rather well with the 1Day record (red trace).
>>>> Would this be a workable approach ?
>>>> Best,
>>>> Gilles.
>>>>
>>>>
>>>>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 19 Jan 2021 13:37:53 -0500
> From: Dave Daniel <kc0wjn at gmail.com>
> To: Discussion of precise time and frequency measurement
> <time-nuts at lists.febo.com>
> Subject: Re: [time-nuts] Long Wave Radio-Frequency standard testing
> Message-ID: <2235BF02-8FFF-4000-85A5-119BD358AB19 at gmail.com>
> Content-Type: text/plain; charset=utf-8
>
> Or one can replace those values with zero. That eliminates them; averaging then proceeds without those values altering the most probable correct average.
>
> DaveD
>
>>
>>>
>>>> Le 18 janv. 2021 ? 22:06, Bob kb8tq <kb8tq at n1k.org> a ?crit :
>>>>
>>>> Hi
>>>>
>>>> As you throw away samples that are far off the mean, you reduce the sample
>>>> rate ( or at least create gaps in the record). Dealing with that could be difficult.
>>>>
>>>> Bob
>>>>
>>>>>> On Jan 18, 2021, at 1:33 PM, Gilles Clement <clemgill at gmail.com> wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Very cool !!!
>>>>>>
>>>>>> The red trace is obviously the one to focus on. Some sort of digital loop that
>>>>>> only operates under the ?known good? conditions would seem to make sense.
>>>>>>
>>>>>> Thanks for sharing
>>>>>>
>>>>>> Bob
>>>>>
>>>>> Hi,
>>>>> I tried something with the idea to consider night records fluctuations as ? outliers ? as compared to day records.
>>>>> Indeed the 3 days record mean value is flat and the histogram quite gaussian.
>>>>> So I processed the 3 days record (green trace) with Stable32?s ? Check Function ?,
>>>>> while removing outliers with decreasing values of the Sigma Factor. The graph below shows the outcome.
>>>>> The graph with Sigma=0.8 (blue trace) connects rather well with the 1Day record (red trace).
>>>>> Would this be a workable approach ?
>>>>> Best,
>>>>> Gilles.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> time-nuts mailing list -- time-nuts at lists.febo.com
>>>>> To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
>>>>> and follow the instructions there.
>>>>
>>>>
>>>> _______________________________________________
>>>> time-nuts mailing list -- time-nuts at lists.febo.com
>>>> To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
>>>> and follow the instructions there.
>>>
>>> _______________________________________________
>>> time-nuts mailing list -- time-nuts at lists.febo.com
>>> To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
>>> and follow the instructions there.
>>
>>
>
> Message: 5
> Date: Tue, 19 Jan 2021 16:27:35 -0500
> From: Bob kb8tq <kb8tq at n1k.org>
> To: Discussion of precise time and frequency measurement
> <time-nuts at lists.febo.com>
> Subject: Re: [time-nuts] Long Wave Radio-Frequency standard testing
> Message-ID: <601C2AE8-4750-49DC-801B-B1C6C8D228AB at n1k.org>
> Content-Type: text/plain; charset=utf-8
>
> Hi
>
> Assuming the goal is a normal ADEV or xDEV sort of calculation:
>
> If you replace the raw phase values with zero that can mess things up
>
> 0 seconds +20 ns
> 1 seconds +22 ns
> 2 seconds +23 ns
> 3 seconds +25 ns
> 4 seconds +27ns
> 5 seconds +29 ns
>
> If you ?loose? one of those 20 to 30 ns values and replace it with zero, you have significantly
> changed the data set.
>
> Even if you are looking at deltas, zero stuffing would be problematic with that
> (contrived) phase data set.
>
> 1 seconds +2
> 2 seconds +1
> 3 seconds +2
> 4 seconds +2
> 5 seconds +2
>
> If the objective is something like a PLL then ?hold at the last value? is the only practical
> answer to the question. You don?t *have* the next value and you need to stuff something
> into the control loop computation.
>
> Bob
>
>
>> On Jan 19, 2021, at 1:37 PM, Dave Daniel <kc0wjn at gmail.com> wrote:
>>
>> Or one can replace those values with zero. That eliminates them; averaging then proceeds without those values altering the most probable correct average.
>>
>> DaveD
>>
>>>>>
>>>>> _______________________________________________
>>>>> time-nuts mailing list -- time-nuts at lists.febo.com
>>>>> To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
>>>>> and follow the instructions there.
>>>>
>>>> _______________________________________________
>>>> time-nuts mailing list -- time-nuts at lists.febo.com
>>>> To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
>>>> and follow the instructions there.
>>>
>>>
>>> _______________________________________________
>>> time-nuts mailing list -- time-nuts at lists.febo.com
>>> To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
>>> and follow the instructions there.
>>
>> _______________________________________________
>> time-nuts mailing list -- time-nuts at lists.febo.com
>> To unsubscribe, go to http://lists.febo.com/mailman/listinfo/time-nuts_lists.febo.com
>> and follow the instructions there.
>
>
>
>
> ------------------------------
>
> Message: 6
> Date: Tue, 19 Jan 2021 17:19:30 -0500
> From: Dave Daniel <kc0wjn at gmail.com>
> To: Discussion of precise time and frequency measurement
> <time-nuts at lists.febo.com>
> Subject: Re: [time-nuts] Long Wave Radio-Frequency standard testing
> Message-ID: <DD608F22-023E-48FD-BD8E-821DA0A5D148 at gmail.com>
> Content-Type: text/plain; charset=utf-8
>
> Answers inline
>
>> On Jan 19, 2021, at 16:27, Bob kb8tq <kb8tq at n1k.org> wrote:
>>
>> Hi
>>
>> Assuming the goal is a normal ADEV or xDEV sort of calculation:
>>
>> If you replace the raw phase values with zero that can mess things up
>>
>> 0 seconds +20 ns
>> 1 seconds +22 ns
>> 2 seconds +23 ns
>> 3 seconds +25 ns
>> 4 seconds +27ns
>> 5 seconds +29 ns
>>
>> If you ?loose? one of those 20 to 30 ns values and replace it with zero, you have significantly
>> changed the data set.
>
> Ok
>
>>
>> Even if you are looking at deltas,
>
> Nope. It doesn?t work for deltas.
>
More information about the Time-nuts_lists.febo.com
mailing list