[time-nuts] sample data for exp+linear fit

[Magnus Danielson]

Hi Jim,

On 10/05/2013 07:44 PM, Jim Lux wrote:
> Here's some sample raw data for those interested in what I'm working
> with.  Thanks a bunch for all the suggestions.
>
> The overall goal is to take this data, remove the DC bias, linear
> trend, and exponential transient baseline; then cleverly excise (or
> exclude from processing) areas with obvious artifacts (see Figure 15,
> IQ pair 4) for an example of an artifact.
>
> These data plots have had the mean removed, but no detrending done. 
> The raw data does NOT have the mean removed.
>
> As with many real software development things, the basic algorithms
> are simple, it's handling real data with real glitches and anomalies
> that is important.  So when I collect the data, before I load it into
> a more lengthy processing step, I want to do a quick evaluation and
> see if the overall data take was bad, or maybe I can process just part
> of the data (I'd rather have gaps than bad values).
>
>
> http://www.luxfamily.com/jimlux/Figure%209.pdf
> http://www.luxfamily.com/jimlux/Figure%2011.pdf
> http://www.luxfamily.com/jimlux/Figure%2013.pdf
> http://www.luxfamily.com/jimlux/Figure%2015.pdf
>
> Raw data for 4 channels
> http://www.luxfamily.com/jimlux/data.txt
I loaded I1 and Q1 into TimeLab, and removing the quadratic as well as
dropping the first samples, when comparing to the un-processed data
there is only minor changes. Turns out that there is noise and a slope
down as tau rages, and there isn't much drift effect on the ADEV. MADEV
is much cleaner naturally. There is a sine additive noise, which best is
removed by doing a notch filter.

So, I would do this:

1) Drop first 10 samples
2) Notch filter for removing sine noise
3) Estimate quadratic with least square, and remove
4) MADEV

Notching out sine noise is not something I've seen in the literature,
but I have tried it and it works really well as long as one care about
unity gain. It will also make the least square approximation not being
fooled by the noise. It will also not obscure the noise processes that
ADEV/MADEV do. The TDEV plot is probably giving you most info, and it's
only about tau 300 s (assuming sample every s, so scale accordingly)
where the drift processing kicks in and helps to sort things out.

Cheers,
Magnus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IQTDEV.png
Type: image/png
Size: 64726 bytes
Desc: not available
URL: <http://febo.com/pipermail/time-nuts_lists.febo.com/attachments/20131005/25bb3696/attachment.png>