[time-nuts] St Veran gravity red shift mission - logging

[Magnus Danielson]

Dear all,

I have written multiple times about the mission, but I thought I share a 
separate view this time, and I think it may be equally interesting. This 
time I would like to bring up the topic of logging. You can see traces 
in various posts from before, but maybe I can contribute a somewhat more 
comprehensive picture.

I realized early that there was many things to log for this mission. For 
starters, I wanted to make sure I was logging the 5071A cesium clocks 
state, to see how it was behaving. Much of the state is control loops 
which compensate for environmental effects. There is also a temperature 
reading in the clock. Just logging these as we expose the clock for the 
elements of being in the boot of my car, moved to another car, moved 
into the lab of the observatory and back will improve confidence that 
the clock was happy with life in it's own reference, which is an assumed 
property.

I also wanted to log the power-system since the charing and discharging 
of batteries is critical, and add additional environmental sensors.

I had from before developed a tool to log my EFOS-B active hydrogen 
maser, using a python script that then post data into an InfluxDB 
database and at the back-end of that pulls data out into Grafana for 
graphs. I learned immensly from doing this, so I realized that I should 
attempt to extend that to incorporate these new loggings.

In essence I use a Raspberry Pi 4B with an external harddisk (where the 
InfluxDB files lives). I bought a USB hub with separate power supply and 
7 locations.

I used two DC/DC converters to make 24 V of the LOAD PANEL convert over 
to the 5V needed. Turns out I had power limit issues causing problem, so 
I had to move the Raspberry Pi to use a wallwart sitting of the 
generated 230 VAC.

For fun I found a Raspberry Pi hat called Environ+ which has a BME280 
sensor, which had been suggested by others, so I bought that and 
installed. Further, I have a DPM7885 pressure sensor laying around, so 
hooking that one to the 12 V bus and then using a USB-Serial adaptor got 
that started.

So, first I added the 5071A logging, using a USB-Serial adaptor. Coding 
that up was fairly quick. As always, some work is needed to get it to 
sync up cleanly. Then I added the DPM7885 with a similar dance. That was 
quite easy. Each of these had their own temperature sensor.

These where in place before leaving home for my summerhouse in southern 
Sweden, a 600 km drive.

Later I had the time to add the BME280 sensor thought he Pyhton library 
code provided. Picking up the example code turned out to be a fairly 
quick adaptation. With that I added another temperature sensor, another 
pressure sensor and then humidity sensor. The BME280 has compensation 
factors in it, and I validated that the code actually did use the 
validation schemes. I could however see remaining temperature dependence 
that was uncompensated, because, that is what you see in your graphs if 
you look and is curious (who? me?).

In addition I also wanted to add the TADR TICC, and that was fairly 
quick integration. I did however end up having problem that the PPS from 
the 5071A only cause a few triggers on the TICC ChB, so I could not 
really use it as intended.

That is how far I got while at my summerhouse, then I had to leave for a 
1100 km drive to south of Belgium, the next day was a good 850 km drive 
to just outside Grenoble in south of France directly followed with the 
remaining 220 km drive to Saint Veran village, in just three days.

I was not able to add the Victron VE Direct logging until up on the 
mountain, after restoring the setup. Again I used the Python library 
available and it's example code and was fairly quickly able to integrate 
it. Naturally I made an off by 1 on the digits, so my current readings 
got 10 times larger than correct, but I fixed that.

I had intended to let my Mosaic-T eject NMEA messages and parse that and 
ingest into the database, to get time, position and height. That did not 
get done during the mission.

The logging system was riddled by a number of power issues, much of all 
can be blamed on having too little time to do things properly, and to 
little time to test. Also, with power problems came reboot issues, such 
that needing to have a screen present during booting as well as not 
automatically start logging. These things I was not able to fix during 
the mission, but I have fixed that since, since it bugged me and I want 
to be prepared for the future, both my own lab and any similar missions. 
It's only when you attempt to do stuff you learn how difficult it is.

The fact that the screen was starting to fail ended up being a problem 
too. I have since made the Raspberry Pi able to boot safely without screen.

Now, since I have three USB devices showing up as /dev/ttyUSB0, 
/dev/ttyUSB1 and /dev/ttyUSB2 in the Raspbian/Debian Linux distro, and I 
will not know which order they show up, I needed to provide a mapping of 
known order. The mapping is done by a system called udev, and you can 
add your own rules. This was well known, and I thought I had resolved 
it. Turns out no, so I had to redo it. Many of the documentation and 
examples is outdated and does not work with modern code. However, much 
of the basic hints is correct, it's more details of encoding. Once I 
sorted out that, I was able to make the devices show up as /dev/ttyUSB4, 
/dev/ttyUSB5 and /dev/ttyUSB6 but with a fixed mapping. This simplifies 
much of the manual hazzle of figuring out which ended up being which.

Then, it was time to setup automatic logging. I decided to take on the 
modern solution of systemd, and using the example it took me only a few 
attempts to get it shaped up to work right. I have now built 3 example 
logging using systemd to do my DPM7885, BME280 and VEDirect loggings. 
The 5071A used was then already sitting in the basement of Observatoir 
Royal de Belgique so I did not bother to do that one, but using the 
examples, it is trivial to do that one with the examples I have, so when 
needed I can do that.

I chose to log data every 10 s, except for the VE Direct which spews out 
data every second, so just consume that and push it onto the database 
was easiest.

In Grafana, setting up plots was done, and it was fairly 
straight-forward. I aim to cleanup my plot setup so that it can be 
exportable and important into another setup.

The code for all this you can find at my GitHub repository 
https://github.com/sa0mad/masermon

I aim to add more stuff over time, and if people find it useful, more 
things can be integrated. I aim to add Grafana setups once I have them 
in suitable form.

I've found that integration of many different systems with a dash of 
Pyhton and then dump datastreams into InfluxDB have been a great 
strategy. I use UTC time-stamps for everything and I use NTP to get the 
Raspberry Pi have the right time. I did not bother to integrate the GNSS 
Receivers time, but that would be a next logical step, especially since 
Mosaic-T has both NTP and PTP support, but since it ended up having 
issues, I did not bother.

Graphing data was a lot of fun, and you can see on the pressure the 
decent into Saint Véran village but also the drive over the Col d'Izoard 
at 2360 m.  You can see temperature swings.

Putting up a difference between the pressure sensors, you can see how 
they have especially difference scale sensitivity but also temperature 
sensitivity.

Plotting the temperatures it was easy to see the different elevation of 
temperature due to where they where mounted, but also how processing on 
Raspberry Pi made its hat temperature vary.

I have not yet had the time to analyze all the data, and already started 
doing a tool to extract data and process least square fit. That data, 
the TICC, turns out to be quite meaningless thought. It seems like the 
PPS was returned through the receiver rather than being separate, so I 
was measuring extremely small shift in delays.

Among the clean-ups will also checking the function of the TICCs etc.

I must say that the TADR TICC is almost ideal for a setup like this, 
with low power and the performance it gives. Details on how it starts up 
and how I can control it and sync up needs more work, and I just had not 
had time to make that bullet proof. For logging I need to have a fixed 
way for a script to establish contact, set it up exactly as needed for 
that measurement and then have robust output. I ended up loosing data 
because I had not been able to make the TICC logging robust from various 
start-up scenarios. That issue occurred in various degrees with all 
equipment, and just takes time to sort out. Time was not plentiful 
before this project. Testing and testing again would be needed to know 
how to get things robust and in place. I have learned immensly about 
what can mess up, and will test a lot now.

So, a very intense vacation, and I hope many of the learnings and 
associated code may inspire and come to use.

Oh, yes. Many of you may wonder "What about the result then?". We keep 
doing reference measurement and will be post-processing data when people 
are back from vacation etc. We all have to be patient. I try to share a 
lot of other things.

Cheers,
Magnus