[time-nuts] How best to exchange Large files?

Hal Murray hmurray at megapathdsl.net
Wed Feb 22 00:43:56 UTC 2012


albertson.chris at gmail.com said:
> What a complex setup!  Can't you user some of the Linux server's disk space
> and write the files directly to that disk?   Seems that would be much
> better. 

[snip of neat brick story]

> I've used this story in data processing system designs when I see people
> moving data around.     I say "think, where is the final place you want your
> data.  Then just put it there to begin with and be done with it."

> So, just open some kind of connection and put the data where it goes. Even a
> little Arduino has more then enough power for that. 

The cost of that approach is another level of complexity at the system 
administration level.  You have to keep both systems running in order to 
collect data.

If I'm collecting interesting data, I prefer to keep the system simple.  I 
have one running on a UPS.  I get a lot longer holdover by collecting data on 
a stand alone small box than I would if the UPS had to run the "final" system 
too.

I set things up to break log files into day sized chunks.  They get names 
like foo.20120221, with foo being a link to the current one.  (Scheme copied 
from ntpd.)  Most of the time, they are a reasonable size.  Then I use 
standard unix utilities to copy them over to where I process them.  I think 
rsync does most of the work.  It's buried in a script.

One advantage of an explicit copy step is you get a layer of redundancy.  
That may not be worth anything if you already have a good backup plan.





-- 
These are my opinions, not necessarily my employer's.  I hate spam.







More information about the Time-nuts_lists.febo.com mailing list