[amsat-bb] Fox Telemetry Server Outage

[Chris Thompson]

Many may have noticed that amsat.org was offline for about 20 hours
yesterday.  We are investigating the root cause.  Something impacted the
TCP stack on the server.  That caused the telemetry server to spin out of
control. In that state it did two things, neither of them good and neither
of them in the design spec.

Firstly, the telemetry server spewed error messages to the log at such a
rate that it impacted the website and other systems on amsat.org.  A fix
will be implemented so this does not happen again.

Secondly, the telemetry server accepted connections from FoxTelem,
confirmed it had received the data and then failed to process it. This
should not be possible given the error handling logic, so we are trying to
work out how to prevent in the future.

Those of you who keep track of your uploads, will notice that some records
were not updated yesterday, even though they were sent.  I'm going to
release a tool to allow re-upload of the missing day.  Please don't purge
your FOXDB files if you have data.

Let me know if you have questions or suggestions
73

Chris

-- 
Chris E. Thompson
chrisethompson at gmail.com
g0kla at arrl.net