Brendon Jones's blog
Short week due to Easter holidays.
Went through the protocol buffer descriptions for all of the AMP test formats and wrote some short documentation about the fields, and some sample code to fetch, unpack and understand test result messages.
While working on the documentation I found a few instances of test options that were inconsistent or didn't quite behave correctly. In particular I made the TCP ping size parameter behave the same as the ICMP one (total packet size rather than payload size), and made sure that setting just the UDP payload size in the DNS test would correctly add an EDNS header.
Updated some of the signals used with the amplet client to provide
better management - as well as being able to reload configuration from
disk, it can now force a refetch of remote schedule files with a SIGUSR2.
Also made sure that all children (tests, servers, etc) have their
signals unblocked and the signal handler restored to the default.
Libwandevent sets all these in the main process, which was being
propagated to the children and causing some unexpected behaviour. The
init scripts now try to kill the entire process group of the amplet
client, which means children should now get the signal too.
Renamed server processes in ps so that it was obvious what task they
Refactored some more of the repeated server code out of the
udpstream/throughput tests so they are now a lot cleaner. Moved some of
the test server control message code around so that it was grouped
together in a sensible place.
Spent some time updating unit tests to work properly with the new
watchdog and control API. Improved checks to make sure that only valid
control messages are being parsed. Other small fixes to make sure that
errors are caught and reported properly.
Started refactoring the test control connections to use an SSL BIO so
that exactly the same code paths can be used to read and write control
messages whether SSL is in use (amplet, standalone tests) or not
(standalone tests), which has removed/simplified a lot of code. Also
figured out how to properly do non-blocking IO when the BIO functions
behave differently to normal read/write.
Went with Shane to visit Lightwire on Thursday and had a discussion
about how we can make event detection, measurements, graphs etc work
better for them.
Finished adding the ability to set DSCP bits for all the amplet tests
individually as well as globally. Slightly tidied up the way the global
options are turned into individual test options now that there are a few
more of them.
Tidied up the management connections to try to reuse the existing SSL
connection that started the server, rather than always expecting a
separate connection (as is sometimes the case when run standalone). As
part of this, added SSL support to the standalone tests, so now they can
be run standalone with/without SSL, or using it to connect to a normal
Reworked the way watchdogs worked to make sure they will properly
monitor new server threads, or remotely scheduled tests. The central
watchdog management has now been replaced by a timer inside each
server/test process that will ensure the test completes on time.
Decided to start doing systemd scripts properly and wrote a service file
for the amplet client. Also slightly tweaked the debhelper scripts that
are run to make sure the client doesn't start without configuration and
end up reporting errors to systemd. Had to officially split the Debian
directories now for Wheezy and Jessie as they are starting to diverge
Started work on adding the ability to set the differentiated services
bits in the IP header for all of the AMP tests. This can be set at a
global level, or on a per test basis. So far only the icmp test will
obey the setting, I'll update the rest of the tests next week.
Spent some time trying to remove an unnecessary extra control connection
for tests involving servers started by a remote amplet client. It looks
like I should be able to reuse the connection used to start the server
as the ongoing control channel, but I'm not quite sure how to make this
work best with standalone tests (that expect the server to already be
running, and don't currently encrypt anything). I should be able to tell
if I have a secure control connection or not and take the appropriate
actions, but a bit more planning is required.
Finished moving the main control socket to use protocol buffers to start
test servers. Updated the control socket to accept multiple message
types and added the ability to run a one-off test remotely using the
control port on an amplet client. This needed a few more smarts to
decide where the results should be sent - to the broker, to a remote
user, or to stdout and to print them as appropriate.
Fixed a bug in the HTTP test where it couldn't match a URL with a
fetched object in the broken situation where the path was outside the
root (e.g. src="../images/foo.png" when already at the top level).
Updated the standalone tests to accept SSL options so that they can be
run targeting existing amplet2-clients, using the control socket to
start a test server.
Finished removing the test specific options from the main test
management protocol. Each test now deals with it's own options (if any)
that are embedded in the top level protocol buffer message.
Refactored the main loops of the throughput and udpstream tests to be a
lot more readable and to make it obvious what is going on for each of
the message types.
Lots of small fixes for things in the udpstream test - making sure
packet contents (timestamps etc) are sensible for both 32 and 64 bit
architectures, median/percentiles correctly calculated with small
numbers of packet delay variances.
Started work on moving the main control socket (used to start the test
servers, and soon, to run on-demand tests) to using protocol buffers.
Started to remove the test specific options that had crept into the
generic control message definitions. Unfortunately the protobuf-c
library doesn't appear to do extensions properly yet, so I've had to
work around that to embed test specific options inside the top level
Throughput test now uses protocol buffers for all of the messages
involved in arranging and reporting the test. This is all achieved
through the same functions used by the udpstream test, which should now
be generic enough that other tests requiring custom servers can be written.
Wrote a first pass at the save function for the udpstream test, which
may need some modification once database schemas for storing the results
Updated the UDP stream test to report on the different periods of lost
and received packets during the test, to show how any lost packets were
distributed (bursty, random, etc).
Updated the messaging to now include result packets to retrieve results
from the remote endpoint. These are combined with the local result set
so that results are reported for both test directions.
Lots of small fixes to make sure that the right things are happening -
packet sizes are correctly calculated, sockets are closed appropriately,
memory is tidied up.
Finished moving all the control connections to protocol buffers,
including sending a full set of test options to the server when the test
The udpstream test now calculates packet delay variation between
consecutive packets and reports summary statistics across the whole
stream. The test schedule (directions, sizes, etc) can now be controlled
properly on the client, which will inform the server of what it needs to
do and when.