Brendon Jones's blog
Spent the week working on parallelising the docker build system and adding support for armhf. I now have a script that will spin up a docker instance for each flavour of Debian/Ubuntu, download and install build dependencies, build the appropriate packages and copy them back into the host system.
I got standalone cross-compilation of armhf packages working within a single instance, but appear to have missed something when adding it to the rest of the build scripts - something is looking in the wrong paths for the dependencies. In the mean time it works fine inside the emulator, though is a little slow.
Tidied up a couple of issues encountered while installing Wheezy packages for the amp-web server components - eventing schema is now created at install time rather than run time and permissions are set correctly so that events can be fetched by the web user. Fixed the AS name parsing code to always return something sensible, after discovering that our data source had messed up many of the AS name fields, so we were using incomplete data.
Updated some of the server install instructions to match changes to packages and dependencies. Fixed the rabbitmq installation instructions to use their new keys.
Integrated the access control list with the control socket processing code, so now all connections are checked for valid permissions (start test servers, run tests remotely). I've got slightly stuck on making sure that failures are reported back properly rather than just failing, as it gets convoluted when a remotely started test needs to start a server elsewhere but lacks the server permissions.
Used the new docker build system to build Wheezy packages for the AMP server components. Haven't got it working in parallel, but it is still very useful to build in a clean environment to make sure dependencies are correctly recorded. Updated skeptic to use the new packages and ran into some issues (mostly permissions and non-packaged dependency related) that weren't encountered during previous rollouts on Jessie or test runs in a VM.
Spent some time finishing up the access control for amplet control connections and writing some basic unit tests. The lists all perform as expected, with more specific rules overriding less specific ones. The next step will be to integrate it into the control socket processing so that it gets checked when clients connect and issue commands.
Had a look at using docker to build Debian packages with some help from Brad. Got it working with the amplet2-client packages, looks like it could be pretty useful, especially making sure it's a clean build and only using explicitly defined dependencies. Had a bit of a look at spawning a bunch of shells and performing parallel builds across all Debian/Ubuntu flavours at once rather than doing it in serial. Got it mostly working but I'm not happy with the way I get the built packages on completion - how to trigger it and where to place them.
Spent some time writing some basic documentation about installing and configuring the server software to collect and display AMP data. Most aspects are covered at least briefly now.
Updated the ampweb packages to disable the default test user and to be slightly smarter about dealing with user configuration in general - the package now treats them as actual configuration files and so won't clobber them on install. Rebuilt the packages and pushed them out to the lamp server.
Started work on implementing some simple access control for the amplet client control interface, to limit who is allowed to run test servers or perform other control functions. This is the first step before I implement the ability to push schedules directly to clients.
Short week due to holidays and illness.
Made a few more fixes to the ampweb packages that were installed last week to help fix some issues we noticed during installation, including updating default configuration files to have more sensible values.
Spent some time updating documentation for the amplet client and writing man pages for some binaries that were missing them.
Spent most of the week finishing up packaging all the server-side components for AMP. Had to add a couple of patches as part of the package build process to set logging and pidfile locations, which I should go back and try to fix in a more correct manner. Spent some time installing and uninstalling packages to make sure postinst scripts were working properly to create users, databases, etc. Updated the Lightwire AMP server with the new packages, which went fairly smoothly but did have some issues with missing python dependencies (not in Debian, or too old) that were only required in some code paths.
Also updated the New Zealand AMP mesh to the newest client version. Went pretty smoothly which was nice.
Fixed up the rabbit queues/bindings that I had broken when I accidentally used a queue resource in place of a routing key when exploring the erlang commands. Figured out the syntax I needed to properly create the binding to an existing queue and now have all the configuration sorted.
Spent the rest of the week packaging the remaining software that we deploy on the AMP servers - ampweb, ampy, nntsc, ampsave, netevmon. Quite a few of them have progressed a lot since they were last packaged, so lots of dependencies needed to be updated, users created, databases created and populated, etc. Almost have all the packages built and ready for a test deployment now.
Spent some time working on issues a new user had installing the amplet client on a new Ubuntu machine. The Jessie packages almost work fine, though there seems to be a change in behaviour (or a bug) in the libcurl-gnutls library which prevents curl from working with any of our SSL sites. In the end after a lot of chasing things around, the easiest workaround seems to be to build against the OpenSSL flavour of libcurl instead.
Got the amplet client building on Centos again so that I could update a server used as an endpoint for cooperative tests. It is now running throughput and udpstream tests to get sample data for prophet, as well as testing out the new client build.
Started to look over all the old Debian packaging I had done for the server side software to get it all working for the current versions. Updated packages for the simple parts (pywandevent, libnntsc, amppki). Got stuck writing erlang to declare queues, exchanges and bindings on the server because the default tools can't do that (and I don't want to install extra tools). Currently have declarations working fine but in creating the bindings I've created a broken queue that breaks the web interface.
Spent most of the week working on installing the server-side components of AMP, which took a lot longer than I thought it would. Ran into issues with deploying ampweb when it's not at the root of the website - a lot of URLs were absolute and URL parsing was expecting a particular layout or number of elements which was no longer the case. Fixed all the obvious ones, but more were still showing up a few days later and have also been fixed.
Found and fixed a few small edge cases in recent ampweb/ampy features that hadn't been tested with the sort of data I was wanting to look at. Also had issues with trying to test the changes I had made to make sure they worked elsewhere, as the influx database on prophet was misbehaving and making it difficult to fetch data in some circumstances.
Spent some time tidying up control messages and configuration when scheduling tests that require cooperation from the server. As part of the previous changes the port number was no longer being sent to tests, which meant it could only operate using the default port - this is now fixed and works for both scheduled and standalone tests. Also fixed up some parameter parsing when running standalone tests where empty parameter lists were not being created properly.
Wrote some basic unit tests for the udpstream test and it's control messages. Fixed a possible memory leak when failing to send udpstream packets. Made sure documentation and protobuf files agreed on default values of test parameters.
Started to install the server-side components of AMP on another machine for a test deployment so that I can use the documentation I write as I go to help build/update the packaging for the most recent versions.