User login

Richard Sanger's blog

24

Jun

2014

Added a basic test for the entire parallel libtrace system, based upon the existing test-format code. This tests most of the basic functions including pausing and restarting a trace, however is limited to only the file based formats.

I setup a BSD VM at home and spent some time tidying up the code to ensure it still compiled. Merged in the latest upstream fixes, and spent a while finding a build bug where visibility for some 'exported' functions were hidden.

17

Jun

2014

I investigated implementation of the BPF filters in regards to thread safety. It turns out that what I originally suspected is true, the compilation is not thread safe but the running of the filter is. The possible problem was dependent on where the scratch memory with the filter was stored, it appears that when running the filter this is allocated on the stack within the pcap code and any of the kernel implementations. I suspect that the LLVM JIT does the same, however since this is disabled by default I'm not too concerned for now. The BPF filter compilation on the other hand uses lots of globals in libpcap which is not thread safe, this case I have already locked against previously.

Hooked up configuration options for the tick packets allowing them to be turned on and off with a specified interval to the nearest 1msec resolution.

04

Jun

2014

I gave my practice presentation to the 520 class last Wednesday, it went well.

I finished updating tracertstats to use the tick packets and removed the previous system of holding temporary results. This has also removed a lot of duplicated code which is nice.

I've started on the mid-term report which is due this Friday, I'm hoping to be able to reuse much of the introductory content in the final report.

28

May

2014

I've been working on implementing empty tick messages, these can be produced every X seconds to assist programs that want to report results in tracetime (wall time).

I had two main choices here to produce these messages, either start a new thread to send these messages or use timer_create() to enter a signal handler. I've opted for a separate thread due to it not being obvious what thread is best to put said signal handler on, a separate thread will find the least busy core and cannot block other threads out of shared resources by holding a lock etc.

Later this will also be easier to customise the timers behaviour such as accounting for skew in the format.

I've also created/updated slides for my honours practice talk.

It's been a busy week with other assignments, so this is not fully integrated with tracertstats yet and needs a configuration option added, however it is functioning.

20

May

2014

Finished up with the pausing code, it now is working for files and will not loose any data and similarly functions as expected for various different formats, those with native multi-threaded support and those that don't. Packets waiting in the result queues are also being copied so that they wont be pointing to bad memory if pausing a zero-copy format actually closes it such as ring: and invalidates the memory.

Implemented restarting traces for single threaded traces (which I had neglected so far), so a full pause and start cycle is possible. Fixed some other minor bugs and tidy ups such as zeroing structures to play nicely with valgrind.

13

May

2014

Working through pausing code to ensure all packets are cleaned up correctly and that packets are not lost in the case of a file etc. I've implemented pausing for the case a hasher thread is in use, this ensures all queued packets are emptied to the perpkt threads for processing. This is the favored behaviour if reading from a file because it means that packets are not lost. Added 'state' to the trace object to help ensure pause cannot be called twice, or start when a pause is still in progress etc.

I still have a couple of things to complete the pause functionality nicely, the main thing that's left to finish off is to make copies of packets in the reducer/reporter queues if the buffers belong to the trace format, since pausing is often implemented by the underlying format as closing the trace and invalidating these packets.
I might also look into the data structures used here since FIFO operation is pretty much the only way this will be used so optimisation would be good.
This pause code will also be called first before stopping a trace say from a SIGINT.

Found and fixed a couple of other bugs as I've been working on this.

15

Apr

2014

Put together some tests for the data structures I've been using within the parallel libtrace implementation. Tidied up a lot of compiler warnings.

Added a global flag to the library to track if the parallel API is being used to provide backwards comparability with existing code. The only place where this is a problem is destroying packets after the trace has already been destroyed. Now the existing tests pass again.

01

Apr

2014

Finally starting to get underway with some coding again now that the paper work is out of the way.

I created a forked copy of Libtrace on Github to keep my work separate it is available here https://github.com/rsanger/libtrace/. I worked through my existing code from summer to ensure that any recent patches are applied properly and some tidy ups. I'm hoping making this public will also force me to keep things a bit tidier.
This coming week I will be focusing on writing some tests for the data structures and the parallel routines in general, as well as a test to measure performance.

24

Mar

2014

Worked on my proposal this week, along with some more research review.
I found quite an interesting paper 'libpcap-MTA General Purpose Packet Capture Library with Multi-Thread' which was in Chinese where they created a parallel version of libpcap. Many of the issues that where discussed confirmed what I had found out over summer. Made myself a github account now that the libtrace repository and downloaded a copy of the repository.

17

Mar

2014

Started to work (again) on my honours project which is a continuation of my summer research work. Submitted a blurb about my project and re-read through a couple of the academic papers I had originally looked at to revise where to take the project next.