Christopher Lorier's blog
I changed rfserver over to a proper MPLS approach, but am having trouble with the ovs recirculation branch. My packets are hitting one table then disappearing before they reach the next..
Hit a little hitch with implementing polling packet counts. The poller works fine---aside from breaking forwarding within the fabric.
The problem is that I'm only able to push one layer of label and I need to edit the label twice. I could push metadata instead, and then push the complete label in a single action, but that would scale quadratically in the number of switches. I also thought about using the vlan pcp field as a bonus tag, but that seemed a little bit messy. Plus that's only 3 bits, so it limits me to 8 switches.
So instead I'm gonna use a development branch of ovs with MPLS support. It's doing it the proper way, but for the time being it is pretty much completely unsupported by anything else.
So my rebase last monday turned out to be more catastrophic than I had thought. Trying to move on and unearthing the hidden consequences took up most of last week. But I have now moved on to working on polling paths. I have built the poller itself and am currently adding the messages to routeflow it needs. After that I just have to plug it in to rfserver and I can take it for a spin.
I have completed all the work to get the arbitrary_topology working tidily. I tried to rebase and tidy it all up this morning but it wasnt a great success.. It has mostly been compressed into one giant commit now. I will look at rebasing that nicely so I can put it online, but my motivation to continue with that has taken a bit of a hit.
The next thing is to work out how I can implement the path end polling with the current state of ovs. IE no layered MPLS.
I think it should be achievable with masked VLan tags, which will probably turn out to be a much simpler approach. If a little hacky.
So, I have been completely revising RFServer at the moment. Getting rid of most of the remnants of mongodb to hopefully have something a lot simpler to deal with.
This is mostly because I noticed about 4 thousand or so bugs with the code in my 591 to deal with network partitions. Hopefully, this will help reduce that number a little..
I added multiple table support to RouteFlow, and am now trying to add my 591 work on top of that, but it is taking longer than I expected.
Having multiple tables does simplify the structure of things and fixes most of the interface issues I had with the older version of this code, but in spite of this I am having a lot of problems getting this to work.
So I did some tests with how counters are affected by changing flows in open vswitch. There is a bug in earlier versions of ovs that would have ruined everything, but fortunately it's fixed in the latest version. It also seems like every now and then one packet will not get counted if you change flows. This seems to happen less than once in a million packets though, so I cant imagine it being a big deal.
Other than that I have been adding the changes from my 591 project onto the latest release of vandervecken and working out the best approach to adding support for multiple tables to RFServer.
After a week spent mostly breaking things and panicking that what I was working on was not working, I managed to figure out how to synchronise my openflow counters.
So before pushing the label for the lsp through the fabric, I can push another label. Then when the egress node pops the first label it can send the packet to another table with threads matching the internal label. This is just to count the packets arriving.
By periodically changing the internal label I can get synchronised counts of packets arriving at each end during that period.
I set up an snmp poller to look at consistency of packet counts at two ends of a link and it looks fairly promising. I'm currently setting up an openflow system for some more extensive testing.
I've also been looking at the latest versions of RouteFlow as well to get an idea of what has changed since I last worked on it. And investigating what could be done to include multiple table support and other things needed for what I am doing.
I've been writing the proposal and thinking about how things will scale.
To poll both ends of a path I can add an extra flow on each switch specifying the ingress switch, so that when a packet leaves the system it is counted with the other packets that entered the fabric at that switch. This will require tables and stacked mpls labels (3 layers of mpls), though it could probably be made to work with two.
This way I can poll both ends of a path, but inbetween I am aggregating paths, since the alternative means I have a number of flows to create paths on each switch that is quadratic in the number of switches in the fabric. This is going to be a complication for accurately locating problems just by polling counters.