User login

Search Projects

Project Members

Network Flow Visualisation

With Cisco network equipment so ubiquitous, I will utilise Netflow to collect and store traffic information and present it to users in a web-based application. Users will be able to get such information as where their data is being sent, from where is it being received, how much data have they sent/received within a variable time period and which devices are connected to the network. It will be a dashboard application with various graphs, gauges etc. where users can filter the information displayed.

24

Aug

2015

I have created a script for generating some random data based on the data supplied by Brad. This is for development purposes so I can implement certain features where otherwise I would be unable. I have added an `application` field (containing HTTP, DNS etc) and a `flows` field. I will also add some IP addresses of some common applications (youtube, gmail) to get the data looking like it is from a router with a default gateway to the Internet. Controlling the database also allows me to decrease the amount of entries in the database to improve the performance of the application during the presentations.

17

Aug

2015

Made good progress with the application this week. Left to go are location statistics, application statistics, flows count and IP-based differentiation. I will also add an 'interval' fields to my configuration file so the user can decide how long to keep flows for.

Planning to start user evaluations next week ready for the presentations.

10

Aug

2015

Brad managed to supply me with some IPFIX data. So now the only thing that I am missing is Application/layer 4 information. I may just make some up for development and demonstration purposes.

The application currently displays the top downloaders, uploaders, and top domains. Each device can be filtered to discover the specific details for that device. Next I will add a comparison of bytes and flows, which would be a comparison of a user's exchanged data and the number of places they communicated with. I will also add some (made up...) application statistics.

03

Aug

2015

Given the lack of support for Netflow v9 and IPFIX, I have decided to make my application compatible with Netflow v5, v9 and IPFIX. To do this, I simply check the database for MAC addresses. If there are none, then I resort to using source IP addresses to differentiate between MAC addresses. As a result, the device-specific info is not as accurate compared to using MAC addresses. Concerning the user interface, I am gradually adding new features week by week.

27

Jul

2015

The application can now sort by device MAC, direction, ports and time (monthly, daily and hourly). It displays the usage, protocols and usage timeline. Next will be to create visuals to give an overview of all the devices impact on the network, and give the user the option to change from graphs to tables in order to get more detailed statistics.

I am looking at changing my collector (yet again) back to Netflow since I can see in my application that the amount of data uploaded is about 4 times that of downloaded data. This is because of sFlow sampling every 1/1024 packets on each interface i.e. 1 uplink interface vs 24 local interfaces. Netflow version 9 isn't possible given the hardware available so I looked into software which could export Netflow version 9 for me, and mirror the desired interfaces to the software. I found a program called softflowd which does this, but for some reason the MAC addresses are always zero. I have tested it on multiple machines with different collectors. I have contacted the creator of softflowd to see why the captured MAC addresses are always zero, and if it wouldn't be too much work for me to enable it to do so. Other than softflowd, there are no open source probes which can export Netflow version 9.

If this can't be done, I'll switch to using IP addresses to identify the local devices. While this won't be as useful due to dynamic IP's, it will make the application a lot more flexible.

13

Jul

2015

I made good progress with the application. I have set up most of the back end, particularly the values to be used to query the database. I can request flow information for the network as a whole or on a per-device basis. I have also added the ability to assign a name to a MAC address. I was thinking of using the hostnames of the hosts but it would be kind of hacky given only the MAC address, plus reverse-DNS would have to be set up on the local network.

Although sFlow was useful since it supported MAC addresses, the sampling isn't ideal for getting an accurate picture of the devices behaviour on the network. I found a program called softflowd which listens on an interface and is capable of exporting Netflow version 9. It looks like Netflow V9 is going to be the only protocol that can be used with my application since it supports everything I require, in particular MAC addresses, direction information and application information. Currently I inspect the interface index to determine direction in my parser script, which means that the SNMP must be configured to assign these values which isn't likely on a home network. I hope to get port mirroring set up on a switch and use softflowd to construct and export Netflow V9 packets to my collector. This won't affect the application itself because the database schema will be the same.

15

Jun

2015

Switched from using Netflow to sFlow since it turns out it is the most convenient way to get the information I need with the equipment available. Brad gave me some info on the switch which is exporting me traffic so I can differentiate between incoming and outgoing flows. I am having to manually check the interfaces in my parser program to get direction information. This isn't the most flexible way of doing it but the versions of Netflow and sFlow that are available do not support direction information and only handle incoming packets, not outgoing ones. I plan on making a configuration file which will contain the interface on which packets are sent to the Internet for the network on which my application is being installed. My parser program will use this to ascertain which flows are outgoing.

Currently I maintain 2 databases; one for ingress flows and one for egress. This is a result of me having to coordinate the individual flows which sFlow exports for each ingress interface. To reduce the amount of entries in my databases, I will only collect information from a couple of incoming interfaces on the local network. I will listen to all incoming packets on the uplink interface to the Internet.

Shane said altering lpicollector to export MAC addresses would be too much trouble so I skipped that idea. I still plan on using it later on to get application layer information and associate them with the flow exports produced by sFlow.

This week I will do as much of the application as possible with the data I have.

26

May

2015

To get my application using an open-source collector instead of nProbe, I had the idea of writing a Python program to parse the output files by nfcapd. nfcapd has the option to call a program when a new file becomes available, so I call nfdump and read all flows from the new file and output them as comma separated strings. I then pipe this output to my Python program so save to a database. This solution will also let me control the size of the database as well as deleting flows that are older than a certain date.

18

May

2015

Went to see Brad on friday about getting nProbe installed somewhere so I can collect some traffic information. Had to wait for a reply from nTop with the license to activate it. Saw Brad again this morning and got it activated, so now I can experiment with nProbe and carry on with the application development.
Unfortunately the only open source collector I could find that supports databases only supports NetFlow version 5, but I want to target IPFIX as it is standard. I will look further into lpicollector since it is open source, and see if there is a practical way to get application information from each flow.

17

May

2015

This week and last I implemented a few basic filters, configured the time series graph scales and added a table displaying the most prominent flows and the associated device.
Turns out pcap doesn't store direction information so I couldn't implement the ingress/egress filter. pcap-ng supports direction though Libtrace doesn't support this format yet. Should be easy enough to implement when I have the information available.
Recently I have been planning my presentation which is this Wednesday.
Plan on seeing Brad either this week or next week to get NetFlow configured so I can start working with real data.

04

May

2015

This week and last I implemented a few basic filters, configured the time series graph scales and added a table displaying the most prominent flows and the associated device.
Turns out pcap doesn't store direction information so I couldn't implement the ingress/egress filter. pcap-ng supports direction though Libtrace doesn't support this format yet. Should be easy enough to implement when I have the information available.
Recently I have been planning my presentation which is this Wednesday.
Plan on seeing Brad either this week or next week to get NetFlow configured so I can start working with real data.

11

Apr

2015

The first few weeks I focused on the blurb and proposal.

Throughout the next few weeks I installed and got familiar with Python, Django (web application framework), SQLite (database) and Flot (Javascript graphs). My aim was to get the web page layout completed so that I can focus on collecting, storing and querying the Netflow data.

I found a trace file and wrote a script to output Python code which I piped into the Python terminal to save it to a SQLite database. I then queried the data in Django and graphed some information including protocol counts and a usage timeline.

Last week I discovered Shane's Libprotoident library for application layer protocol identification for flows. I thought it would be great if I could utilise this to display application information for the flows I will collect. I went to see Shane, who told me a previous students project used Libprotoident to request the type of information I would like. I will look into this over the next few weeks.

This week I have been busy with assignments. I hope to continue with the web interface early next week.