I like ntop; but the version available through most OEes is the old version (the old version did not have a integrated database/save feature). The new version (I believe) requires a paid license.

I use nfsen on my collector, the router dumps in netflow v5 - using the softflowd software packaged with LEDE.

I'll search for a 'white paper' on the format/syntax of the output data.

- Lynwood KB3VWG

FreeBSD has source for ntop, nfsen, and softflowd. It appears that ntop and nfsen each have their own private file layout. I'm still looking at softflowd, but my first impression is that its output is datagrams headed for a collector which would in turn write it to disk - probably in its own private format.

I had hoped to avoid the overhead of sending the data in NetFlow packets to a separate collector.

Point is that I've already got the data exported from the router. Now I'm trying to write it to disk in a format that one of the analysis packages will cope with. The two file formats I've looked at seem obscure. - Brian

On Sun, May 21, 2017 at 11:37:28AM -0400, lleachii--- via 44Net wrote:

(Please trim inclusions from previous messages) _______________________________________________ I like ntop; but the version available through most OEes is the old version (the old version did not have a integrated database/save feature). The new version (I believe) requires a paid license.

I use nfsen on my collector, the router dumps in netflow v5 - using the softflowd software packaged with LEDE.

I'll search for a 'white paper' on the format/syntax of the output data.

• Lynwood

KB3VWG

---

44Net mailing list 44Net@hamradio.ucsd.edu http://hamradio.ucsd.edu/mailman/listinfo/44net

I think nsen (and softflowd) can read a previously-captured pcap file, derive flow data from the captured packets, and turn it in to Cisco netflow packets to send to a collector.

Looks like I'm going to have to ditch the file I'm currently writing (instead of altering its format) and concentrate on generating and sending Cisco netflow packets to a collector, as that's apparently the common format involved in this whole scheme.

Ah well, what's another socket write to the router daemon.

Thanks folks! - Brian

On Sun, May 21, 2017 at 11:51:12AM -0400, lleachii--- via 44Net wrote:

(Please trim inclusions from previous messages) _______________________________________________ Brian,

As I recall, nfsen saves in pcap...verify, but that might be more helpful to you/us.

• KB3VWG

---

44Net mailing list 44Net@hamradio.ucsd.edu http://hamradio.ucsd.edu/mailman/listinfo/44net

I've got the router daemon generating and sending what look like valid Cisco netflow v5 packets, but the available specifications for the packets don't include the values of the content of some fields in the packets.

It's always easier to reverse-engineer protocols when you have a few example of valid packets. According to the Wikipedia article on netflow, UDP packets to port 2055 are what I'm looking for. They consist of a 24-byte header followed by up to 30 data records of 48 bytes each, for a maximum packet length of 1464 bytes.

If one of you who has an operational netflow setup would be kind enough to capture ONE netflow packet using tcpdump and send me the hex dump of the packet, I can extract the real-world values from the packet and make mine conform. All I really need are the values of the 'engine_type' and 'engine_id', bytes 21 and 22 (counting from zero as the first byte) of the packet data. A dump of the netflow header (the first 24 bytes of one packet) or a whole packet in hex would do very nicely, thank you.

Thanks! - Brian

On Sun, May 21, 2017 at 09:10:52AM -0700, Brian Kantor wrote:

I think nsen (and softflowd) can read a previously-captured pcap file, derive flow data from the captured packets, and turn it in to Cisco netflow packets to send to a collector.

Looks like I'm going to have to ditch the file I'm currently writing (instead of altering its format) and concentrate on generating and sending Cisco netflow packets to a collector, as that's apparently the common format involved in this whole scheme.

Ah well, what's another socket write to the router daemon.

Thanks folks!

• Brian

Thank you, version 5 was what I was looking for. Thanks! - Brian

On Mon, May 22, 2017 at 11:43:54AM -0400, lleachii--- via 44Net wrote:

(Please trim inclusions from previous messages) _______________________________________________ Brian,

I sent a copies of version 9 and version 5 packets to you off thread.

• Lynwood

KB3VWG

---

44Net mailing list 44Net@hamradio.ucsd.edu http://hamradio.ucsd.edu/mailman/listinfo/44net

Thanks, no, I don't need a sample packet any more. Lynwood was good enough to send me one a few days ago. Turns out that the fields I was worrying about are all zero. I did have the 'seq' header field incrementing incorrectly, but that's fixed. Reverse-engineering software is one of the things I'm paid to do, so I had the tools needed for this job right at hand.

Because I couldn't get the (essentially proprietary) collector disk formats, I gave up on writing the storage disk files directly from the router process, and went to the overhead of generating, sending, and collecting the netflow v5 packets.

I'm now sending data that is compatable with the 'flow-tools' collector, so it's likely that any of the other capture and analysis tools would be happy with the packets I'm generating (and sending to the collector over the loopback interface). I'll give some of them a try.

So thanks for the offer, but I think it's a solved problem already. - Brian

On Thu, May 25, 2017 at 09:30:26AM +0200, Borja Marcos wrote:

Being late to the party. Let me know if you still need it. I can send you samples of Mikrotik and Juniper Netflow packets.

Anyway expect some chaos regarding Netflow. Originally it was just a Cisco thing and manufacturers and programmers make their own decisions, sometimes quite surprising.

Borja.

On 21 May 2017, at 17:51, lleachii--- via 44Net 44net@hamradio.ucsd.edu wrote:

As I recall, nfsen saves in pcap...verify, but that might be more helpful to you/us.

Nfdump uses its own format to store data. However you can use nfreplay(1) to read those files and send a Netflow stream to another application.

Also, helpfully, nfcapd (the Netflow capture program used by the nfsen suite) can relay Netflow packets to other destinations in real time. So, you can set up a Netflow feed to nfsen and feed the information in parallel (in real time) to other Netflow collectors.

Borja.

Thanks, I'll check that out. - Brian

On Wed, May 24, 2017 at 08:41:47PM -0700, Stacy wrote:

On 05/21/2017 08:47 AM, Brian Kantor wrote:

...

(Please trim inclusions from previous messages) _______________________________________________ FreeBSD has source for ntop, nfsen, and softflowd. It appears that ntop and nfsen each have their own private file layout. I'm still looking at softflowd, but my first impression is that its output is datagrams headed for a collector which would in turn write it to disk - probably in its own private format.

I had hoped to avoid the overhead of sending the data in NetFlow packets to a separate collector.

Point is that I've already got the data exported from the router. Now I'm trying to write it to disk in a format that one of the analysis packages will cope with. The two file formats I've looked at seem obscure.

• Brian

Brian, Try Graylog. It has a netflow (v5) collector that you can use to send the data to it and then analyze it. Plus, with the elasticsearch back end, you can then also use tools like Kibana to really dig down into the data for analysis.

-Stacy

On 21 May 2017, at 17:12, Neil Johnson neil.johnson@erudicon.com wrote:

(Please trim inclusions from previous messages) _______________________________________________

• ntop has a nice user interface http://www.ntop.org

• flow-tools is a little harder to setup but more versatile. There is a gui

for it called flow-viewer https://sourceforge.net/projects/flowviewer//

ntop is a bit more limited, better suited to running a collector for a small network.

I tried it long ago, though (I was bsd@ntop.org for a while, even) but I haven’t tried it lately. As far as I know you need a paid option in order to import Netflow flows.

Borja.