21 May
2017
21 May
'17
9:41 a.m.
I'm now gathering netflow-like statistics from the router daemon.
It's a lot of data.
I've been unable to find a clear definition of the standard (v1 or v5)
netflow disk file format, so I don't have input suitable for any of the
good analysis tools. Does anyone have such a description?
And what are your favourite analysis tools?
- Brian
21 May
21 May
11:02 a.m.
Brain,
NFsen[1] or NTop[2] could both act as a netflow collector.
1: http://nfsen.sourceforge.net/
2: http://www.ntop.org/products/netflow/nprobe/netflow-lite-plugin/
---
73 Rudy pd0zry
11:12 a.m.
- ntop has a nice user interface http://www.ntop.org
- flow-tools is a little harder to setup but more versatile. There is a gui
for it called flow-viewer https://sourceforge.net/projects/flowviewer//
-Neil
On Sun, May 21, 2017 at 8:41 AM, Brian Kantor <Brian(a)ucsd.edu> wrote:
(Please trim inclusions from previous messages)
_______________________________________________
I'm now gathering netflow-like statistics from the router daemon.
It's a lot of data.
I've been unable to find a clear definition of the standard (v1 or v5)
netflow disk file format, so I don't have input suitable for any of the
good analysis tools. Does anyone have such a description?
And what are your favourite analysis tools?
- Brian
_________________________________________
44Net mailing list
44Net(a)hamradio.ucsd.edu
http://hamradio.ucsd.edu/mailman/listinfo/44net
--
Neil Johnson
11:37 a.m.
I like ntop; but the version available through most OEes is the old
version (the old version did not have a integrated database/save
feature). The new version (I believe) requires a paid license.
I use nfsen on my collector, the router dumps in netflow v5 - using the
softflowd software packaged with LEDE.
I'll search for a 'white paper' on the format/syntax of the output data.
- Lynwood
KB3VWG
11:44 a.m.
Correction, I use NetFlow version 9.
Also, the newer version of ntop is called ntop-ng.
These documents describe the format of the output of a version 9 packet:
http://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_pape…
http://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6…
- KB3VWG
11:47 a.m.
FreeBSD has source for ntop, nfsen, and softflowd. It appears that
ntop and nfsen each have their own private file layout. I'm still
looking at softflowd, but my first impression is that its output
is datagrams headed for a collector which would in turn write it
to disk - probably in its own private format.
I had hoped to avoid the overhead of sending the data in NetFlow
packets to a separate collector.
Point is that I've already got the data exported from the router.
Now I'm trying to write it to disk in a format that one of the
analysis packages will cope with. The two file formats I've
looked at seem obscure.
- Brian
On Sun, May 21, 2017 at 11:37:28AM -0400, lleachii--- via 44Net wrote:
(Please trim inclusions from previous messages)
_______________________________________________
I like ntop; but the version available through most OEes is the old version
(the old version did not have a integrated database/save feature). The new
version (I believe) requires a paid license.
I use nfsen on my collector, the router dumps in netflow v5 - using the
softflowd software packaged with LEDE.
I'll search for a 'white paper' on the format/syntax of the output data.
- Lynwood
KB3VWG
_________________________________________
44Net mailing list
44Net(a)hamradio.ucsd.edu
http://hamradio.ucsd.edu/mailman/listinfo/44net
11:51 a.m.
Brian,
As I recall, nfsen saves in pcap...verify, but that might be more
helpful to you/us.
- KB3VWG
12:10 p.m.
I think nsen (and softflowd) can read a previously-captured pcap file,
derive flow data from the captured packets, and turn it in to Cisco
netflow packets to send to a collector.
Looks like I'm going to have to ditch the file I'm currently writing
(instead of altering its format) and concentrate on generating and
sending Cisco netflow packets to a collector, as that's apparently the
common format involved in this whole scheme.
Ah well, what's another socket write to the router daemon.
Thanks folks!
- Brian
On Sun, May 21, 2017 at 11:51:12AM -0400, lleachii--- via 44Net wrote:
(Please trim inclusions from previous messages)
_______________________________________________
Brian,
As I recall, nfsen saves in pcap...verify, but that might be more helpful to
you/us.
- KB3VWG
_________________________________________
44Net mailing list
44Net(a)hamradio.ucsd.edu
http://hamradio.ucsd.edu/mailman/listinfo/44net
12:23 p.m.
Brian,
Yes, that's correct, I extract the data from my PHYs using a software
that outputs in in netflow version 9.
There may be software that does an all-in-one; but I've never seen one
implemented; except in devices that integrate/render a GUI and/or its
own datastore for the flow date (e.g. Palo Alto, I also believe the new
Cisco NetFire product will also do something similar to this).
They also have the feature to read/save in pcap. Nfsen is the a GUI for
a backend that saves in that format.
In the terms of a Netflow architecture:
My NetFlow Exporter - LEDE device running softlowd in -v 9 mode
Collector - nfsen on a Linux Server
Flow Storage - pcap files processed using nfsen on the same server
Analysis Console - HTTP client browses to nfsen GUI
- KB3VWG
I think nsen (and softflowd) can read a
previously-captured pcap file,
derive flow data from the captured packets, and turn it in to Cisco
netflow packets to send to a collector.
22 May
22 May
6:20 a.m.
I've got the router daemon generating and sending what look like valid
Cisco netflow v5 packets, but the available specifications for the packets
don't include the values of the content of some fields in the packets.
It's always easier to reverse-engineer protocols when you have a few
example of valid packets. According to the Wikipedia article on netflow,
UDP packets to port 2055 are what I'm looking for. They consist of a
24-byte header followed by up to 30 data records of 48 bytes each, for
a maximum packet length of 1464 bytes.
If one of you who has an operational netflow setup would be kind enough
to capture ONE netflow packet using tcpdump and send me the hex dump
of the packet, I can extract the real-world values from the packet and
make mine conform. All I really need are the values of the 'engine_type'
and 'engine_id', bytes 21 and 22 (counting from zero as the first byte)
of the packet data. A dump of the netflow header (the first 24 bytes
of one packet) or a whole packet in hex would do very nicely, thank you.
Thanks!
- Brian
On Sun, May 21, 2017 at 09:10:52AM -0700, Brian Kantor wrote:
I think nsen (and softflowd) can read a
previously-captured pcap file,
derive flow data from the captured packets, and turn it in to Cisco
netflow packets to send to a collector.
Looks like I'm going to have to ditch the file I'm currently writing
(instead of altering its format) and concentrate on generating and
sending Cisco netflow packets to a collector, as that's apparently the
common format involved in this whole scheme.
Ah well, what's another socket write to the router daemon.
Thanks folks!
- Brian
11:43 a.m.
Brian,
I sent a copies of version 9 and version 5 packets to you off thread.
- Lynwood
KB3VWG
12:06 p.m.
Thank you, version 5 was what I was looking for.
Thanks!
- Brian
On Mon, May 22, 2017 at 11:43:54AM -0400, lleachii--- via 44Net wrote:
(Please trim inclusions from previous messages)
_______________________________________________
Brian,
I sent a copies of version 9 and version 5 packets to you off thread.
- Lynwood
KB3VWG
_________________________________________
44Net mailing list
44Net(a)hamradio.ucsd.edu
http://hamradio.ucsd.edu/mailman/listinfo/44net
25 May
25 May
3:30 a.m.
On 22 May 2017, at 12:20, Brian Kantor
<Brian(a)UCSD.Edu> wrote:
If one of you who has an operational netflow setup would be kind enough
to capture ONE netflow packet using tcpdump and send me the hex dump
of the packet, I can extract the real-world values from the packet and
make mine conform. All I really need are the values of the 'engine_type'
and 'engine_id', bytes 21 and 22 (counting from zero as the first byte)
of the packet data. A dump of the netflow header (the first 24 bytes
of one packet) or a whole packet in hex would do very nicely, thank you.
Being late to the party. Let me know if you still need it. I can send you samples of
Mikrotik and
Juniper Netflow packets.
Anyway expect some chaos regarding Netflow. Originally it was just a Cisco thing and
manufacturers
and programmers make their own decisions, sometimes quite surprising.
Borja.
3:54 a.m.
Thanks, no, I don't need a sample packet any more. Lynwood was good
enough to send me one a few days ago. Turns out that the fields I
was worrying about are all zero. I did have the 'seq' header field
incrementing incorrectly, but that's fixed. Reverse-engineering
software is one of the things I'm paid to do, so I had the tools
needed for this job right at hand.
Because I couldn't get the (essentially proprietary) collector disk
formats, I gave up on writing the storage disk files directly from
the router process, and went to the overhead of generating, sending,
and collecting the netflow v5 packets.
I'm now sending data that is compatable with the 'flow-tools' collector,
so it's likely that any of the other capture and analysis tools would
be happy with the packets I'm generating (and sending to the collector
over the loopback interface). I'll give some of them a try.
So thanks for the offer, but I think it's a solved problem already.
- Brian
On Thu, May 25, 2017 at 09:30:26AM +0200, Borja Marcos wrote:
Being late to the party. Let me know if you still need
it. I can send you samples of Mikrotik and
Juniper Netflow packets.
Anyway expect some chaos regarding Netflow. Originally it was just a Cisco thing and
manufacturers
and programmers make their own decisions, sometimes quite surprising.
Borja.
4:18 a.m.
By the way, it appears that we're storing about 40 MB of flow
data each day, so I can easily keep 10 or more days of data
on hand. Maybe even a month.
- Brian
3:25 a.m.
On 21 May 2017, at 17:51, lleachii--- via 44Net
<44net(a)hamradio.ucsd.edu> wrote:
As I recall, nfsen saves in pcap...verify, but that might be more helpful to you/us.
Nfdump uses its own format to store data. However you can use nfreplay(1) to read those
files
and send a Netflow stream to another application.
Also, helpfully, nfcapd (the Netflow capture program used by the nfsen suite) can relay
Netflow
packets to other destinations in real time. So, you can set up a Netflow feed to nfsen and
feed
the information in parallel (in real time) to other Netflow collectors.
Borja.
24 May
24 May
11:41 p.m.
On 05/21/2017 08:47 AM, Brian Kantor wrote:
(Please trim inclusions from previous messages)
_______________________________________________
FreeBSD has source for ntop, nfsen, and softflowd. It appears that
ntop and nfsen each have their own private file layout. I'm still
looking at softflowd, but my first impression is that its output
is datagrams headed for a collector which would in turn write it
to disk - probably in its own private format.
I had hoped to avoid the overhead of sending the data in NetFlow
packets to a separate collector.
Point is that I've already got the data exported from the router.
Now I'm trying to write it to disk in a format that one of the
analysis packages will cope with. The two file formats I've
looked at seem obscure.
- Brian
Brian,
Try Graylog. It has a netflow (v5) collector that you can use to send
the data to it and then analyze it. Plus, with the elasticsearch back
end, you can then also use tools like Kibana to really dig down into the
data for analysis.
-Stacy
11:43 p.m.
Thanks, I'll check that out.
- Brian
On Wed, May 24, 2017 at 08:41:47PM -0700, Stacy wrote:
On 05/21/2017 08:47 AM, Brian Kantor wrote:
(Please trim inclusions from previous messages)
_______________________________________________
FreeBSD has source for ntop, nfsen, and softflowd. It appears that
ntop and nfsen each have their own private file layout. I'm still
looking at softflowd, but my first impression is that its output
is datagrams headed for a collector which would in turn write it
to disk - probably in its own private format.
I had hoped to avoid the overhead of sending the data in NetFlow
packets to a separate collector.
Point is that I've already got the data exported from the router.
Now I'm trying to write it to disk in a format that one of the
analysis packages will cope with. The two file formats I've
looked at seem obscure.
- Brian
Brian,
Try Graylog. It has a netflow (v5) collector that you can use to send the
data to it and then analyze it. Plus, with the elasticsearch back end, you
can then also use tools like Kibana to really dig down into the data for
analysis.
-Stacy
25 May
25 May
3:28 a.m.
On 25 May 2017, at 05:41, Stacy
<kg7qin(a)arrl.net> wrote:
Brian,
Try Graylog. It has a netflow (v5) collector that you can use to send the data to it and
then analyze it. Plus, with the elasticsearch back end, you can then also use tools like
Kibana to really dig down into the data for analysis.
Yes, Elastic+Kibana can be really helpful. The graphs generated by nfsen are just traffic
volume graphs. It’s often helpful to graph
other counts (like number of source or destination IP addresses involved, for example) and
that’s very easy to do with Kibana.
Borja.
3:22 a.m.
On 21 May 2017, at 17:12, Neil Johnson
<neil.johnson(a)erudicon.com> wrote:
(Please trim inclusions from previous messages)
_______________________________________________
- ntop has a nice user interface http://www.ntop.org
- flow-tools is a little harder to setup but more versatile. There is a gui
for it called flow-viewer https://sourceforge.net/projects/flowviewer//
ntop is a bit more limited, better suited to running a collector for a small network.
I tried it long ago, though (I was bsd(a)ntop.org for a while, even) but I haven’t tried it
lately.
As far as I know you need a paid option in order to import Netflow flows.
Borja.
3:20 a.m.
On 21 May 2017, at 15:41, Brian Kantor
<Brian(a)UCSD.Edu> wrote:
(Please trim inclusions from previous messages)
_______________________________________________
I'm now gathering netflow-like statistics from the router daemon.
It's a lot of data.
I've been unable to find a clear definition of the standard (v1 or v5)
netflow disk file format, so I don't have input suitable for any of the
good analysis tools. Does anyone have such a description?
Sorry to be late to the party. The most widely used versions now are v5 and v9. v9
is more flexible and it even supports IPv6. Not only Cisco and Juniper but other router
manufacturers like Mikrotik and Ubiquiti export usable Netflow packets.
There is also a new, more open standard format called IPFIX.
And what are your favourite analysis tools?
I guess most people are using nfdump/nfsen like myself. It’s not perfect but it can be
very useful.
I run it on FreeBSD, of course benefiting from the wonders of ZFS :)
For some reason (I don't remember why) finally I didn't use the ports version, but
I installed
the dependencies from ports and I set up nfsen manually. Also, I use nginx rather than
Apache
as a front-end and I run PHP in fpm mode instead of being a web server module.
I can help, let me know if you need it.
For a more modern, trendy approach, you could try with Elasticsearch + Kibana + Logstash.
Elasticsearch is a powerful search database and engine, Kibana allows you to make searches
and
create nice graphs and Logstash is the ingestion tool that understands many formats
(including Netflow).
That’s the approach I am trying now at home, in parallel with nfsen+nfdump.
At work (AS3262) we have a new Elastic setup but so far I am keeping my trusty, old
nfdump+nfsen.
Borja - EA2EKH
2738
days inactive
2742
days old
20 comments
6 participants
participants (6)
-
Borja Marcos -
Brian Kantor -
lleachii＠aol.com -
Neil Johnson -
Rudy Hardeman (PD0ZRY) -
Stacy