%include "default.mgp" %default 1 bgrad %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page %nodefault %back "blue" %center %size 7 Flow-based network accounting with Linux OLS 2005 (July 22, 2005) %center %size 4 by Harald Welte %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux Contents Introduction Network Acounting Existing Tools ip_conntrack_acct ctnetlink / conntrack tool ulogd2 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux Introduction Who is speaking to you? an independent Free Software developer who earns his living off Free Software since 1997 who is one of the authors of the linux kernel firewall system called netfilter/iptables who has recently given lots of non-technical presentations about GPL enforcement who is happy to again speak about a technical subject today %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux Network Accounting Counting of metadata of network traffic Optionally Summarizing Kind of metadate dependant on application number of packets number of bytes Scope per timeframe per connection per flow %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux Network Accounting Reasons for network accounting volume or bandwith based billing monitoring of network utilization / disstribution research on network usage patterns, ... %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux Existing accounting solutions Existing accounting solutions for Linux nacctd (net-acct) ipt_LOG based ipt_ULOG based iptables-based (ipac-ng) ipt_ACCOUNT ntop %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux nacctd / net-acct nacctd Oldest tool available, at least since 1995 Originally developed by Ulrich Callmeier Later unmaintained, multiple forks Principle of operation: Capture all packets using libpcap (AF_PACKET) try to aggregate packets into flows log to ASCII file (some branches provide SQL backends) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux ipt_LOG based ipt_LOG iptables "LOG" target, available in all 2.4.x and 2.6.x kernels Designed to log policy violations, not accounting data Not intended for logging of high data volumes Principle of Operation Iptables rule with "LOG" target for to-be-logged packets syslogd writes one line for each packet Perl scripts (or similar) used to parse syslog files Summary Doesn't scale since it abuses ipt_LOG for unintended purpose %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux ipt_ULOG based ipt_ULOG iptables "ULOG" target, available in almost all 2.4.x and 2.6.x kernels Designed to efficiently log policy violations, not accounting data Principle of Operation Copy header of packets into buffer Flush buffer to userspace Have a daemon parse packet headers in buffer Write information to some form of storage Summary Scales way better than ipt_LOG I still abusing an interface for a different purpose Still needs to transfer all packets to userspace %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux ip_tables counter based Accounting based on ip_tables Every ip_tables ruleset has per-rule packet and byte counters A number of ready-built tools exist to parse and summarize Most commonly used is "ipac-ng", supports storage in SQL DB Principle of Operation Careful placement of fallthrough-rules Executing "iptables -L -vn" or "iptables-save -c" displays counters Counters can be reset by "iptables -Z" Summary Scales well with high traffic Scales badly for lots of different accounting groups (which require lots of rules) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux ipt_ACCOUNT ipt_ACCOUNT http://www.intra2net.com/opensource/ipt_account/ A special purpose iptables target, requires kernel patch Principle of Operation Keeps byte counters per IP address in a given subnet (/24, eg.) Counters can be read by special "iptaccount" commandline tool Summary Is limited to networks up to /8 Granularity only down to per-ip level Highly optimized, but special-purpose %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux ip_conntrack_acct ip_conntrack based accounting The netfilter connection tracking subsystem runs on almost any firewall Accounting is usually done at the edge of a network, where a firewall is placed ip_conntrack already maintains some ~ 350 bytes of state per connection Principle of Operation Add per-connection, per-direction packet and byte counters Read the counters from userspace (/proc/net/ip_conntrack or ctnetlink-based) Summary adds little extra overhead if ip_conntrack is used already Not recommended for non-firewall systems %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux ip_conntrack_acct Userpace interfaces /proc/net/ip_conntrack shows one line per connection if CONFIG_IP_NF_CT_ACCT is enabled, "packets=5749 bytes=423453" is added for each direction Pro: Easy to use Con: Not always accurate No way to reset counters Inefficient %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux ip_conntrack_acct ctnetlink based interface What is ctnetlink? it's a netlink-based interface to ip_conntrack allows reading/deleting/updating/creating conntrack entries from usrspace exists as out-of-kernel patch for many years Extending ctnetlink with ip_conntrack_acct Simple: Add counter information to TLV's passed from kernel to userspace Additional features Add new IPCTNL_MSG_CT_GET_CTRZERO command request for atomic get-counters-and-zero %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux ip_conntrack_acct Possible ctnetlink based implementations polling-based use GET_CTRZERO in a regular sampling interval add up counters with every call Pro: configurable granularity Con: overhead increases with short samling interval event-based listen for ctnetlink DELETE event messages store flow-based information only once at the end of every connection Pro: Very easy to implement Con: Data only available after connection finishes %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux ip_conntrack_acct Programs to use ip_conntrack_acct 'conntrack' tool http://svn.netfilter.org/trunk/conntrack Try "conntrack -E conntrack" for event-based output Try "conntrack -L conntrack" for polling Try "conntrack -L conntrack -z" for poll with zeroing counters %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux ip_conntrack_acct Programs to use ip_conntrack_acct ulogd2 http://svn.netfilter.org/branches/ulogd2 next-gerneration of 'ulogd' can log per-packet and per-flow information can aggregate per-packet to per-flow information can run multiple 'plugin stacks' for multiple outputs can export per-flow data in IPFIX format is not fully implemented yet, but pretty far ahead %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page Flow-based network accounting with Linux Thanks Thanks to Alan Cox, Alexey Kuznetsov, David Miller, Andi Kleen for implementing (one of?) the world's best TCP/IP stacks Paul 'Rusty' Russell for starting the netfilter/iptables project for trusting me to maintain it today Astaro AG for sponsoring parts of my netfilter work Free Software Foundation for the GNU Project for the GNU General Public License %size 3 http://gnumonks.org/ %size 3 http://netfilter.org/ %size 3 http://svn.netfilter.org/