diff options
Diffstat (limited to '2005/flow-accounting-lt2005/flow-accounting-lt2005.mgp')
| -rw-r--r-- | 2005/flow-accounting-lt2005/flow-accounting-lt2005.mgp | 299 |
1 files changed, 299 insertions, 0 deletions
diff --git a/2005/flow-accounting-lt2005/flow-accounting-lt2005.mgp b/2005/flow-accounting-lt2005/flow-accounting-lt2005.mgp new file mode 100644 index 0000000..601eb88 --- /dev/null +++ b/2005/flow-accounting-lt2005/flow-accounting-lt2005.mgp @@ -0,0 +1,299 @@ +%include "default.mgp" +%default 1 bgrad +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +%nodefault +%back "blue" + +%center +%size 7 + + +Flow-based network accounting with Linux +OLS 2005 (July 22, 2005) + +%center +%size 4 +by + +Harald Welte <hwelte@hmw-consulting.de> + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +Contents + + Introduction + Network Acounting + Existing Tools + ip_conntrack_acct + ctnetlink / conntrack tool + ulogd2 + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +Introduction + + Who is speaking to you? + an independent Free Software developer + who earns his living off Free Software since 1997 + who is one of the authors of the linux kernel firewall system called netfilter/iptables + who has recently given lots of non-technical presentations about GPL enforcement + who is happy to again speak about a technical subject today + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +Network Accounting + + + Counting of metadata of network traffic + Optionally Summarizing + Kind of metadate dependant on application + number of packets + number of bytes + Scope + per timeframe + per connection + per flow + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +Network Accounting + + + Reasons for network accounting + volume or bandwith based billing + monitoring of network utilization / disstribution + research on network usage patterns, ... + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +Existing accounting solutions + + + Existing accounting solutions for Linux + nacctd (net-acct) + ipt_LOG based + ipt_ULOG based + iptables-based (ipac-ng) + ipt_ACCOUNT + ntop + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +nacctd / net-acct + + + nacctd + Oldest tool available, at least since 1995 + Originally developed by Ulrich Callmeier + Later unmaintained, multiple forks + Principle of operation: + Capture all packets using libpcap (AF_PACKET) + try to aggregate packets into flows + log to ASCII file (some branches provide SQL backends) + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +ipt_LOG based + + + ipt_LOG + iptables "LOG" target, available in all 2.4.x and 2.6.x kernels + Designed to log policy violations, not accounting data + Not intended for logging of high data volumes + Principle of Operation + Iptables rule with "LOG" target for to-be-logged packets + syslogd writes one line for each packet + Perl scripts (or similar) used to parse syslog files + Summary + Doesn't scale since it abuses ipt_LOG for unintended purpose + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +ipt_ULOG based + + + ipt_ULOG + iptables "ULOG" target, available in almost all 2.4.x and 2.6.x kernels + Designed to efficiently log policy violations, not accounting data + Principle of Operation + Copy header of packets into buffer + Flush buffer to userspace + Have a daemon parse packet headers in buffer + Write information to some form of storage + Summary + Scales way better than ipt_LOG + I still abusing an interface for a different purpose + Still needs to transfer all packets to userspace + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +ip_tables counter based + + + Accounting based on ip_tables + Every ip_tables ruleset has per-rule packet and byte counters + A number of ready-built tools exist to parse and summarize + Most commonly used is "ipac-ng", supports storage in SQL DB + Principle of Operation + Careful placement of fallthrough-rules + Executing "iptables -L -vn" or "iptables-save -c" displays counters + Counters can be reset by "iptables -Z" + Summary + Scales well with high traffic + Scales badly for lots of different accounting groups (which require lots of rules) + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +ipt_ACCOUNT + + + ipt_ACCOUNT + http://www.intra2net.com/opensource/ipt_account/ + A special purpose iptables target, requires kernel patch + Principle of Operation + Keeps byte counters per IP address in a given subnet (/24, eg.) + Counters can be read by special "iptaccount" commandline tool + Summary + Is limited to networks up to /8 + Granularity only down to per-ip level + Highly optimized, but special-purpose + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +ip_conntrack_acct + + + ip_conntrack based accounting + The netfilter connection tracking subsystem runs on almost any firewall + Accounting is usually done at the edge of a network, where a firewall is placed + ip_conntrack already maintains some ~ 350 bytes of state per connection + Principle of Operation + Add per-connection, per-direction packet and byte counters + Read the counters from userspace (/proc/net/ip_conntrack or ctnetlink-based) + Summary + adds little extra overhead if ip_conntrack is used already + Not recommended for non-firewall systems + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +ip_conntrack_acct + + +Userpace interfaces + /proc/net/ip_conntrack + shows one line per connection + if CONFIG_IP_NF_CT_ACCT is enabled, "packets=5749 bytes=423453" is added for each direction + Pro: + Easy to use + Con: + Not always accurate + No way to reset counters + Inefficient + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +ip_conntrack_acct + + + ctnetlink based interface + What is ctnetlink? + it's a netlink-based interface to ip_conntrack + allows reading/deleting/updating/creating conntrack entries from usrspace + exists as out-of-kernel patch for many years + Extending ctnetlink with ip_conntrack_acct + Simple: Add counter information to TLV's passed from kernel to userspace + Additional features + Add new IPCTNL_MSG_CT_GET_CTRZERO command request for atomic get-counters-and-zero + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +ip_conntrack_acct + + +Possible ctnetlink based implementations + polling-based + use GET_CTRZERO in a regular sampling interval + add up counters with every call + Pro: configurable granularity + Con: overhead increases with short samling interval + event-based + listen for ctnetlink DELETE event messages + store flow-based information only once at the end of every connection + Pro: Very easy to implement + Con: Data only available after connection finishes + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +ip_conntrack_acct + + +Programs to use ip_conntrack_acct + 'conntrack' tool + http://svn.netfilter.org/trunk/conntrack + Try "conntrack -E conntrack" for event-based output + Try "conntrack -L conntrack" for polling + Try "conntrack -L conntrack -z" for poll with zeroing counters + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +ip_conntrack_acct + + +Programs to use ip_conntrack_acct + ulogd2 + http://svn.netfilter.org/branches/ulogd2 + next-gerneration of 'ulogd' + can log per-packet and per-flow information + can aggregate per-packet to per-flow information + can run multiple 'plugin stacks' for multiple outputs + can export per-flow data in IPFIX format + is not fully implemented yet, but pretty far ahead + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%page +Flow-based network accounting with Linux +Thanks + + Thanks to + Alan Cox, Alexey Kuznetsov, David Miller, Andi Kleen + for implementing (one of?) the world's best TCP/IP stacks + Paul 'Rusty' Russell + for starting the netfilter/iptables project + for trusting me to maintain it today + Astaro AG + for sponsoring parts of my netfilter work + Free Software Foundation + for the GNU Project + for the GNU General Public License +%size 3 + http://gnumonks.org/ +%size 3 + http://netfilter.org/ +%size 3 + http://svn.netfilter.org/ + |