diff options
Diffstat (limited to '2004/netfilter-failover-ols2004/OLS2004-proceedings/welte/netfilter-failover-ols2002.tex')
| -rw-r--r-- | 2004/netfilter-failover-ols2004/OLS2004-proceedings/welte/netfilter-failover-ols2002.tex | 504 |
1 files changed, 504 insertions, 0 deletions
diff --git a/2004/netfilter-failover-ols2004/OLS2004-proceedings/welte/netfilter-failover-ols2002.tex b/2004/netfilter-failover-ols2004/OLS2004-proceedings/welte/netfilter-failover-ols2002.tex new file mode 100644 index 0000000..bf8d142 --- /dev/null +++ b/2004/netfilter-failover-ols2004/OLS2004-proceedings/welte/netfilter-failover-ols2002.tex @@ -0,0 +1,504 @@ +\documentclass[twocolumn]{article} +\usepackage{ols} +\begin{document} + +\date{} + +\title{\Large \bf How to replicate the fire - HA for netfilter based firewalls} + +\author{ +Harald Welte\\ +{\em Netfilter Core Team + Astaro AG}\\ +{\normalsize laforge@gnumonks.org/laforge@astaro.com, http://www.gnumonks.org/} +} + +\maketitle + +\thispagestyle{empty} + +\subsection*{Abstract} + With traditional, stateless firewalling (such as ipfwadm, ipchains) there is +no need for special HA support in the firewalling subsystem. As long as all +packet filtering rules and routing table entries are configured in exactly the +same way, one can use any available tool for IP-Address takeover to accomplish +the goal of failing over from one node to the other. + + With Linux 2.4.x netfilter/iptables, the Linux firewalling code moves beyond +traditional packet filtering. Netfilter provides a modular connection tracking +susbsystem which can be employed for stateful firewalling. The connection +tracking subsystem gathers information about the state of all current network +flows (connections). Packet filtering decisions and NAT information is +associated with this state information. + + In a high availability scenario, this connection tracking state needs to be +replicated from the currently active firewall node to all standby slave +firewall nodes. Only when all connection tracking state is replicated, the +slave node will have all necessarry state information at the time a failover +event occurs. + + The netfilter/iptables does currently not have any functionality for +replicating connection tracking state accross multiple nodes. However, +the author of this presentation, Harald Welte, has started a project for +connection tracking state replication with netfilter/iptables. + + The presentation will cover the architectural design and implementation +of the connection tracking failover sytem. With respect to the date of +the conference, it is to be expected that the project is still a +work-in-progress at that time. + +\section{Failover of stateless firewalls} + +There are no special precautions when installing a highly available +stateless packet filter. Since there is no state kept, all information +needed for filtering is the ruleset and the individual, seperate packets. + +Building a set of highly available stateless packet filters can thus be +achieved by using any traditional means of IP-address takeover, such +as Hartbeat or VRRPd. + +The only remaining issue is to make sure the firewalling ruleset is +exactly the same on both machines. This should be ensured by the firewall +administrator every time he updates the ruleset. + +If this is not applicable, because a very dynamic ruleset is employed, one +can build a very easy solution using iptables-supplied tools iptables-save +and iptables-restore. The output of iptables-save can be piped over ssh +to iptables-restore on a different host. + +Limitations +\begin{itemize} +\item +no state tracking +\item +not possible in combination with NAT +\item +no counter consistency of per-rule packet/byte counters +\end{itemize} + +\section{Failover of stateful firewalls} + +Modern firewalls implement state tracking (aka connection tracking) in order +to keep some state about the currently active sessions. The amount of +per-connection state kept at the firewall depends on the particular +implementation. + +As soon as {\bf any} state is kept at the packet filter, this state information +needs to be replicated to the slave/backup nodes within the failover setup. + +In Linux 2.4.x, all relevant state is kept within the {\it connection tracking +subsystem}. In order to understand how this state could possibly be +replicated, we need to understand the architecture of this conntrack subsystem. + +\subsection{Architecture of the Linux Connection Tracking Subsystem} + +Connection tracking within Linux is implemented as a netfilter module, called +ip\_conntrack.o. + +Before describing the connection tracking subsystem, we need to describe a +couple of definitions and primitives used throughout the conntrack code. + +A connection is represented within the conntrack subsystem using {\it struct +ip\_conntrack}, also called {\it connection tracking entry}. + +Connection tracking is utilizing {\it conntrack tuples}, which are tuples +consisting out of (srcip, srcport, dstip, dstport, l4prot). A connection is +uniquely identified by two tuples: The tuple in the original direction +(IP\_CT\_DIR\_ORIGINAL) and the tuple for the reply direction +(IP\_CT\_DIR\_REPLY). + +Connection tracking itself does not drop packets\footnote{well, in some rare +cases in combination with NAT it needs to drop. But don't tell anyone, this is +secret.} or impose any policy. It just associates every packet with a +connection tracking entry, which in turn has a particular state. All other +kernel code can use this state information\footnote{state information is +internally represented via the {\it struct sk\_buff.nfct} structure member of a +packet.}. + +\subsubsection{Integration of conntrack with netfilter} + +If the ip\_conntrack.o module is registered with netfilter, it attaches to the +NF\_IP\_PRE\_ROUTING, NF\_IP\_POST\_ROUTING, NF\_IP\_LOCAL\_IN and +NF\_IP\_LOCAL\_OUT hooks. + +Because forwarded packets are the most common case on firewalls, I will only +describe how connection tracking works for forwarded packets. The two relevant +hooks for forwarded packets are NF\_IP\_PRE\_ROUTING and NF\_IP\_POST\_ROUTING. + +Every time a packet arrives at the NF\_IP\_PRE\_ROUTING hook, connection +tracking creates a conntrack tuple from the packet. It then compares this +tuple to the original and reply tuples of all already-seen connections +\footnote{Of course this is not implemented as a linear search over all existing connections.} to find out if this just-arrived packet belongs to any existing +connection. If there is no match, a new conntrack table entry (struct +ip\_conntrack) is created. + +Let's assume the case where we have already existing connections but are +starting from scratch. + +The first packet comes in, we derive the tuple from the packet headers, look up +the conntrack hash table, don't find any matching entry. As a result, we +create a new struct ip\_conntrack. This struct ip\_conntrack is filled with +all necessarry data, like the original and reply tuple of the connection. +How do we know the reply tuple? By inverting the source and destination +parts of the original tuple.\footnote{So why do we need two tuples, if they can +be derived from each other? Wait until we discuss NAT.} +Please note that this new struct ip\_conntrack is {\bf not} yet placed +into the conntrack hash table. + +The packet is now passed on to other callback functions which have registered +with a lower priority at NF\_IP\_PRE\_ROUTING. It then continues traversal of +the network stack as usual, including all respective netfilter hooks. + +If the packet survives (i.e. is not dropped by the routing code, network stack, +firewall ruleset, ...), it re-appears at NF\_IP\_POST\_ROUTING. In this case, +we can now safely assume that this packet will be sent off on the outgoing +interface, and thus put the connection tracking entry which we created at +NF\_IP\_PRE\_ROUTING into the conntrack hash table. This process is called +{\it confirming the conntrack}. + +The connection tracking code itself is not monolithic, but consists out of a +couple of seperate modules\footnote{They don't actually have to be seperate +kernel modules; e.g. TCP, UDP and ICMP tracking modules are all part of +the linux kernel module ip\_conntrack.o}. Besides the conntrack core, there +are two important kind of modules: Protocol helpers and application helpers. + +Protocol helpers implement the layer-4-protocol specific parts. They currently +exist for TCP, UDP and ICMP (an experimental helper for GRE exists). + +\subsubsection{TCP connection tracking} + +As TCP is a connection oriented protocol, it is not very difficult to imagine +how conntection tracking for this protocol could work. There are well-defined +state transitions possible, and conntrack can decide which state transitions +are valid within the TCP specification. In reality it's not all that easy, +since we cannot assume that all packets that pass the packet filter actually +arrive at the receiving end, ... + +It is noteworthy that the standard connection tracking code does {\bf not} +do TCP sequence number and window tracking. A well-maintained patch to add +this feature exists almost as long as connection tracking itself. It will +be integrated with the 2.5.x kernel. The problem with window tracking is +it's bad interaction with connection pickup. The TCP conntrack code is able to +pick up already existing connections, e.g. in case your firewall was rebooted. +However, connection pickup is conflicting with TCP window tracking: The TCP +window scaling option is only transferred at connection setup time, and we +don't know about it in case of pickup... + +\subsubsection{ICMP tracking} + +ICMP is not really a connection oriented protocol. So how is it possible to +do connection tracking for ICMP? + +The ICMP protocol can be split in two groups of messages + +\begin{itemize} +\item +ICMP error messages, which sort-of belong to a different connection +ICMP error messages are associated {\it RELATED} to a different connection. +(ICMP\_DEST\_UNREACH, ICMP\_SOURCE\_QUENCH, ICMP\_TIME\_EXCEEDED, +ICMP\_PARAMETERPROB, ICMP\_REDIRECT). +\item +ICMP queries, which have a request->reply character. So what the conntrack +code does, is let the request have a state of {\it NEW}, and the reply +{\it ESTABLISHED}. The reply closes the connection immediately. +(ICMP\_ECHO, ICMP\_TIMESTAMP, ICMP\_INFO\_REQUEST, ICMP\_ADDRESS) +\end{itemize} + +\subsubsection{UDP connection tracking} + +UDP is designed as a connectionless datagram protocol. But most common +protocols using UDP as layer 4 protocol have bi-directional UDP communication. +Imagine a DNS query, where the client sends an UDP frame to port 53 of the +nameserver, and the nameserver sends back a DNS reply packet from it's UDP +port 53 to the client. + +Netfilter trats this as a connection. The first packet (the DNS request) is +assigned a state of {\it NEW}, because the packet is expected to create a new +'connection'. The dns servers' reply packet is marked as {\it ESTABLISHED}. + +\subsubsection{conntrack application helpers} + +More complex application protocols involving multiple connections need special +support by a so-called ``conntrack application helper module''. Modules in +the stock kernel come for FTP and IRC(DCC). Netfilter CVS currently contains +patches for PPTP, H.323, Eggdrop botnet, tftp ald talk. We're still lacking +a lot of protocols (e.g. SIP, SMB/CIFS) - but they are unlikely to appear +until somebody really needs them and either develops them on his own or +funds development. + +\subsubsection{Integration of connection tracking with iptables} + +As stated earlier, conntrack doesn't impose any policy on packets. It just +determines the relation of a packet to already existing connections. To base +packet filtering decision on this sate information, the iptables {\it state} +match can be used. Every packet is within one of the following categories: + +\begin{itemize} +\item +{\bf NEW}: packet would create a new connection, if it survives +\item +{\bf ESTABLISHED}: packet is part of an already established connection +(either direction) +\item +{\bf RELATED}: packet is in some way related to an already established connection, e.g. ICMP errors or FTP data sessions +\item +{\bf INVALID}: conntrack is unable to derive conntrack information from this packet. Please note that all multicast or broadcast packets fall in this category. +\end{itemize} + + +\subsection{Poor man's conntrack failover} + +When thinking about failover of stateful firewalls, one usually thinks about +replication of state. This presumes that the state is gathered at one +firewalling node (the currently active node), and replicated to several other +passive standby nodes. There is, howeve, a very different approach to +replication: concurrent state tracking on all firewalling nodes. + +The basic assumption of this approach is: In a setup where all firewalling +nodes receive exactly the same traffic, all nodes will deduct the same state +information. + +The implementability of this approach is totally dependent on fulfillment of +this assumption. + +\begin{itemize} +\item +{\it All packets need to be seen by all nodes}. This is not always true, but +can be achieved by using shared media like traditional ethernet (no switches!!) +and promiscuous mode on all ethernet interfaces. +\item +{\it All nodes need to be able to process all packets}. This cannot be +universally guaranteed. Even if the hardware (CPU, RAM, Chipset, NIC's) and +software (Linux kernel) are exactly the same, they might behave different, +especially under high load. To avoid those effects, the hardware should be +able to deal with way more traffic than seen during operation. Also, there +should be no userspace processes (like proxes, etc.) running on the firewalling +nodes at all. WARNING: Nobody guarantees this behaviour. However, the poor +man is usually not interested in scientific proof but in usability in his +particular practical setup. +\end{itemize} + +However, even if those conditions are fulfilled, ther are remaining issues: +\begin{itemize} +\item +{\it No resynchronization after reboot}. If a node is rebooted (because of +a hardware fault, software bug, software update, ..) it will loose all state +information until the event of the reboot. This means, the state information +of this node after reboot will not contain any old state, gathered before the +reboot. The effect depend on the traffic. Generally, it is only assured that +state information about all connections initiated after the reboot will be +present. If there are short-lived connections (like http), the state +information on the just rebooted node will approximate the state information of +an older node. Only after all sessions active at the time of reboot have +terminated, state information is guaranteed to be resynchronized. +\item +{\it Only possible with shared medium}. The practical implication is that no +switched ethernet (and thus no full duplex) can be used. +\end{itemize} + +The major advantage of the poor man's approach is implementation simplicity. +No state transfer mechanism needs to be developed. Only very little changes +to the existing conntrack code would be needed in order to be able to +do tracking based on packets received from promiscuous interfaces. The active +node would have packet forwarding turned on, the passive nodes off. + +I'm not proposing this as a real solution to the failover problem. It's +hackish, buggy and likely to break very easily. But considering it can be +implemented in very little programming time, it could be an option for very +small installations with low reliability criteria. + +\subsection{Conntrack state replication} + +The preferred solution to the failover problem is, without any doubt, +replication of the connection tracking state. + +The proposed conntrack state replication soltution consists out of several +parts: +\begin{itemize} +\item +A connection tracking state replication protocol +\item +An event interface generating event messages as soon as state information +changes on the active node +\item +An interface for explicit generation of connection tracking table entries on +the standby slaves +\item +Some code (preferrably a kernel thread) running on the active node, receiving +state updates by the event interface and generating conntrack state replication +protocol messages +\item +Some code (preferrably a kernel thread) running on the slave node(s), receiving +conntrack state replication protocol messages and updating the local conntrack +table accordingly +\end{itemize} + +Flow of events in chronological order: +\begin{itemize} +\item +{\it on active node, inside the network RX softirq} +\begin{itemize} +\item + connection tracking code is analyzing a forwarded packet +\item + connection tracking gathers some new state information +\item + connection tracking updates local connection tracking database +\item + connection tracking sends event message to event API +\end{itemize} +\item +{\it on active node, inside the conntrack-sync kernel thread} + \begin{itemize} + \item + conntrack sync daemon receives event through event API + \item + conntrack sync daemon aggregates multiple event messages into a state replication protocol message, removing possible redundancy + \item + conntrack sync daemon generates state replication protocol message + \item + conntrack sync daemon sends state replication protocol message +(private network between firewall nodes) + \end{itemize} +\item +{\it on slave node(s), inside network RX softirq} + \begin{itemize} + \item + connection tracking code ignores packets coming from the interface attached to the private conntrac sync network + \item + state replication protocol messages is appended to socket receive queue of conntrack-sync kernel thread + \end{itemize} +\item +{\it on slave node(s), inside conntrack-sync kernel thread} + \begin{itemize} + \item + conntrack sync daemon receives state replication message + \item + conntrack sync daemon creates/updates conntrack entry + \end{itemize} +\end{itemize} + + +\subsubsection{Connection tracking state replication protocol} + + + In order to be able to replicate the state between two or more firewalls, a +state replication protocol is needed. This protocol is used over a private +network segment shared by all nodes for state replication. It is designed to +work over IP unicast and IP multicast transport. IP unicast will be used for +direct point-to-point communication between one active firewall and one +standby firewall. IP multicast will be used when the state needs to be +replicated to more than one standby firewall. + + + The principle design criteria of this protocol are: +\begin{itemize} +\item + {\bf reliable against data loss}, as the underlying UDP layer does only + provide checksumming against data corruption, but doesn't employ any + means against data loss +\item + {\bf lightweight}, since generating the state update messages is + already a very expensive process for the sender, eating additional CPU, + memory and IO bandwith. +\item + {\bf easy to parse}, to minimize overhead at the receiver(s) +\end{itemize} + +The protocol does not employ any security mechanism like encryption, +authentication or reliability against spoofing attacks. It is +assumed that the private conntrack sync network is a secure communications +channel, not accessible to any malicious 3rd party. + +To achieve the reliability against data loss, an easy sequence numbering +scheme is used. All protocol messages are prefixed by a seuqence number, +determined by the sender. If the slave detects packet loss by discontinuous +sequence numbers, it can request the retransmission of the missing packets +by stating the missing sequence number(s). Since there is no acknowledgement +for sucessfully received packets, the sender has to keep a reasonably-sized +backlog of recently-sent packets in order to be able to fulfill retransmission +requests. + +The different state replication protocol messages types are: +\begin{itemize} +\item +{\bf NF\_CTSRP\_NEW}: New conntrack entry has been created (and +confirmed\footnote{See the above description of the conntrack code for what is +meant by {\it confirming} a conntrack entry}) +\item +{\bf NF\_CTSRP\_UPDATE}: State information of existing conntrack entry has +changed +\item +{\bf NF\_CTSRP\_EXPIRE}: Existing conntrack entry has been expired +\end{itemize} + +To uniquely identify (and later reference) a conntrack entry, a +{\it conntrack\_id} is assigned to every conntrack entry transferred +using a NF\_CTSRP\_NEW message. This conntrack\_id must be saved at the +receiver(s) together with the conntrack entry, since it is used by the sender +for subsequent NF\_CTSRP\_UPDATE and NF\_CTSRP\_EXPIRE messages. + +The protocol itself does not care about the source of this conntrack\_id, +but since the current netfilter connection tracking implementation does never +change the addres of a conntrack entry, the memory address of the entry can be +used, since it comes for free. + + +\subsubsection{Connection tracking state syncronization sender} + +Maximum care needs to be taken for the implementation of the ctsyncd sender. + +The normal workload of the active firewall node is likely to be already very +high, so generating and sending the conntrack state replication messages needs +to be highly efficient. + +\begin{itemize} +\item + {\bf NF\_CTSRP\_NEW} will be generated at the NF\_IP\_POST\_ROUTING + hook, at the time ip\_conntrack\_confirm() is called. Delaying + this message until conntrack confirmation happens saves us from + replicating otherwise unneeded state information. +\item + {\bf NF\_CTSRP\_UPDATE} need to be created automagically by the + conntrack core. It is not possible to have any failover-specific + code within conntrack protocol and/or application helpers. + The easiest way involving the least changes to the conntrack core + code is to copy parts of the conntrack entry before calling any + helper functions, and then use memcmp() to find out if the helper + has changed any information. +\item + {\bf NF\_CTSRP\_EXPIRE} can be added very easily to the existing + conntrack destroy function. +\end{itemize} + + +\subsubsection{Connection tracking state syncronization receiver} + +Impmentation of the receiver is very straightforward. + +Apart from dealing with lost CTSRP packets, it just needs to call the +respective conntrack add/modify/delete functions offered by the core. + + +\subsubsection{Necessary changes within netfilter conntrack core} + +To be able to implement the described conntrack state replication mechanism, +the following changes to the conntrack core are needed: +\begin{itemize} +\item + Ability to exclude certain packets from being tracked. This is a + long-wanted feature on the TODO list of the netfilter project and will + be implemented by having a ``prestate'' table in combination with a + ``NOTRACK'' target. +\item + Ability to register callback functions to be called every time a new + conntrack entry is created or an existing entry modified. +\item + Export an API to add externally add, modify and remove conntrack + entries. Since the needed ip\_conntrack\_lock is exported, + implementation could even reside outside the conntrack core code. +\end{itemize} + +Since the number of changes is very low, it is very likely that the +modifications will go into the mainstream kernel without any big hazzle. + +\end{document} |