%include "cnc-style.mgp" %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page %nodefault %pcache 1 1 0 1 %size 7, font "standard", fore "white", vgap 20, back "black" %bimage "fundo-cnc.png" 1024x768 %center %size 7 Quality of Service in IP Networks %center %size 4 by Harald Welte %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks Contents Definition of QoS Why QoS IP Networks are not designed for QoS How to do the impossible What can Linux based systems help Advanced Concepts (DiffServ, IntServ, RSVP, ...) References / Further Reading %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks Definiton of QoS Provide Service Differentiation Performance Assurance by Bandwitdh guarantees for streaming multimedia traffic priorizing certain important applications Latency guarantees for voice over IP for interactive character-oriented applications (ssh,telnet) Packet-loss guarantees for unreliable layer-4 protocols to avoid retransmits %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks Why QoS Decide how and who available bandwidth is devided Limit available bandwidth for certain users / applications Guarantee bandwidth for certain users / applications Divide bandwidth more equally between users / applications %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks IP networks not designed for QoS Properties of IP-based networks: offer a "best-effort" service make NO guarantees about bandwidth latency packet loss provide a non-reliable packet transport Conclusion: IP networks are not suitable for QoS %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks How to do the Impossible %size 4 As IP Networks including Hardware (Routers, ...) are widely deployed, all QoS efforts have to layer on top of the existing technology. There's no real solution to control latency latency widely dependent on routing, which may be dynamic There's no real solution to control packet loss packet loss may occurr on any intermediate router But we can control bandwidth usage! The sender can limit bandwidth for outgoing streams Intermediate routers BEFORE a bottleneck can control bandwidth usage %size 5 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks What can Linux systems do? Bandwidth limiting at the sender application not many applications support it server often out of control (on Internet, ...) server doesn't know what's between him and the client Bandwidth control on intermediate router before bottleneck Ideal case because this is where packet loss would occurr Sophisticated queue scheduling on the outgoing queue Variety of different queue scheduling algorithms Flow throttling at the Receiver Worst case, because influence is limited Theoretically possible for TCP, no implementation yet. Ingress qdisc might help %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks Bandwidth limiting at server Some Internet Servers support bandwidth limiting ProFTPd (builtin support) Apache (using contributed mod_bandwidth) Using those features it is easy to limit maximum bandwidth used per connection maximum bandwidth used per client (IP/network) maximum bandwidth used by one virtual host (webserver/ftpserver) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks Router before bottleneck %size 4 The router receives more packets on his incoming interface(s) than it can send out on the outgoing interface. It has to build a queue of packets (usually a FIFO one) and starts dropping packets as soon as the queue is full %image "qos-1.png" 0 100 30 The idea is to change this queue, thus decide which packets get enqueued in which order how many packets get queued which packets get dropped in case of a filling queue %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks The Linux 2.2 / 2.4 Solution Packet Scheduling algorithms in the Kernel CBQ - Class Based Queue RED - Random Early Drop SFQ - Stochastic Fairness Queueing TEQL - True Link Equalizer TBF - Token Bucket Filter tc command of iproute2 package for configuration almost no documentation very few examples on the internet Packet Classification tc builtin classes (route, u23, ...) all iptables/netfilter matches by using fwmark Conclusion: Linux is the best suited general-purpose operating system for QoS, but almost nobody is using it because lack of knowledge. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks Available queuing algorithms CBQ - Class Based Queue hierarchical bandwidth classes used as basis in almost all cases TBF - Token Bucket Filter really accurate algorithm uses a lot of CPU not possible for high bandwidth links (>1MBit) SFQ - Stochastic Fairness Queueing less accurate algorithm tries to distinguish between individual streams does round robin between those streams TEQL - True Link Equalizer allows to 'bundle' interfaces RED - Random Early Detect / Drop simulates congested link by statistic packet dropping uses almost no CPU recommended for high-bandwidth backbones others (WRR, TCINDEX, DSMARK, ..) WRR not officially included in kernel, similar to CBQ others mostly used for DiffServ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks The big picture Overview of the a packet's journey %size 3 %font "typewriter" Incoming Packets %size 3 %font "typewriter" | %size 3 %font "typewriter" V %size 3 %font "typewriter" Packet Classification classify %size 3 %font "typewriter" (ipchains/iptables) set nfmark %size 3 %font "typewriter" | %size 3 %font "typewriter" V %size 3 %font "typewriter" Routing decision %size 3 %font "typewriter" | %size 3 %font "typewriter" V %size 3 %font "typewriter" TC filter select classes based on nfmark %size 3 %font "typewriter" / | \ %size 3 %font "typewriter" / | \ %size 3 %font "typewriter" / | \ %size 3 %font "typewriter" Different Bandwidth classes bandwidth classes (CBQ) %size 3 %font "typewriter" \ | / %size 3 %font "typewriter" \ | / %size 3 %font "typewriter" \ | / %size 3 %font "typewriter" Enqueuing output queue discipline %size 3 %font "typewriter" | %size 3 %font "typewriter" V %size 3 %font "typewriter" Outgoing packets %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks Example scenario usin CBQ %size 4 Let's assume we have a link with 10 MBit maximum available bandwidth. We offer two major services to the outside world: Anonymous FTP and a Webserver offering important Information. FTP Bulk data transfers are using up almost all available bandwidth, thus slowing down accesses to our website :( We want to have FTP transfers use up to 8MBit and reserve 2MBit for WWW. Implementation uses CBQ for bandwidth divisions. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks Example scenario %size 3 attach a CBQ to the device %size 3 %font "typewriter" tc qdisc add dev eth0 root handle 10: cbq bandwidth 10Mbit avpkt 1000 %size 3 %font "standard" create CBQ classes %size 3 %font "typewriter" tc class add dev eth0 parent 10:0 classid 10:1 cbq bandwidth 10MBit rate 10MBit allot 1514 weight 1Mbit prio 8 maxburst 20 avpkt 1000 tc class add dev eth0 parent 10:1 classid 10:100 cbq bandwidth 10MBit rate 8MBit allot 1514 weight 800kbit prio 5 maxburst 20 avpkt 1000 bounded tc class add dev eth0 parent 10:1 classid 10:200 cbq bandwidth 10MBit rate 2MBit allot 1514 weight 200kbit prio 5 maxburst 20 avpkt 1000 bounded %size 3 %font "standard" add filter rules %size 3 %font "typewriter" tc filter add dev eth0 parent 10:1 protocol ip handle 6 fw classid 10:100 iptables -t mangle -A PREROUTING -j MARK -p tcp --sport 20 --set-mark 6 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks Further optimization %size 4 Now we have achieved bandwidth division between two services. Within one service, however, one individual user with a high bandwith link can still use up most of our bandwidth, slowing down other user. We can improve this behaviour of changing the scheduling algorithm from it's default (fifo) %size 3 %font "typewriter" tc qdisc add dev eth0 parent 10:100 sfq quantum 1514b perturb 15 tc qdisc add dev eth0 parent 10:200 sfq quantum 1514b perturb 15 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page QoS in IP Networks Further reading / Links Bandwidth limiting on Servers ProFTPd http://www.proftpd.net/ Apache mod_bandwidth / mod_bwshare ftp://ftp.cohprog.com/pub/apache/module/mod_bandwidth.c http://www.topology.org/src/bwshare/ Queue scheduling Advanced Routing HOWTO http://www.ds9a.nl/2.4Routing/ Linux QoS HOWTO http://www.ittc.ukans.edu/~rsarav/howto/ iproute2+tc This presentation Authors Homepage http://www.gnumonks.org/