summaryrefslogtreecommitdiff
path: root/2017/netfilter-netdev2.2/netfilter-keynote.adoc
blob: d75dee25115bfe8dc559dd231b57ef586269c17a (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
netfilter archeology: 18 years from 2.3 to 4.x
==============================================
:author:	Harald Welte <laforge@gnumonks.org>
:copyright:	2017 by Harald Welte (License: CC-BY-SA)
:backend:	slidy
:max-width:	45em

== What is this about

[role="incremental"]
* netfilter history
* netfilter who-is-who
* netfilter anecdotes
* netfilter folklore
* netfilter world domination


== Context

[role="incremental"]
* late 1990ies
* Internet was still new to many people
* Internet Security was still rather new
** think e.g. of the "ping of death" problems.I
* no git, not even subversion, but: CVS(!)
** even pre-bitkeeper, so no kernel global revision control
* no virtual machines, testing on physical boxes, long boot cycles
* no authorship annotation / commit history

== pre-netfilter

The pre-netfilter days

[role="incremental"]
* Linux 1.2, 1.3 and 2.0 had `ipfwadm` (Jos Vos et al)
* Linux 2.2 had `ipchains` (Rusty Russell)
** Rusty was doing some sysadmin work at an ISP and was doing his job so well that he had plenty of spare time
** He was _immensely_ inspired by a talk by DaveM on beating the hell out of Solaris on SPARC
** wanted to do more Linux stuff, met WatchGuard
** proposd to do a proper redesign of the Linux firewall if they pay him for 6-12 months
** ... which they did, so mid-1998 to mid-1999, he hacked away on it.

== Creation Timeline

[quote, Rusty Russell]
Who the hell are you, and why are you playing with my kernel?
I want to clear up some people's misconceptions: I am no kernel guru. I know this, because my kernel work has brought me into contact with some of them: David S. Miller, Alexey Kuznetsov, Andi Kleen, Alan Cox. However, they're all busy doing the deep magic, leaving me to wade in the shallow end where it's safe.

[role="incremental"]
* July 20, 1998: Rusty posts initial netfilter design to netdev list
* January 29, 1999: netfilter v0.1 released 
* August 26, 1999: netfilter included in kernel 2.3.15
* November 1999: _core team_ established with Marc Boucher + Rusty Russell

== netfilter v0.1

----
Date: 1999-01-29 10:36:34
From: Paul Rusty Russell <Paul.Russell@rustcorp.com.au>
Subject: [ipchains-dev] netfilter v0.1 released.

Hi all,

Just in case people are sick of all this "stable kernel" crap, and want their interesting behaviour back, here's the first alpha-quality cut of my new firewall/NAT/masquerading/transproxy/redirect/portforward framework, from
        ftp://ftp.rustcorp.com/netfilter/netfilter-1999-01-29.tar.bz2

This work is a result of the vast amount of user feedback I've had on things such as transparent proxying, masquerading, ipfwadm, ipchains, etc.  Includes a 400k patch against 2.2.0. (84 files changed, 2509 insertions, 12097 deletions).

It's all set in slush at this stage, so if you have any comments, please feel free to leap forward and abuse me...
----

In other news of January 1999:

* Yahoo bought Geocities
* Clinton faced impeachemnt trial


== netfilter merged in kernel 2.3.15

----
    From: Linus Torvalds <torvalds@transmeta.com>
 Subject: Linux-2.3.15..
    Date: Wed, 25 Aug 1999 16:36:10 -0700 (PDT)

There's a rather huge patch-set out there now, taking the 2.3.x series to 2.3.15.  [...]

Other features that don't impact everybody, but are rather major:

* firewalling is gone (again), replaced by an even more generic netfilter facility.
[...]

Have fun,
  Linus
----

In other news:

* East Timor becomes independent of Indonesia
* Vladimir Putin becomes Prime Minister of Russia

== Rusty (at Linux Beer Hike 2000)

image:rusty2000.jpg[]

== Marc (at OLS 2000)

image:marc-ols2000-zoom.png[]

== Core Team Timeline

* November 1999: _core team_ established with _Marc Boucher_ + _Rusty Russell_
* Sydney Linux Expo: _James Morris_ joins core team
* September 2000: _Harald Welte_  joins core team
* November 2001: _Jozsef Kadlecsik_ joins core tema
* August 2003: _Martin Josefsson_  joins core team
** Rusty, Marc and James become _emeritus_ members
* January 2004: _Patrick McHardy_ joins core team
* October 2005: _Yasuyuki Kozakai_ joins core team
* February 2007: _Pablo Neira_ joins core team
* October 2012: _Eric Leblond_ and _Florian Westphal_ join core team
** Harald, Martin and Yasuyuki enter _emeritus_ status

== James (in 2008)

image:james.jpg[width="60%"]

== Humor

From http://www.netfilter.org/about.html#history

Following James' assimilation into the collective, our efforts were mainly directed towards preparations for
the release of Netfilter as part of the upcoming 2.4 kernel. _It was the dawn of the third age of Linux
firewalling; a time of great struggle and heroic deeds. It was our last, best hope for peace. Great
communities were founded, old civilizations were lost, and new alliances were formed._ James' missions during
this period included the _continued perversion of the networking code_, such that it was now possible to load
an ASN.1 parser into the kernel and _inflict grave terror upon unsuspecting SNMP packets_; and to extend the
IP stack into userspace with Perl. _Now peering squarely into the abyss, we noticed the good deeds of a young
kernel warrior_ named Harald Welte, who seemed to actually understand the NAT code.  Accordingly, his
distinctiveness was added to the collective. _With balance restored, the netfilter juggernaut was now free to
accelerate into the brave new world of Linux 2.4 and face it's greatest challenge: users._


== Humor

----
Date: Fri, 13 Oct 2000 16:26:06 +1100
From: Rusty Russell <rusty@linuxcare.com.au>
To: netfilter@lists.samba.org, netfilter-devel@lists.samba.org
Subject: [CORE TEAM] New Member Announce

The Netfilter Core Team is proud to welcome Harald Welte into its hallowed botherhood.

Harald Welte has frequently answered user questions on the mailing list, and authored the IRC connection tracking and NAT modules.  He even documented what he'd done!  And then fixed some of the bugs!

This shocking and revolutionary approach to software development will fill a much-needed void in the Netfilter Team.  Assuming he survives the inauguration ceremony.
----

In other news:

[role="incremental"]
* Bill Gatest steps down as CEO of Microsoft

== Harald

image:laforge-hat.jpg[]

== Jozsef

----
Date: Fri, 7 Dec 2001 21:19:57 +1100 (EST)
From: James Morris <jmorris@intercode.com.au>
Subject: [netfilter-announce] [ANNOUNCE] New Core Team Member - Jozsef Kadlecsik

The Netfilter Core Team is proud to announce the addition Jozsef Kadlecsik as a new member.

Jozsef joins us as a dedicated and talented member of the Netfilter development community.

His demonstrated insight and high coding standards will be highly valuable assets to the project as development focus shifts to the 2.5 kernel series.

Welcome Jozsef!

- James, on behalf of the Netfilter Core Team.
--
James Morris
<jmorris@intercode.com.au>
----

== Jozsef

image:jozsef.jpg[width="40%"]

== Martin

image:video-not-available-youtube-error.png[]



== Documentation

One key aspect was lots of good, easy to read documentation

* netfilter hacking HOWTO
* netfilter extensions HOWTO
* Linux 2.4 Packet Filtering HOWTO
* Linux Networking-concepts HOWTO
* NAT HOWTO

Getting into the project as both a user or developer was helped enormously by the HOWTOs.

The original versions of those documents were all created in early 2000.


== The netfilter scoreboard

* a _scoreboard_ was established
** high-score for number of patches/contributions
** counts not only code but also documentation updates
** manually maintained by scoreboard
** bonus points for patches that apply to correct version
* motivation for developers, particularly junior ones!

Remember, this was the pre-git and even pre-bitkeeper days!

Guess these days, people would count this as _gamification_?

== The netfilter scoreboard (April 2002)

image:netfilter-scoreboard-20020408.png[]

== The netfilter scoreboard (April 2002)

image:netfilter-scoreboard-andras.png[]



== modularity / extensibility

* netfilter is just a set of hooks for call-back functions
* iptables matches and targets are just plug-ins for both kernel and userspace
* good documentation on the APIs and how to write one
* get people involved, implement their favorite feature

Problem: How to distribute / maintain them?

== patch-o-matic

----
1) The netfilter core team is maintaining a set of extensions / new·features which are not yet committed to the mainstream kernel tree.

They are a collection of maybe-broken maybe-cool third-party extensions.

Please note that you cannot apply any combination of any of those patches.  Some of them are incompatible...·

If you want to try some extensions, and be sure that they don't break each other, you can do the following:

  % ./runme base KERNEL_DIR=<<where-you-built-your-kernel>>

It will modify you kernel source (so back it up first!).  You will have to recompile / rebuild your kernel and modules.

Alternatively, if you really know what your are doing, you can use the following command in order to offer you the full list of choices.  Be aware that we don't prevent you from shooting yourself in the foot.

  % ./runme extra KERNEL_DIR=<<where-you-built-your-kernel>>
----

== patch-o-matic

----
Date: Mon, 30 Oct 2000 14:28:30 +1100
From: Rusty Russell <rusty@linuxcare.com.au>
To: Netfilter Development Mailinglist <netfilter-devel@us4.samba.org>

On Mon, Oct 23, 2000 at 09:15:23PM -1100, Daniel Stone wrote:
> This, to me, reflects a problem. Basically, I can only see two things causing this:
> a) no testing at all, or
> b) a mis-paste. Please tell me it was the latter.

Completely untested.  I looked at the patch as I threw it into patch-o-matic.

That's what patch-o-matic is for: to get stuff out there without waiting for the Rusty Linus planet alignment thing...

Rusty.
--
Hacking time.
----

== Early Success

What contributed to the early success with lots of devlopers wriing netfilter/iptables code:

* loads of good documentation
* modular framework with
** netfilter hooks
** pluggable iptables targets + matches
* system for maintaining non-mainline code + merging it

=> Everyone could easily write his favorite match/target/plugin


== core team emeritus members

----
Date: Fri, 09 Jan 2004 15:17:19 +1100
From: Rusty Russell <rusty@rustcorp.com.au>
Subject: [ANNOUNCE] Core Team Announces Emeritus Members

The Netfilter Core Team has long discussed the issue of Core Team members who are no longer active.  Dismissing them from the Core Team would deny them the benefits of such a prestigious title, should any become apparent.

Hence the conclusion is that Marc Boucher, James Morris and Rusty Russell are now "emeritus"[1] members of the Netfilter Core Team.

[...]

[1] Latin for "burnt-out freeriding slacker", I believe.
----

== the story behind

[role="incremental"]
* Until recently, I thought
** Rusty simply had too many other tempting distracting projects (kernel module loader, qemu, ...)

[role="incremental"]
* Recently, Rustyy told me
** it was a deliberate decision to leave netfilter
** the new core team and maintainers should run the project without interference from the project father


== Pablos first messages

----
Date: Wed, 12 Nov 2003 00:06:11 +0100
From: pablo neira <pablo@eurodev.net>
To: netfilter-devel@lists.netfilter.org
Subject: ip_conntrack_get

Hi everyone,

I've been for almost two weeks trying to understand netfilter code, at this point I'm trying to understand conntrack table code.

I have a problem with how conntrack manages the ip_conntrack_info stuff.
[...]

Don't blame me if it's obvious for you, I'm just a guy trying to understand a *really really nice piece of code*. Thanks!

cheers,
Pablo
----

== Pablos first messages

----
Date: Wed, 03 Dec 2003 15:23:39 +0100
From: pablo neira <pablo@eurodev.net>
To: netfilter-devel@lists.netfilter.org
Subject: sending event to user space

Hi list!

I programmed a dummy module for netfilter which tries to match a packet and if it does, it will do nothing (NF_ACCEPT) but I would want it to send an event to a program in user space to do something, how can I do that?

So something like:

a) packet gets my hook and do NF_ACCEPT.
b) modules sends an event to user space.
c) program in user space does something.

Thanks!
Pablo

P.S: thanks for this *great piece of code*!!
----


== Pablo (fast forward four years)

----
Date: Thu, 15 Feb 2007 14:02:03 +0900 (JST)
From: Yasuyuki KOZAKAI <yasuyuki.kozakai@toshiba.co.jp>
Subject: [ANNOUNCE]: New Coreteam Member Pablo Neira Ayuso

The Netfilter Core Team is proud to announce the addition of Pablo Neira Ayuso as a new member.

He has repeatedly demonstrated high insight and coding standards, and has already been responsible for several parts of the codebase, especially ctnetlink, conntrack and conntrackd.

By joining the Core Team, Pablo will definitely help advance the development of the Netfilter project to a higher level.

Welcome Pablo!

Yasuyuki,
on behalf of the Netfilter Core Team.
----

== Pablo (2009)

image:pablo-2009.jpg[width="60%"]

== Harald (2006)

As I'm showing various old pictures of other people, for fairness' sake...

image:harald-2006.jpg[width="30%"]



== bugs

----
Date: Mon, 14 Jan 2002 22:33:16 +1100
From: Rusty Russell <rusty@rustcorp.com.au>
To: "David S. Miller" <davem@redhat.com>
Subject: Re: Kernel 2.4.16: NetFilter Bug Report

In message <20020113.231425.73653921.davem@redhat.com> you write:
> Any ideas on that NAT timer one from cat@zip.com.au?  I've for now asked Marcelo to revert that 2.4.17 change until a different fix is obtained.

I am a fucking retard.

I was looking at what was wrong with the code, and came up with *five* separate problems.  I know the compat layer was a hack, but what the *fuck* was I doing?

Read and weep,
Rusty.
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
----

== nfsim / testsuite

Big problem with lots of code, including netfilter: Lack of automatic testing.

Rusty returns to netfilter with _nfsim_, a netfilter simulator, co-authored with Jeremy Kerr.

* nfsim runs netfilter kernel code in userspace against test suite
* emulates kernel environment in userspace
* imports netfilter kernel code + builds it in userspace
* {get,set}sockopt() wrapper for userspace tools
* can simulate allocation failures
* manual control over time (important for conntrack state tables)

=> Great Idea, and lots of useful work


== nfsim / testsuite

Reality sucks:

* very few contributions
* very few users beyond Rusty + Jeremy
* very limited adoption/use by netfilter developers

== nfsim / testsuite

----
Date: Tue, 31 May 2005 23:48:24 +1000
From: Rusty Russell <rusty@rustcorp.com.au>
To: Patrick McHardy <kaber@trash.net>

On Tue, 2005-05-31 at 15:02 +0200, Patrick McHardy wrote:
>  Second of all, I spent like 10 hours to verify the
> proposed fixes, and I am still convinced that it is correct.

Which shows exactly *why* we have a testsuite.  Dammit, I didn't spend
all those hours on it for fun.

You spent *10* hours, and the testsuite runs in 5 seconds (60 seconds
counting build time the first time).

<sigh>
----

== nfsim / testsuite

Reality sucks:

* patches get validated only in test suite, not real kernel

== nfsim / testsuite

----
Date: Sun, 23 Jan 2005 20:15:17 -0800
From: "David S. Miller" <davem@davemloft.net>
Subject: Re: [PATCH 3/2] Fix compile with NAT but without modules

On Mon, 24 Jan 2005 01:33:47 +0100 Patrick McHardy <kaber@trash.net> wrote:

> I'll apply your patches and push them to Dave tomorrow, bkbkits.com
> is unreachable currently so I can't resync my tree.

To be frank, this is one of several severe fallouts from Rusty's patches.  I really think they were not ready for submission when he sent them to me.  It even broke the build if you had modules enabled in any
way.

I'm only mentioning this because it appears that nfsim is becomming partially a crutch, because I know this is what Rusty and others use heavily for testing.  Which is fine, but if your patches break the build in many ways in the real kernel tree you're relying too heavily on the userland simulator IMHO.
----

== nfsim / testsuite

Reality sucks:

* very few contributions
* very limited adoption/use by netfilter developers
* bit-rot of kernel environment simulation
* constant lag in terms of completeness
** no netlink simulation, i.e. no nfnetlink/ctnetlink/nf_queue/nf_log

=> no replacement / successor, till today :(


== nfsim / testuite

So all we can do is join DaveM and pray for code correctness

image:davem-praying-zoom.png[]



== humor

----
Date: Wed, 26 Feb 2003 11:54:59 +1100
From: Rusty Russell <rusty@rustcorp.com.au>
To: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu>
Subject: [netfilter-core] Re: conntrack patches

> Hi Rusty,
>
> Last year I started to go trough all of your unpublished conntrack related
> patches. [...]

> Are you working on the patches or plan to finish them?
> Or can I go back and complete the half-done job on the patches?

Jozsef,
        Let me put it this way: take over those patches and I will name my first child after you.

How many beers did I owe you now?
Rusty.
----


== Harald and IPv6 NAT

----
Date: Thu, 20 Nov 2003 14:40:42 +0100
From: Harald Welte <laforge@netfilter.org>
Cc: netfilter-devel@lists.netfilter.org
Subject: Re: NAT for IPv6

On Wed, Nov 19, 2003 at 01:38:47PM +0100, Maciej Soltysiak wrote:
> out of curiousity - are there plans to incorporate NAT into ip6tables
> or future pkttables ?

over my dead body.  NAT is what broke ipv4 end-to-end.  Let's not do the
same with ipv6.

The only reasonable application is ipv4-to-ipv6 transition-nat.

--
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie
----



== nftables (2009)

----
Date: Wed, 18 Mar 2009 05:29:42 +0100
From: Patrick McHardy <kaber@trash.net>
To: Netfilter Development Mailinglist <netfilter-devel@vger.kernel.org>
CC: Linux Netdev List <netdev@vger.kernel.org>
Subject: [ANNOUNCE]: First release of nftables

Finally, with a lot of delay, I've just released the first full public
version of my nftables code (including userspace), which is intended to
become a successor to iptables. Its written from scratch and there are
numerous differences to iptables in both features and design, so I'll
start with a brief overview.

There are three main components:

- the kernel implementation
- libnl netlink communication
- nftables userspace frontend
----

== nftables (2013)

In 2013, nftables goes mainline!

----
commit 96518518cc417bb0a8c80b9fb736202e28acdf96
Author: Patrick McHardy <kaber@trash.net>
Date:   Mon Oct 14 11:00:02 2013 +0200
----

image:pablo-iptables-bye-zoom.png[]





== netfilter Workshops

* 1998/1999/2000: Informal meetings of some of the people involved
** like James + Marc + Rusty at Sydney Linux Expo
** like Harald + Rusty at Linux Beer Hike
* _workshop_ established from 2001 onwards to get developers meet up
* not every year, but almost: 13 workshops in 18 years
* invitation-only
* organization done by community for community
* sponsors typically among commercial netfilter users

== netfilter Workshops


* 2001: Enschede, Netherlands
* 2003: Budapest, Hungary
* 2004: Erlangen, Germany
* 2005: Seville, Spain
* 2007: Karlsruhe, Germany
* 2008: Paris, France
* 2010: Seville, Spain
* 2011: Freiburg im Breisgau, Germany
* 2013: Copenhagen, Denmark
* 2014: Montpellier, France
* 2015: Budapest, Hungary
* 2016: Amsterdam, Netherlands
* 2017: Faro, Portugal

== Workshop 2003: Group Picture

image:nfws2003_group.png[]

== Workshop 2005: Fun fact

image:unix-extinguisher-2005.jpg[]

== Workshop 2013

image:nfws2013.jpg[width="100%"]

== Workshop 2014

image:nfws2014.jpg[width="100%"]

== Workshop 2015

image:nfws2015.jpg[width="100%"]

== Workshop 2016

image:nfws2016.jpg[width="100%"]







== Thanks

* to the audience, for bearing with me
* to the netdev 2.2 committee, for inviting me
* to Rusty, for being my hero
* to Pablo, for picking up the pieces when I left
* to Dave, for being everyone's hero
* to regit, for group (and other) pictures
* to every single netfilter contributor out there

== EOF

End of File
personal git repositories of Harald Welte. Your mileage may vary