%include "default.mgp" %default 1 bgrad %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page %nodefault %back "blue" %center %size 7 How to do Embedded Linux [not] right Infineon / February 2009 %center %size 4 by Harald Welte netfilter.org / openmoko.org / openpcd.org gpl-violations.org / openezx.org / gnufiish.org berlin.ccc.de / openBSC.gnumonks.org deDECTed.org / hmw-consulting.de / viatech.com %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Introduction Who is speaking to you? an independent Free Software developer, consultant and trainer 15 years experience using/deploying and developing for Linux on server and workstation 12 years professional experience doing Linux system + kernel level development strong focus on network security and embedded expert in Free and Open Source Software (FOSS) copyright and licensing digital board-level hardware design, esp. embedded systems active developer and contributor to many FOSS projects thus, a techie, who will therefore not have fancy animated slides ;) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Introduction Why am I qualified? The 'Linux community' POV Former kernel subsystem maintainer (netfilter/iptables) Initiator of OpenEZX project Author of various drivers for embedded hardware The 'embedded Linux done the rigth way' POV Lead System Architect (SW+HW) at Openmoko, Inc. Co-creator of Open Hardware + Software for RFID OpenPCD, OpenPICC, librfid, libmrtd The 'chip manufacturer' POV Open Source Liaison at VIA Technologies, Inc. The 'customer of consumer-grade embedded Linux' POV Done reverse-engineering on hundreds of devices for gpl-violations.org %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Linux is everywhere! %image "linux_netfilter_singapore_entertainment.jpg" %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Linux is everywhere Linux is everywhere! Linux mobile phones (Motorola, Openmoko) In-flight entertainment systems Embedded networking gear DSLAMs rack monitoring Public payphones ATM's / PoS / vending machines Now even Fitness gear (Daum Ergometer) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Strengths of FOSS [0/4] What are the true strengths of FOSS Innovative and creative development Security due to code review / bugreport / patches Long-term maintainable code Stable and reliable systems %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Strengths of FOSS [1/4] Innovative and creative development easy-to-read existing codebase standard (FOSS) development tools thus, easy to modify and add features community will build around great new features/apps %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Strengths of FOSS [2/4] Security due to code review / bugreport / patches all the code is out there many people are familiar with existing architecture code quality requirements usually very high community process allows quick and fast integration of bugfix %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Strengths of FOSS [3/4] Long-term maintainable code, because there's a lot of attention on good software architecture many developers are familiar with the shared/common API's code quality requirements usually very high all code in mainline gets maintained and updated %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Strengths of FOSS [4/4] Stable and reliable systems, because code quality requirements usually very high kernel releases are quite frequent all mainline code is automatically ported to new releases %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Reality Check So we should have the perfect world tons of embedded Linux products all of them maitainable, secure, stable encouraging lots of creative work on top of their codebase What is the reality tons of embedded Linux products none of the strengths of FOSS present in 99% of them %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Differences to PC Linux Differences to Linux on a PC In the PC world, I can download the latest kernel from kernel.org compile + install it on almost every current+old platform have an almost 100% chance that it will boot and support all the major peripherals only some more obscure hardware might not have drivers yet update at any time to the latest 2.6.x.y maintenance release update at any time to the next 2.6.(x+1) release %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Differences to PC Linux Differences to Linux on a PC In the PC world, I can take about any major Linux distribution, based on my own preference install and run that very distribution on about any hardware distribution kernels are very close to mainline these days benefit of regular security updates by distributions %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Differences to PC Linux Differences to Linux on a PC In the Embedded world every CPU/SoC maker runs their own kernel tree often one kenrel tree per product, based on different mainline versions ages-old base revisions a neverending security nightmare no benefit from recent new features in mainline non-standard subsystems (e.g. different USB device or SDIO stack) proprietary drivers cause lock-in to old kernels %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Differences to PC Linux Differences to Linux on a PC In the Embedded world I often do not have a choice of which distro to run There might actually be no distribution No regular security updates Often no package management for deploying those updates If there are distributions, they either need to use the kernel from the BSP (which is ages old) or creare yet another custom off-mainline branch/port %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Differences to PC Linux %center THIS SUCKS! %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right What does the vendor get So what do the embedded vendors get? unstable software security nightmares unmaintainable code no innovation no user-contributed bug fixes %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right What does the vendor get unstable software because the code really sucks in many cases because they patch around problems rather than solving them %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right What does the vendor get security nightmares because they use stoneage forks of the kernel because they never contributed their code mainline because those forks can never be merged back with mainline again %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right What does the vendor get unmaintainable code because they have one fork of the code per device (product) because their code quality sucks %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right What does the vendor get no innovation because they try to hide their code (gpl-violations.org) because their R&D environment is non-standard weird cross-toolchains that nobody has seen before weird filesystems with custom patches that nobody knows because they add proprietary components to lock developers from adding features e.g. the entire web-based UI for embedded networking gear binary-only kernel modules that force people to use old kernels with no interesting new features because it is, overall, way too hard to develop on/for their platform because they don't disclose serial console and/or JTAG access %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Reality Check So why do they still do it? there can only be one conclusion: they never understood the real potential of FOSS! all they do is try to compete with what proprietary competitors do they never think about creating platforms, every product is distinct/separate they have no interest in improving their products %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right What does the community get So what does the community get? products that run some crappy fork of Linux somewhere under the hood but which we cannot really touch/modify without a lot of effort we face opposition from the product maker if we want to help him to improve %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right TODO (devica maker) What should the device vendor do? stop thinking in terms of selling black boxes defining products that take advantage of the true strength of FOSS compete against the proprietary competition on a level that they can't match give up the idea of defining all aspects of an appliance rather think of building an extensible platforms and let community innovate %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right A note to chip vendors There are two types of customers The Linux-aware customer understands FOSS much better than you do will share the criticism of this talk will likely go to a competitor who understands Linux better The Linux-unaware customer who just uses Linux to save per-unilt OS royalties who doesn't really care about most issues presented here who will create inferior products Linux-awareness is increasing, not decreasing now is already late, but if you don't have proper FOSS support on your agenda now, you will likely loose the "openness competition" %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right TODO (chip maker) What should the chip vendors do? engange in "sustainable development" develop against latest mainline make your development trees public (use git!), don't just release stable snapshots as BSP to your tier-one customers actively interact with the community learn how to play by the rules (coding style, use common interfaces, no proprietary drivers) don't just do big code drops every now and then since most of your customers' skill is very low they cannot tell you what they actually want/need on a technical level you have to understand Linux better than them, and better than the competition %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right TODO (chip maker) What should the chip vendors do? Externally Don't mistake FOSS as just a technology. It is a R&D philosophy! Provide public reference manuals with no NDA If you have to resort to NDA, make sure they are FOSS friendly Ensure you don't license IP cores that conflict with FOSS Internally Draft a proper in-house FOSS strategy with clear goals Don't expect your product mangers or engineers to know everything about FOSS without proper training Hire people with strong community background into your R&D and managemnt to facilitate the know-how transfer %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Lessons Learnt (chip maker) Chip Product Managers need to learn There's more Linux use with their chips than they ever learn Their customers are not just the tier-one customers If you want to support Linux, do it the mainline way. If you support only N number of distributions, your 'N' will be growing and you'll be wasting R&D resources to support each one of them That there is no single 'contact window' / entity for Linux The big change is the FOSS development model, not the Linux API's Linux-aware customers care not only about performance+price, but also about the quality of the Linux port code %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Lessons Learnt (chip maker) BSP R&D Managers need to learn Linux is not just a set of API's for their developers to learn Linux is FOSS. The FOSS R&D model is different Their engineers need to be encouraged to communicate and thus, need real internet (git, mailing lists, ..) have to get clear indication what is confidential and what not FOSS is about cooperation. You don't need to reinvent the wheel don't include your own 802.11 stack in your wifi driver If you do not merge your code mainline, you end up in a maintenance nightmare %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Lessons Learnt (chip maker) BSP Software Engineers need to learn Linux is not just a set of API's How and where to ask the right kind of questions How and where to look for the latest code Code is written to be read by other people, not just to execute %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right TODO (community) What should the community do Provide non-partisan documentation on FOSS advantages, proper FOSS development hardware companies are intrested to learn, but don't know who to ask. If they ask a commercial distributor, then they get in bed with them, which is not the same as working with the mainline development community Something like a mentoring program take software/driver R&D by their hand and walk them through the mainline merge Don't scare them away They have to be taught about the communication / review culture Your valid criticism to patches has to be explained %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Problems (chipmaker) Patent licensing schemes e.g. MPEG patent license doesn't at all work in a FOSS model Technology licensing schemes e.g. Sony Memorystick. Impossible for a chip maker to provide FOSS driver Everyone in the industry has those very same problems Chip makers should cooperate to present their case together to the respective licensors %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Problems (chipmaker) Patent trolls openly accessible documentation invites patent trolls Why? patent trolls rarely read+understand FOSS code thus, open documentation increases the risk to be hit However this is a by-product of how the patent system currently works if there is a patent violation in the product, then it exists whether or not there is documentation %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Outlook Outlook We will see even more embedded Linux, e.g. Mobile phones We will see even more restricted devices (Tivo-ization, DRM for code) which go on _very_ thin ice even with GPLv2 which almost completely remove all freedoms of FOSS We have some faint dim light at the end of a very far tunnel e.g. projects like Openmoko, who truly see openness as a feature e.g. chip makers who slowly open up a bit more on the PC side, Intel defiitely is setting the best overall example on the Embedded side, I see some - but way too small - imovement in major players (TI, Marvell, Samsung, Infineon, ...) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right The Embedded SoC market In the Embedded SoC market your typical customer is a chinese company with very limited technical skill and resources almost all products are equal, market extremely competitive thus, licensing proprietary OS or software stack is not feasible thus, SoC makers get pushed to provide not only base OS/drivers, but full 'turnkey solution' reference design which also means your actual SoC customer does not understand Linux, neither technically nor the Open Source model (and its benefits) if you are the first to provide a solution that fully embraces Open Source, your products will be more stable and feature-rich than the competition %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Questions Questions to the audience: Do you really enter a legal grey area and make life hard for yourself and your customers/users by doing binary-only kernel modules? Do you want to ignore the security benefits from open source? Can you really afford to not tap into the creative power and workforce of the FOSS community? Do you really want to use Linux just as a cheap replacement for other OS, or do you want to embrace its true strength? Competing with openness against proprietary competitors is competing at a level they can never match! %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %page How to do Embedded Linux [not] right Thanks Thanks for your attention. Some Time left for Q&A Feel free to mail me at laforge@gnumonks.org