Dissecting modern (3G/4G) cellular modems ========================================= :author: Harald Welte, Holger Hans Peter Freyther :copyright: Harald Welte, Holger Hans Peter Freyther (License: CC-BY-SA) :backend: slidy :max-width: 45em //include::33c3-modems.css[] == This talk * Our motivation * A bit of History * Selecting a device * An unexpected surprise * Firmware upgrade * Outlook/Recommendations/Wishes == Motivation // 9 years of Osmocom? // 3G and 4G development // Hardware for decoding * Implementing GSM specifications for the last decade (OpenMoko, Osmocom) * 8 years since _Anatomy of Smartphone Hardware_ at 25C3 * 7 years since OsmocomBB for GSM * Used and built M2M devices using 2G modems at work * so we're looking for a modem that can be used for ** our next-generation M2M/embedded devices ** testing/logging/tracing Osmocom 3G/4G network-side software ** building more tools to help understanding cellular technology == Cellular Modems in M2M image:images/sl6087_hw.png[height=300,role="gimmick_right"] * Assume you want to build a M2M device * Classic approach to M2M/Embedded cellular: ** Cellular modem with AT commands over Serial/USB ** Main Processor runs M2M application * if you run Application in Modem, you can save PCB space, power and BOM cost ** OpenAT by Sierra Wireless *** Write C code using OpenAT APIs *** Dynamically loaded into the RTOS *** Runs without privilege separation, MMU *** Protocol to multiplex AT, log, debug *** Discontinued HW platform => Locked in *** Various other limitations == Device requirements Our requirements for a good modem ** Ability to run application code inside modem ** Avoid modem supplier vendor lock-in (EOL, ...) ** Get textual logging when handling messages ** Get a copy of the radio network messages and export to GSMTAP *** Like Tobias Engels https://github.com/2b-as/xgoldmon[x-goldmon] *** But for all GPRS, EGPRS, UMTS and LTE messages == Qualcomm DIAG protocol * Qualcomm DIAG in many products (DVB-H, GSM, ...) * https://events.ccc.de/congress/2011/Fahrplan/attachments/2022_11-ccc-qcombbdbg.pdf[Presented] by Guillaume Delugre at 28C3 * Simple HDLC frame (0x7e), cmd, data, CRC16 * Events, Logging, Command/Response * Thousands of different message structures * ModemManager, gsm-parser consume only a small fraction image:images/diag_frame.svg[width="90%"] == Selecting a device image:images/28c3_option_stick.png[width="30%",role="gimmick_right"] * Old Option Icon 225 stick exposes DIAG out of the box * Quectel UC20 (2G+3G) expose DIAG by default ** but no LTE support * Quectel EC20 (2G+3G+4G) expose DIAG by default ** 2G, 3G and 4G sounds quite nice ** EC20 not only a LGA solder module but also as mini-PCIe *** convenient for early testing / prototyping without custom board image:images/ec20.png[height=300,role="gimmick_right"] * EC20 using a Qualcomm MDM9615 chipset ** Also used in the iPhone5 ** Almost no documentation on MDM9615 available ** Still, a good candidate for starting our research... // Erst ein mal EC20 und sagen wieso es interessant ist // und dann, dass es Linux hat.. um dann ein Block diagram // zu haben? [role="change_topic"] == An unexpected surprise == Firmware update, hints of Linux * Got a firmware upgrade to fix stability / bugs * Looks like it contains traces of Linux? * Looks like it uses fastboot for the update * Other people have already found Linux in MDM9615 based products (e.g https://media.defcon.org/DEF%20CON%2023/DEF%20CON%2023%20presentations/DEFCON-23-Mickey-Shkatov-Jesse-Michael-Scared-poopless-LTE-and-your-laptop-UPDATED.pdf[Mickey Shkatov] at DEFCON 23) * But why would there be Linux inside a Modem? ** Qualcomm is known for their REX/AMSS on Hexagon baseband ?!? * And if it contains Linux, GPL requires them to mention that, include License text and provide source code ?!? == GPL compliance * No written offer, let's see if it runs Linux * Armijn Hemels `gpltool.git` has `unyaffs` to unpack yaffs * `strings`, etc. clearly reveal Linux, glibc, busybox ** other interesting strings like `AT+QLINUXCMD=?` show up * The fun and exploration begins... ** technical analysis (serial console, firmware reversing, ...) ** legal enforcement to get source code of GPL/LGPL components (Harald is founder of http://gpl-violations.org[gpl-violations.org]) == Hardware based analysis * mPCIe modules often expose additional signals like PCM audio on non-standard pins * existing PC/embedded mainboards don't use those signals * create Osmocom mPCIe-breakout board to access those signals * https://osmocom.org/projects/mpcie-breakout/wiki image:images/mpcie_breakout.jpg[width="70%"] == Serial Console * EC20 solder module documents DBG_UART pinout, but not all modules have it enabled? * serial console is at 1.8V, but the 1.8V supply is not accessible (so not easy to add external level shifter / Vref) * create Osmocom multi-voltage USB-UART with selectable 1.8, 2.3, 2.5, 2.8, 3.0 and 3.3V logic level image:images/mv_uart.jpg[width="40%",role="gimmick_right"] * https://osmocom.org/projects/mv-uart/wiki * root password (DES hash): `oelinux123` == Retro-fitting Serial Console to mPCIe module * unfortunately the DBG_UART on the LGA module solder pads is not exposed to mPCIE * some soldering required to retro-fit a 2.54mm header: image:images/ec20_uart.jpg[width="70%"] == GPL compliance * Linux basis created by Qualcomm and used by Quectel ** https://wiki.codeaurora.org/xwiki/bin/QLBEP/ ** Many branches, releases, which to use? [quote, Tonino Perazzi] I tried instruction above to build yaffs2 for MDM9615, so I downloaded source `M9615AAAARNLZA1611161.xml` but during compilation I faced some libs that are missing such as libQMI and acdb-loader.. image:images/qualcom_many_releases.png[width="80%"] == GPL compliance [qanda] Asking for the complete and corresponding source:: [quote,Quectel] ** The source code of Qflash tool in Linux is attached, [...] [qanda] Asking again for the complete and corresponding source:: [quote,Quectel] We never been in legal dispute and we always make sure to understand IPR ahead of using technology belonging to third party. image:images/quectel_ipr.jpg[width="100%"] == GPL compliance [qanda] Asking for the complete and corresponding source:: [quote,Quectel] We appreciate the efforts that your client had put into the open source project netfilter/_iptable_. However, [...] *your client does not have the right to empower the copyright*. We think software netfilter/iptable is built on the code operating system _GUN_/Linux, thus subject to GPL terms, where FSF requires that each author of code incorporated in FSF projects either provide copyright assignment to FSF or disclaim copyright. Therefore, It seems that *your client does not have the copyright on netfilter/iptable.* + + As one of the leading providers of wireless solution, *Quectel is always respectful IPR*. We would like to compliant with GPL and do some necessary statements，including a disclaimer or appropriate notices. Under the terms of GPL, we would like to dedicate Kernel code of EC25x to free software community. == GPL compliance [qanda] Asking for the complete and corresponding source:: [quote,Quectel] Many thanks for your detailed explanations GPL/LGPL license terms and the practical methods. I will carefully study your suggestions again and find a proper way to open GLP/LGPL licensed software. Basically, we will simply provide a tarball of open source for download at this time. And release the git repositories in next step. [qanda] Asking for the complete and corresponding source:: [quote,Quectel] We are always willing to achieve GPL compliance. [qanda] Asking for the complete and corresponding source:: [quote,Quectel] So we need some time to know of all things and construct the Open Source projects. Within a short time, we cannot construct a perfect web site to present Open Source things now. However, we will continue to do like that. == GPL compliance [qanda] Your tarball is missing some files:: [quote,Quectel] We have issued all GPL licensed source code. *We have no the xt_dscp file in the project, and nor Qulacomm*. It must be caused by your compilation environment. If you have more question or problem during the development with Quectel module, please add my Skype ID (XXXXX), I will continue to support you on Skype. + *The email will not discuss the compiling issue any more.* == GPL compliance * ... many months later ** we have received various source tarballs ** they contain not only GPL/LGPL code but other FOSS code (thanks!) ** full license compliance still not achieved, but improving... * Sierra Wireless Legato is a positive example of a competitor ** they not only provide the OE/Linux source but extensive documentation! ** but they try to lure customers into a proprietary Legato framework, and thus again vendor-lock-in :( image:images/legato_flash.png[width="80%"] [role="change_topic"] == MDM 9615 HW and SW == Qualcomm Hardware * Qualcomm MDM9615 chipset * Used in the iPhone 5 and automotive * Modems like Quectel EC20, Sierra Wireless MC7355 * No public HW documentation?! * Either not many people study it or are not allowed to share? == MDM 9615 HW Overview * ???? // Block diagram? // Listing of interfaces. // Show it is a highly complex SoC... with even more things // that are unknown.. device tree file, peripheral, etc == How to access the system? * serial console requires soldering re-work and is slow * easy mechanism to get shell and transfer files from/to target * Android `adbd` present on the modem but not exposed via USB * it's possible to re-configure the Linux kernel Android USB Gadget: ** `AT+QLINUXCMD="/usr/bin/usb_uartdiag"` ** device re-enumerates with different composite USB interfaces * Linux kernel driver on host needs patching (static interface mapping assumption) ** patches available in `quectel-experiments.git`, documented in wiki == MDM 9615 AP SW Overview image:images/gandroid_logo.png[height=200,role="gimmick_right"] The software stack seems to be called *Qualcomm LE* * Android Bootloader * Android Linux kernel * Android Debug Bridge (adb) * but: GNU libc, busybox userland * Using OpenEmbedded to build images * Developed and maintained by Qualcomm == Qualcomm Linux kernel overview * Qualcomm Android Linux kernel * Huge changes compared to mainline `git diff -w | wc -l` ** `v3.0.21` in EC20: 1.5 million lines ** `v3.18.20` in EC25: 1.9 million lines * Expected: CPU + peripheral drivers * Less expected: ** smem_log (shared memory logging) ** ipc_log (inter-processOR communication) ** remote spinlocks == Qualcomm Linux kernel subsystems Some of the Qualcomm-specific kernel sub-systems [cols="20%,80%"] |=== |SMD|Shared Memory Device |IPC|Inter Processor Communications |RMNET|Remote Network |BAM|Bus Access Manager |IPA|Internet Packet Accelerator |DIAGFWD|DIAG Forwarding |AF_MSM_IPC|Socket family for Qualcomm IPC |=== == Qualcomm LE System Architecture image:images/qualcomm_le.svg[width="50%",role="gimmick_right"] * simplified block diagram * USB interface fully controlled by Linux AP ** very complex Qualcomm Android USB Gadget ** some endpoints mapped to SMD queues ** other endpoints handled by _regular_ Linux ** GPS NMEA takes completely different path than AT commands, despite both being serial ports? ** DIAG and QMI handled in more complex ways == DIAG in Qualcomm LE * DIAG interface of Modem exposed on SMD * diagfwd distributes messages between USB, SMD and `/dev/diagchar` * Linux userspace processes don't use syslog, but diag msg for logging via `libdiag.so` image:images/diag.svg[width="100%"] == QMI in Qualcomm LE every `rmnet` data device has associated QMI control * on your Linux PC: `qmi_wwan` and `/dev/cdc-wdm` * on Qualcomm LE modem: `/dev/smdcntlN`, multiplexed by `qmuxd` image:images/qmi_smd_qmuxd.svg[width="100%"] == Tools for analysis We created some tools to help our analysis * used OE to build matching `opkg` and OE packages for `socat`, `lsof`, `strace` * FOSS programs for the Linux AP linked against proprietary `libqmi-framework.so` ** `qmi_test`: Simple program to read IMEI via QMI ** `atcop_test`: Test program to implement AT commands in Linux userspace * 100% FOSS programs ** `qmuxd_wrapper`: LD_PRELOAD wrapper for tracing between `qmuxd` and QMI clients ** `libqmi-glib` transport support for `qmuxd` (work in progress) ** `osmo-qcdiag`: Host tool for obtaining DIAG based logs from Linux programs + QMI traces, decoded via `libmi-glib` == Userspace programs We found a bunch of proprietary Linux userspace programs [cols="20%,80%"] |=== |`adbd`|Implements Android Debug Bridge |`atfwd_daemon`|Implement Quectel-Specific AT Commands |`quectel_daemon`|?; various ASoC related bits |`qti`|? |`mbim`|Mobile Broadband IF Model (translates MBIM to QMI) |`QCMAP_ConnectionManager`|runs linux-base WiFi AP/router with LTE backhaul |`quec_bridge`|reads GPS NMEA from `/dev/nmea` and writes it to `/dev/ttyGS0` |=== [role="change_topic"] == Funny bits + pieces == Funny AT commands * `AT+QLINUXCMD`, e.g. switch usb config to get adb ** arbitrary shell commands executed as root on r/w rootfs! * `AT+QFASTBOOT`, switch to the bootloader * `AT+QPRINT`, print dmesg * AT for `system("echo mem > /sys/power/state")` == How many processes does it take to reboot a system? * `rebootdiagapp` registers DIAG command (cmd code 0x29) ** spawns thread that runs `system("qmi_simple_ril_test input=/tmp/reset")` ** `system("echo 'modem reset' > /tmp/reset")` *** makes `qmi_simple_ril_test` send a QMI message to modem ** `system("rm /tmp/reset")` ** writes "REBOOT" to `/dev/rebooterdev` this time using `fwrite()`! * `reboot_daemon` reads `/dev/rebooterdev` ---- read_count = read(pipe_fd,buf,MAX_BUF-1); /* if read REBOOT_STR, then call reboot */ if(strncmp(buf,REBOOT_STR,strlen(REBOOT_STR)) == 0) { debug_printf("going for reboot\n"); printf("reboot-daemon: initiating reboot\n"); system("reboot"); } ---- == C programs that look like shell scripts * strings /usr/bin/quectel_daemon ---- echo "nau8814-aif1" > /sys/devices/platform/soc-audio.0/tx_dai_name cp -f /cache/usb/qcfg_usbcfg /etc/; cp -f /cache/usb/usb /etc/init.d/ echo 90 >/sys/kernel/debug/pm8xxx-pwm-dbg/0/duty-cycle pkill -f "/bin/sh /usr/bin/nmea_demon.sh" ps ef | grep "quec_bridge /dev/nmea /dev/ttyGS0" | grep -v grep cd /cache/ufs;ls ---- [role="change_topic"] == Firmware upgrade == recovery and applypatch * Qualcomm uses https://android.googlesource.com/platform/bootable/recovery.git/+/android-4.0.4_r2.1[recovery.git] from Android ~4.0 * Updates are zip files with deltas, SHA1+RSA * recovery started on boot, drives applypatch ---- // Look for an RSA signature embedded in the .ZIP file comment given // the path to the zip. Verify it matches one of the given public // keys. ---- == Qualcomm EC20 firmware upgrade image:images/redbend.png[width="30%",role="gimmick_right"] * Based on the recovery.git code * But for some reason using RedBend for the update (legacy?) * RSA still linked into the binary but not used * RedBend used by many more companies and systems (e.g. Quectel UC20, automotive) == RedBend (delta update) software * Used in OMA DeviceManagement as well? (e.g. https://www.blackhat.com/docs/us-14/materials/us-14-Solnik-Cellular-Exploitation-On-A-Global-Scale-The-Rise-And-Fall-Of-The-Control-Protocol.pdf[Mathew Solnik]) * Lots of starring at hexdumps, lots of help from Dieter Spaar * Created tools to partially extract and create .diff files * Heavy in pointers/offsets, not robust * Crashes on crafted files * Not cryptographically signed! image:images/delta_header.png[width="80%"] == Firmware upgrade overview image:images/upgrade_process.svg[width="55%",role="gimmick_right"] //[source] ---- $ strings atfwd_daemon | egrep "wget|QCMAP|fota|update.z" ... QCMAP_ConnectionManager /etc/mobileap_cfg.xml n n fotanet /usr/bin/wget -T 20 -t 3 %s -O %s mv %s %s && mkdir -p /cache/fota && echo %s > %s /cache/fota/ipth_config_dfs.txt rm -rf /cache/fota /cache/recovery /cache/update.zip Start download fota for update.zip ---- * atfwd_daemon can be asked to start upgrade * Configure APN, specify URL, store result to update.zip * Add status and reboot to recovery * Apply update.zip and reboot == Recommendation to modem vendors * It is great to have an open and accessible Qualcomm based modem for further research and developing custom applications/extensions * Security issues (particularly unverified FOTA) must be fixed * We need security from attackers _without locking out the user/owner_ ** If vendors introduce verified boot and/or FOTA, allow owner specified keys! * Please keep it open, good for learning and many applications * Allow owners to modify the software of their device * Secure the FOTA upgrading with owner specified keys == Status and Outlook * Status today ** Osmocom wiki with all our findings public now! ** debug tools (`osmo-qcdiag`, LD_PRELOAD wrapper, `qmi_test`, etc.) released ** mpcie-breakout + mv-uart released + available ** `libqmi-glib` integration WIP * Outlook ** we hope to grow documentation in wiki ** please help us out: read code, play with devices + update wiki ** OE/opkg package feed planned ** aim is to have 100% FOSS userland on Cortex-A5 == Unrelated Announcement * Osmocom project has gained support for 3G/3.5G during 2016 * Osmocom suffers from lack of contributions :( * We want to motivate more contributions ** _Accelerate 3.5G_ programme provides 50 free 3.5 femtocells to contributors ** tell us how you would use your free femtocell to improve Osmocom ** Call for Proposals runs until January 31st, 2017. ** see http://sysmocom.de/downloads/accelerate_3g5_cfp.pdf == Questions * Questions? == Links * Our results / hacks ** https://osmocom.org/projects/quectel-modems ** git://git.osmocom.org/quectel-experiments.git ** git://git.osmocom.org/osmo-qcdiag.git ** ftp://ftp.osmocom.org/quectel (mirrored) * Collection of links for further study ** ftp://ftp2.quectel.com/OpenSrc/ ** https://wiki.codeaurora.org/xwiki/bin/QLBEP/ ** https://events.ccc.de/congress/2011/Fahrplan/attachments/2022_11-ccc-qcombbdbg.pdf ** https://media.defcon.org/DEF%20CON%2023/DEF%20CON%2023%20presentations/DEFCON-23-Mickey-Shkatov-Jesse-Michael-Scared-poopless-LTE-and-your-laptop-UPDATED.pdf ** https://github.com/2b-as/xgoldmon ** https://www.blackhat.com/docs/us-14/materials/us-14-Solnik-Cellular-Exploitation-On-A-Global-Scale-The-Rise-And-Fall-Of-The-Control-Protocol.pdf