Dissecting modern (3G/4G) cellular modems ========================================= :author: Harald Welte, Holger Hans Peter Freyther :copyright: Harald Welte, Holger Hans Peter Freyther (License: CC-BY-SA) :backend: slidy :max-width: 45em //include::33c3-modems.css[] == This talk * Our motivation and approach * A bit of History * Selecting a device * An unexpected surprise * Firmware upgrade * Recommendations/Wishes == Motivation // 9 years of Osmocom? // 3G and 4G development // Hardware for decoding * Implementing GSM specifications for the last decade (OpenMoko, Osmocom) * 8 years since _Anatomy of Smartphone Hardware_ at 25C3 * 7 years since OsmocomBB for GSM * Used and built M2M devices using 2G modems at work * Started to build Osmocom 3G/4G software, logs/traces help * Build tools to help understanding cellular technology == History image:images/sl6087_hw.png[height=280,role="gimmick_right"] * OpenAT by Sierra Wireless * Write C code using OpenAT APIs * Dynamically loaded into the RTOS * Runs without privilege separation, MMU * Eclipse based IDE and plugins (in clojure) * Protocol to multiplex AT, log, debug * 2G and 3G modems were available * Discontinued HW platform => Locked in * Various other limitations == Device requirements * Get textual logging when handling messages * Get a copy of the radio network message and export to GSMTAP * Like Tobias Engels https://github.com/2b-as/xgoldmon[x-goldmon] * But for GPRS, 3G and 4G * Enabled by default and not locked down in the future == Qualcomm DIAG protocol * Qualcomm DIAG in many products (DVB-H, GSM, ...) * https://events.ccc.de/congress/2011/Fahrplan/attachments/2022_11-ccc-qcombbdbg.pdf[Presented] by Guillaume Delugre at 28C3 * Simple HDLC frame (0x7e), cmd, data, CRC16 * Thousands of different message structures * Events, Logging, Command/Response * ModemManager, gsm-parser consume only a small fraction image:images/diag_frame.png[width="90%"] == Selecting a device image:images/28c3_option_stick.png[width="30%",role="gimmick_right"] * 3G Options Icon stick exposes DIAG out of the box * Quectel UC20 (2G+3G) enable it by default * Quectel EC20 (2G+3G+4G) enable it by default * 2G, 3G and 4G sounds quite nice * EC20 comes as mini-PCIe module as well == Quectel EC20 image:images/ec20.png[height=200,role="gimmick_right"] * Using a Qualcomm MDM9615 chipset ** Also used in the iPhone5 * Surprisingly runs Linux * Not surprising to people familiar with MDM9615 (e.g https://media.defcon.org/DEF%20CON%2023/DEF%20CON%2023%20presentations/DEFCON-23-Mickey-Shkatov-Jesse-Michael-Scared-poopless-LTE-and-your-laptop-UPDATED.pdf[Mickey Shkatov]) * Almost no documentation available // Erst ein mal EC20 und sagen wieso es interessant ist // und dann, dass es Linux hat.. um dann ein Block diagram // zu haben? [role="change_topic"] == An unexpected surprise == GPL compliance * Got a firmware upgrade to fix stability * Looks like it contains traces of Linux? * No written offer, let's see if it runs Linux * Armijn Hemels `gpltool.git` has `unyaffs` to unpack yaffs * strings, etc., `AT+QLINUXCMD=?` * The fun and exploration begins == GPL compliance * Linux basis created by Qualcomm and used by Quectel * https://wiki.codeaurora.org/xwiki/bin/QLBEP/ * Many branches, releases, which to use? [quote, Tonino Perazzi] I tried instruction above to build yaffs2 for MDM9615, so I downloaded source `M9615AAAARNLZA1611161.xml` but during compilation I faced some libs that are missing such as libQMI and acdb-loader.. image:images/qualcom_many_releases.png[width="80%"] == GPL compliance [qanda] Asking for the complete and corresponding source:: Receiving source for the flash tool == GPL compliance [qanda] Asking for the complete and corresponding source:: We never been in legal dispute and we always make sure to understand IPR ahead of using technology belonging to third party. == GPL compliance [qanda] Asking for the complete and corresponding source:: We appreciate the efforts that your client had put into the open source project netfilter/iptable. However, We have some doubts about the alleged copyright. From our perspective, your client does not have the right to empower the copyright. We think software netfilter/iptable is built on the code operating system GUN/Linux, thus subject to GPL terms, where FSF requires that each author of code incorporated in FSF projects either provide copyright assignment to FSF or disclaim copyright (“we should keep the copyright status of the program as simple as possible. We do this by asking each contributor to either assign the copyright on his contribution to the FSF, or disclaim copyright on it and thus put it in the public domain”). Therefore, It seems that your client does not have the copyright on netfilter/iptable. As one of the leading providers of wireless solution, Quectel is always respectful IPR. We would like to compliant with GPL and do some necessary statements，including a disclaimer or appropriate notices. Under the terms of GPL, we would like to dedicate Kernel code of EC25x to free software community. == GPL compliance [qanda] Asking for the complete and corresponding source:: Many thanks for your detailed explanations GPL/LGPL license terms and the practical methods. I will carefully study your suggestions again and find a proper way to open GLP/LGPL licensed software. Basically, we will simply provide a tarball of open source for download at this time. And release the git repositories in next step. == GPL compliance [qanda] Asking for the complete and corresponding source:: We are always willing to achieve GPL compliance. == GPL compliance [qanda] Asking for the complete and corresponding source:: To be frank, we have no experience over Open Source things before. So we need some time to know of all things and construct the Open Source projects. Within a short time, we cannot construct a perfect web site to present Open Source things now. However, we will continue to do like that. == GPL compliance [qanda] Your tarball is missing some files:: We have issued all GPL licensed source code. We have no the xt_dscp file in the project, and nor Qulacomm. It must be caused by your compilation environment. If you have more question or problem during the development with Quectel module, please add my Skype ID (XXXXX), I will continue to support you on Skype. The email will not discuss the compiling issue any more.'' == GPL compliance * ... many months later * License compliance still not achieved * Sierra Wireless Legato is a positive example of a competitor image:images/legato_flash.png[width="80%"] [role="change_topic"] == MDM 9615 HW and SW == Qualcomm Hardware * Qualcomm MDM9615 chipset * Used in the iPhone 5 and automotive * Modems like Quectel EC20, Sierra Wireless MC7355 * No public HW documentation?! * Either not many people study it or are not allowed to share? == MDM 9615 HW Overview * ???? // Block diagram? // Listing of interfaces. // Show it is a highly complex SoC... with even more things // that are unknown.. device tree file, periperhal, etc == MDM 9615 AP SW Overview image:images/gandroid_logo.png[height=200,role="gimmick_right"] The software stack seems to be called *Qualcomm LE* * Android Bootloader * Android Linux kernel * Android Debug Bridge (adb) * but: GNU libc, busybox userland * Using OpenEmbedded to build images * Developed and maintained by Qualcomm == Qualcomm Linux kernel overview * Qualcomm Android Linux kernel * Huge changes compared to mainline `git diff -w | wc -l` ** `v3.0.21` in EC20: 1.5 million lines ** `v3.18.20` in EC25: 1.9 million lines * Expected: CPU + peripheral drivers * Less expected: ** smem_log, ipc_log, remote spinlocks, etc. == Qualcomm Linux kernel subsystems [cols="20%,80%"] |=== |SMD|Shared Memory Device |IPC|Inter Processor Communications |RMNET|Remote Network |BAM|Bus Access Manager |IPA|Internet Packet Accelerator |DIAGFWD|DIAG Forwarding |=== == Qualcomm LE System Architecture image:images/qualcomm_le.svg[width="50%",role="gimmick_right"] * simplified block diagram * USB interface fully controlled by Linux AP ** very complex Qualcomm Android USB Gadget ** some endpoints mapped to SMD queues ** other endpoints handled by _regular_ Linux ** GPS NMEA takes completely different path than AT commands, despite both being serial ports? ** DIAG and QMI handled in more complex ways == DIAG in Qualcomm LE * DIAG interface of Modem exposed on SMD * diagfwd distributes messages between USB, SMD and `/dev/diagchar` * Linux userspace processes don't use syslog, but diag msg for logging via `libdiag.so` image:images/diag.svg[width="100%"] == QMI in Qualcomm LE every `rmnet` data device has associated QMI control * on your Linux PC: `qmi_wwan` and `/dev/cdc-wdm` * on Qualcomm LE modem: `/dev/smdcntlN`, multiplexed by `qmuxd` image:images/qmi_smd_qmuxd.svg[width="100%"] == Funny commands * `AT+QLINUXCMD`, e.g. switch usb config to get adb ** arbitrary shell commands executed as root on r/w rootfs! * `AT+QFASTBOOT`, switch to the bootloader * `AT+QPRINT`, print dmesg * AT for `system("echo mem > /sys/power/state")` [role="change_topic"] == Firmware upgrade == recovery and applypatch * Android ~4.0 based https://android.googlesource.com/platform/bootable/recovery.git/+/android-4.0.4_r2.1[recovery.git] * Updates are zip files with deltas, SHA1+RSA * recovery started on boot, drives applypatch ---- // Look for an RSA signature embedded in the .ZIP file comment given // the path to the zip. Verify it matches one of the given public // keys. ---- == Qualcomm EC20 firmware upgrade image:images/redbend.png[height=76,role="gimmick_right"] * Based on the recovery.git code * But for some reason using RedBend for the update (legacy?) * RSA still linked into the binary but not used * RedBend used by many more companies and systems (e.g. Quectel UC20, automotive) == RedBend (delta update) software * Used in OMA DeviceManagement? (e.g. https://www.blackhat.com/docs/us-14/materials/us-14-Solnik-Cellular-Exploitation-On-A-Global-Scale-The-Rise-And-Fall-Of-The-Control-Protocol.pdf[Mathew Solnik]) * Lots of starring at hexdumps, lots of help from Dieter Spaar * Created tools to partially extract and create .diff files * Heavy in pointers/offsets, not robust, crashes * Not cryptographically signed! image:images/delta_header.png[width="80%"] == Firmware upgrade overview image:images/upgrade_process.png[width="55%",role="gimmick_right"] //[source] ---- $ strings atfwd_daemon | egrep "wget|QCMAP|fota|update.z" ... QCMAP_ConnectionManager /etc/mobileap_cfg.xml n n fotanet /usr/bin/wget -T 20 -t 3 %s -O %s mv %s %s && mkdir -p /cache/fota && echo %s > %s /cache/fota/ipth_config_dfs.txt rm -rf /cache/fota /cache/recovery /cache/update.zip Start download fota for update.zip ---- * atfwd_daemon can be asked to start upgrade * Configure APN, specify URL, store result to update.zip * Add status and reboot to recovery * Apply update.zip and reboot == Recommedation * Please keep it open, good for learning * Allow owners to modify the software of their device * Secure the FOTA upgrading with owner specified keys == Questions * Questions? == Announcement * 3G femtocells for Osmocom/OpenBSC development == Links * Our results / hacks ** https://osmocom.org/projects/quectel-modems ** FIXME: quectel-experiments.git ** FIXME: quectel source tarball mirror * Collection of links for further study ** ftp://ftp2.quectel.com/OpenSrc/ ** https://wiki.codeaurora.org/xwiki/bin/QLBEP/ ** https://events.ccc.de/congress/2011/Fahrplan/attachments/2022_11-ccc-qcombbdbg.pdf ** https://media.defcon.org/DEF%20CON%2023/DEF%20CON%2023%20presentations/DEFCON-23-Mickey-Shkatov-Jesse-Michael-Scared-poopless-LTE-and-your-laptop-UPDATED.pdf ** https://github.com/2b-as/xgoldmon ** https://www.blackhat.com/docs/us-14/materials/us-14-Solnik-Cellular-Exploitation-On-A-Global-Scale-The-Rise-And-Fall-Of-The-Control-Protocol.pdf