summaryrefslogtreecommitdiff
path: root/2016/33c3/33c3-modems.adoc
blob: 29a54671555708d1ca08c89136e8454ecc61f488 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528

Dissecting modern (3G/4G) cellular modems
=========================================
:author:	Harald Welte, Holger Hans Peter Freyther
:copyright:	Harald Welte, Holger Hans Peter Freyther (License: CC-BY-SA)
:backend:	slidy
:max-width:	45em

//include::33c3-modems.css[]

== This talk

* Our motivation
* A bit of History
* Selecting a device
* An unexpected surprise
* Firmware upgrade
* Recommendations/Wishes

== Motivation

// 9 years of Osmocom?
// 3G and 4G development
// Hardware for decoding
* Implementing GSM specifications for the last decade (OpenMoko, Osmocom)
* 8 years since _Anatomy of Smartphone Hardware_ at 25C3
* 7 years since OsmocomBB for GSM
* Used and built M2M devices using 2G modems at work
* so we're looking for a modem that can be used for
** our next-generation M2M/embedded devices
** testing/logging/tracing Osmocom 3G/4G network-side software
** building more tools to help understanding cellular technology

== Cellular Modems in M2M

image:images/sl6087_hw.png[height=300,role="gimmick_right"]

* Assume you want to build a M2M device
* Classic approach to M2M/Embedded cellular:
** Cellular modem with AT commands over Serial/USB
** Main Processor runs M2M application
* if you run Application in Modem, you can save PCB space, power and BOM cost
** OpenAT by Sierra Wireless
*** Write C code using OpenAT APIs
*** Dynamically loaded into the RTOS
*** Runs without privilege separation, MMU
*** Protocol to multiplex AT, log, debug
*** Discontinued HW platform => Locked in
*** Various other limitations

== Device requirements

Our requirements for a good modem

** Ability to run application code inside modem
** Avoid modem supplier vendor lock-in (EOL, ...)
** Get textual logging when handling messages
** Get a copy of the radio network messages and export to GSMTAP
*** Like Tobias Engels https://github.com/2b-as/xgoldmon[x-goldmon]
*** But for all GPRS, EGPRS, UMTS and LTE messages

== Qualcomm DIAG protocol

* Qualcomm DIAG in many products (DVB-H, GSM, ...)
* https://events.ccc.de/congress/2011/Fahrplan/attachments/2022_11-ccc-qcombbdbg.pdf[Presented] by Guillaume Delugre at 28C3
* Simple HDLC frame (0x7e), cmd, data, CRC16

* Events, Logging, Command/Response
* Thousands of different message structures
* ModemManager, gsm-parser consume only a small fraction

image:images/diag_frame.svg[width="90%"]

== Selecting a device

image:images/28c3_option_stick.png[width="30%",role="gimmick_right"]

* Old Option Icon 225 stick exposes DIAG out of the box
* Quectel UC20 (2G+3G) expose DIAG by default
** but no LTE support
* Quectel EC20 (2G+3G+4G) expose DIAG by default
** 2G, 3G and 4G sounds quite nice
** EC20 not only a LGA solder module but also as mini-PCIe
*** convenient for early testing / prototyping without custom board

image:images/ec20.png[height=300,role="gimmick_right"]

* EC20 using a Qualcomm MDM9615 chipset
** Also used in the iPhone5
** Almost no documentation on MDM9615 available
** Still, a good candidate for starting our research...

// Erst ein mal EC20 und sagen wieso es interessant ist
// und dann, dass es Linux hat.. um dann ein Block diagram
// zu haben?

[role="change_topic"]
== An unexpected surprise

== Firmware update, hints of Linux

* Got a firmware upgrade to fix stability / bugs
* Looks like it contains traces of Linux?
* Looks like it uses fastboot for the update
* Other people have already found Linux in MDM9615 based products (e.g https://media.defcon.org/DEF%20CON%2023/DEF%20CON%2023%20presentations/DEFCON-23-Mickey-Shkatov-Jesse-Michael-Scared-poopless-LTE-and-your-laptop-UPDATED.pdf[Mickey Shkatov] at DEFCON 23)
* But why would there be Linux inside a Modem?
** Qualcomm is known for their REX/AMSS on Hexagon baseband ?!?
* And if it contains Linux, GPL requires them to mention that, include
  License text and provide source code ?!?

== GPL compliance

* No written offer, let's see if it runs Linux
* Armijn Hemels `gpltool.git` has `unyaffs` to unpack yaffs
* `strings`, etc. clearly reveal Linux, glibc, busybox
** other interesting strings like `AT+QLINUXCMD=?` show up
* The fun and exploration begins...
** technical analysis (serial console, firmware reversing, ...)
** legal enforcement to get source code of GPL/LGPL components (Harald is founder of http://gpl-violations.org[gpl-violations.org])

== Hardware based analysis

* mPCIe modules often expose additional signals like PCM audio on
  non-standard pins
* existing PC/embedded mainboards don't use those signals
* create Osmocom mPCIe-breakout board to access those signals
* https://osmocom.org/projects/mpcie-breakout/wiki

image:images/mpcie_breakout.jpg[width="70%"]

== Serial Console

* EC20 solder module documents DBG_UART pinout, but not all modules
  have it enabled?
* serial console is at 1.8V, but the 1.8V supply is not accessible (so
  not easy to add external level shifter / Vref)
* create Osmocom multi-voltage USB-UART with selectable 1.8,
  2.3, 2.5, 2.8, 3.0 and 3.3V logic level

image:images/mv_uart.jpg[width="40%",role="gimmick_right"]

* https://osmocom.org/projects/mv-uart/wiki
* root password (DES hash): `oelinux123`

== Retro-fitting Serial Console to mPCIe module

* unfortunately the DBG_UART on the LGA module solder pads is not
  exposed to mPCIE
* some soldering required to retro-fit a 2.54mm header:

image:images/ec20_uart.jpg[width="70%"]

== GPL compliance

* Linux basis created by Qualcomm and used by Quectel
** https://wiki.codeaurora.org/xwiki/bin/QLBEP/
** Many branches, releases, which to use?

[quote, Tonino Perazzi]
I tried instruction above to build yaffs2 for MDM9615, so I downloaded source `M9615AAAARNLZA1611161.xml` but during compilation I faced some libs that are missing such as libQMI and acdb-loader..

image:images/qualcom_many_releases.png[width="80%"]

== GPL compliance

[qanda]
Asking for the complete and corresponding source::
[quote,Quectel]
** The source code of Qflash tool in Linux is attached, [...]
[qanda]
Asking again for the complete and corresponding source::
[quote,Quectel]
We never been in legal dispute and we always make sure to understand IPR ahead of using technology belonging to third party.

image:images/quectel_ipr.jpg[width="100%"]

== GPL compliance

[qanda]
Asking for the complete and corresponding source::
[quote,Quectel]
	We appreciate the efforts that your client had put into the open source
project netfilter/_iptable_. However, [...] *your client does not have the right to
empower the copyright*. We think software netfilter/iptable is built on
the code operating system _GUN_/Linux, thus subject to GPL terms, where FSF
requires that each author of code incorporated in FSF projects either
provide copyright assignment to FSF or disclaim copyright. Therefore,
It seems that *your client does not have the copyright on netfilter/iptable.* +
 +
As one of the leading providers of wireless solution, *Quectel is always
respectful IPR*. We would like to compliant with GPL and do some necessary
statements´╝îincluding a disclaimer or appropriate notices. Under the terms
of GPL, we would like to dedicate Kernel code of EC25x to free software
community.

== GPL compliance

[qanda]
Asking for the complete and corresponding source::
[quote,Quectel]
	Many thanks for your detailed explanations GPL/LGPL license terms and the practical methods. I will carefully study your suggestions again and find a proper way to open GLP/LGPL licensed software. Basically, we will simply provide a tarball of open source for download at this time. And release the git repositories in next step.

[qanda]
Asking for the complete and corresponding source::
[quote,Quectel]
	We are always willing to achieve GPL compliance.

[qanda]
Asking for the complete and corresponding source::
[quote,Quectel]
	So we need some time to know of all things and construct the Open Source projects. Within a short time, we cannot construct a perfect web site to present Open Source things now. However, we will continue to do like that.

== GPL compliance

[qanda]
Your tarball is missing some files::
[quote,Quectel]
We have issued all GPL licensed source code.
*We have no the xt_dscp file in the project, and nor Qulacomm*. It must be
caused by your compilation environment.
If you have more question or problem during the development with Quectel
module, please add my Skype ID (XXXXX), I will continue to support you
on Skype. +
*The email will not discuss the compiling issue any more.*



== GPL compliance

* ... many months later
** we have received various source tarballs
** they contain not only GPL/LGPL code but other FOSS code (thanks!)
** full license compliance still not achieved, but improving...
* Sierra Wireless Legato is a positive example of a competitor
** they not only provide the OE/Linux source but extensive
documentation!
** but they try to lure customers into a proprietary Legato framework,
and thus again vendor-lock-in :(

image:images/legato_flash.png[width="80%"]

[role="change_topic"]
== MDM 9615 HW and SW


== Qualcomm Hardware

* Qualcomm MDM9615 chipset
* Used in the iPhone 5 and automotive
* Modems like Quectel EC20, Sierra Wireless MC7355
* No public HW documentation?!
* Either not many people study it or are not allowed to share?

== MDM 9615 HW Overview

* ????
// Block diagram?
// Listing of interfaces.
// Show it is a highly complex SoC... with even more things
// that are unknown.. device tree file, peripheral, etc


== How to access the system?

* serial console requires soldering re-work and is slow
* easy mechanism to get shell and transfer files from/to target
* Android `adbd` present on the modem but not exposed via USB
* it's possible to re-configure the Linux kernel Android USB Gadget:
** `AT+QLINUXCMD+"/usr/bin/usb_uartdiag"`
** device re-enumerates with different composite USB interfaces
* Linux kernel driver on host needs patching (static interface
  mapping assumption)
** patches available in `quectel-experiments.git`, documented in wiki


== MDM 9615 AP SW Overview

image:images/gandroid_logo.png[height=200,role="gimmick_right"]

The software stack seems to be called *Qualcomm LE*

* Android Bootloader
* Android Linux kernel
* Android Debug Bridge (adb)
* but: GNU libc, busybox userland
* Using OpenEmbedded to build images
* Developed and maintained by Qualcomm


== Qualcomm Linux kernel overview

* Qualcomm Android Linux kernel
* Huge changes compared to mainline `git diff -w | wc -l`
** `v3.0.21` in EC20: 1.5 million lines
** `v3.18.20` in EC25: 1.9 million lines
* Expected: CPU + peripheral drivers
* Less expected:
** smem_log (shared memory logging)
** ipc_log (inter-processOR communication)
** remote spinlocks

== Qualcomm Linux kernel subsystems

Some of the Qualcomm-specific kernel sub-systems 

[cols="20%,80%"]
|===
|SMD|Shared Memory Device
|IPC|Inter Processor Communications
|RMNET|Remote Network
|BAM|Bus Access Manager
|IPA|Internet Packet Accelerator
|DIAGFWD|DIAG Forwarding
|AF_MSM_IPC|Socket family for Qualcomm IPC
|===

== Qualcomm LE System Architecture

image:images/qualcomm_le.svg[width="50%",role="gimmick_right"]

* simplified block diagram
* USB interface fully controlled by Linux AP
** very complex Qualcomm Android USB Gadget
** some endpoints mapped to SMD queues
** other endpoints handled by _regular_ Linux 
** GPS NMEA takes completely different path than AT commands, despite
both being serial ports?
** DIAG and QMI handled in more complex ways

== DIAG in Qualcomm LE

* DIAG interface of Modem exposed on SMD
* diagfwd distributes messages between USB, SMD and `/dev/diagchar`
* Linux userspace processes don't use syslog, but diag msg for logging via `libdiag.so`

image:images/diag.svg[width="100%"]

== QMI in Qualcomm LE

every `rmnet` data device has associated QMI control

* on your Linux PC: `qmi_wwan` and `/dev/cdc-wdm`
* on Qualcomm LE modem: `/dev/smdcntlN`, multiplexed by `qmuxd`

image:images/qmi_smd_qmuxd.svg[width="100%"]

== Tools for analysis

We created some tools to help our analysis

* used OE to build matching `opkg` and OE packages for `socat`, `lsof`, `strace`
* FOSS programs for the Linux AP linked against proprietary `libqmi-framework.so`
** `qmi_test`: Simple program to read IMEI via QMI
** `atcop_test`: Test program to implement AT commands in Linux userspace
* 100% FOSS programs
** `qmuxd_wrapper`: LD_PRELOAD wrapper for tracing between `qmuxd` and QMI clients
** `libqmi-glib` transport support for `qmuxd` (work in progress)
** `osmo-qcdiag`: Host tool for obtaining DIAG based logs from Linux programs + QMI traces, decoded via `libmi-glib`

== Userspace programs

We found a bunch of proprietary Linux userspace programs

[cols="20%,80%"]
|===
|`adbd`|Implements Android Debug Bridge
|`atfwd_daemon`|Implement Quectel-Specific AT Commands
|`quectel_daemon`|?; various ASoC related bits
|`qti`|?
|`mbim`|Mobile Broadband IF Model (translates MBIM to QMI)
|`QCMAP_ConnectionManager`|runs linux-base WiFi AP/router with LTE backhaul
|`quec_bridge`|reads GPS NMEA from `/dev/nmea` and writes it to `/dev/ttyGS0`
|===

[role="change_topic"]
== Funny bits + pieces

== Funny AT commands

* `AT+QLINUXCMD`, e.g. switch usb config to get adb
** arbitrary shell commands executed as root on r/w rootfs!
* `AT+QFASTBOOT`, switch to the bootloader
* `AT+QPRINT`, print dmesg
* AT for `system("echo mem > /sys/power/state")`

== How many processes does it take to reboot a system?

* `rebootdiagapp` registers DIAG command (cmd code 0x29)
** spawns thread that runs `system("qmi_simple_ril_test input=/tmp/reset")`
** `system("echo 'modem reset' > /tmp/reset")`
*** makes `qmi_simple_ril_test` send a QMI message to modem
** `system("rm /tmp/reset")`
** writes "REBOOT" to `/dev/rebooterdev` this time using `fwrite()`!
* `reboot_daemon` reads `/dev/rebooterdev`

----
read_count = read(pipe_fd,buf,MAX_BUF-1);
/* if read REBOOT_STR, then call reboot */
if(strncmp(buf,REBOOT_STR,strlen(REBOOT_STR)) == 0) {
    debug_printf("going for reboot\n");
    printf("reboot-daemon: initiating reboot\n");
    system("reboot");
}
----

== C programs that look like shell scripts

* strings /usr/bin/quectel_daemon

----
echo "nau8814-aif1" > /sys/devices/platform/soc-audio.0/tx_dai_name
cp -f /cache/usb/qcfg_usbcfg /etc/; cp -f /cache/usb/usb /etc/init.d/
echo 90 >/sys/kernel/debug/pm8xxx-pwm-dbg/0/duty-cycle
pkill -f "/bin/sh /usr/bin/nmea_demon.sh"
ps ef | grep "quec_bridge /dev/nmea /dev/ttyGS0" | grep -v grep
cd /cache/ufs;ls
----

[role="change_topic"]
== Firmware upgrade

== recovery and applypatch

* Qualcomm uses https://android.googlesource.com/platform/bootable/recovery.git/+/android-4.0.4_r2.1[recovery.git] from Android ~4.0
* Updates are zip files with deltas, SHA1+RSA
* recovery started on boot, drives applypatch
----
// Look for an RSA signature embedded in the .ZIP file comment given
// the path to the zip.  Verify it matches one of the given public
// keys.
----

== Qualcomm EC20 firmware upgrade

image:images/redbend.png[width="30%",role="gimmick_right"]

* Based on the recovery.git code
* But for some reason using RedBend for the update (legacy?)
* RSA still linked into the binary but not used
* RedBend used by many more companies and systems (e.g. Quectel UC20, automotive)


== RedBend (delta update) software

* Used in OMA DeviceManagement as well? (e.g. https://www.blackhat.com/docs/us-14/materials/us-14-Solnik-Cellular-Exploitation-On-A-Global-Scale-The-Rise-And-Fall-Of-The-Control-Protocol.pdf[Mathew Solnik])
* Lots of starring at hexdumps, lots of help from Dieter Spaar
* Created tools to partially extract and create .diff files
* Heavy in pointers/offsets, not robust
* Crashes on crafted files
* Not cryptographically signed!

image:images/delta_header.png[width="80%"]


== Firmware upgrade overview

image:images/upgrade_process.svg[width="55%",role="gimmick_right"]
//[source]
----
$ strings atfwd_daemon | egrep  "wget|QCMAP|fota|update.z"

... QCMAP_ConnectionManager /etc/mobileap_cfg.xml n n fotanet
/usr/bin/wget -T 20 -t 3 %s -O %s
mv %s %s && mkdir -p /cache/fota && echo %s > %s
/cache/fota/ipth_config_dfs.txt
rm -rf /cache/fota /cache/recovery /cache/update.zip
Start download fota for update.zip
----

* atfwd_daemon can be asked to start upgrade
* Configure APN, specify URL, store result to update.zip
* Add status and reboot to recovery
* Apply update.zip and reboot

== Recommendation to modem vendors

* It is great to have an open and accessible Qualcomm based modem for
  further research and developing custom applications/extensions
* Security issues (particularly unverified FOTA) must be fixed
* We need security from attackers _without locking out the user/owner_
** If vendors introduce verified boot and/or FOTA, allow owner specified keys!
* Please keep it open, good for learning and many applications
* Allow owners to modify the software of their device
* Secure the FOTA upgrading with owner specified keys

== Status and Outlook

* Status today
** Osmocom wiki with all our findings public now!
** debug tools (`osmo-qcdiag`, LD_PRELOAD wrapper, `qmi_test`, etc.) released
** mpcie-breakout + mv-uart released + available
** `libqmi-glib` integration WIP
* Outlook
** we hope to grow documentation in wiki
** please help us out: read code, play with devices + update wiki
** OE/opkg package feed planned
** aim is to have 100% FOSS userland on Cortex-A5

== Unrelated Announcement

* Osmocom project has gained support for 3G/3.5G during 2016
* Osmocom suffers from lack of contributions :(
* We want to motivate more contributions
** _Accelerate 3.5G_ programme provides 50 free 3.5 femtocells to contributors
** tell us how you would use your free femtocell to improve Osmocom
** Call for Proposals runs until January 31st, 2017.
** see http://sysmocom.de/downloads/accelerate_3g5_cfp.pdf

== Questions

* Questions?



== Links

* Our results / hacks
** https://osmocom.org/projects/quectel-modems
** git://git.osmocom.org/quectel-experiments.git
** git://git.osmocom.org/osmo-qcdiag.git
** ftp://ftp.osmocom.org/quectel (mirrored)
* Collection of links for further study
** ftp://ftp2.quectel.com/OpenSrc/
** https://wiki.codeaurora.org/xwiki/bin/QLBEP/
** https://events.ccc.de/congress/2011/Fahrplan/attachments/2022_11-ccc-qcombbdbg.pdf
** https://media.defcon.org/DEF%20CON%2023/DEF%20CON%2023%20presentations/DEFCON-23-Mickey-Shkatov-Jesse-Michael-Scared-poopless-LTE-and-your-laptop-UPDATED.pdf
** https://github.com/2b-as/xgoldmon
** https://www.blackhat.com/docs/us-14/materials/us-14-Solnik-Cellular-Exploitation-On-A-Global-Scale-The-Rise-And-Fall-Of-The-Control-Protocol.pdf
personal git repositories of Harald Welte. Your mileage may vary