summaryrefslogtreecommitdiff
path: root/2021/osmodevcall-pysim_shell/pysim_shell.adoc
blob: d6c9286d5fb032f0818d8d80491ac7a0eea981ee (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
pySim-shell - next genreation SIM configuration tool
====================================================
:author:	Harald Welte <laforge@gnumonks.org>
:copyright:	2021 by Harald Welte (License: CC-BY-SA)
:backend:	slidy
:max-width:	45em

== why program your own SIM cards?

* because you want to operate  your own cellular network (2G, 3G, 4G, 5G)
** whether FOSS (Osmocom CNI, srsLTE, nextepc/open5gs, free5gc, OpenBTS, yateBTS, ...)
** but also proprietary (e.g. Amarisoft)
* 2G can use random 3rd party cards, but it's not really convenient in most cases
* Anything >= 3G requires you to know the card-specific secrete keys
** in reality that translates to having your own programmable SIM cards

== some smartcard terminology

* card filesystem
** *MF* (Master File): The root directory
** *DF* (Dedicated File): A subdirectory
** *ADF* (Application Dedicated File): Directory of an application (like USIM, ISIM)
** *EF* (Entry File): A regulare file
*** *Transparent EF*: An unstructured file
*** *Linear Fixed EF*: An file consisting of fixed-length records

== specs vs. proprietary

* SIM cards are fully specified by a combination of ISU, ETSI and 3GPP specs
* this covers only the operation after the card has been issued, e.g.
** reading and writing files accessible without PIN auth, after PIN auth
** performing GSM authentication or UMTS AKA
* it does _not_ cover how the card is issued/provisioned
** secret key material (Ki / K / OP / OPc) is not readable from the card
** it is an implementation detail on how the card manufacturer writes those to the card
* this leads to the need for card-specific support / code in software like pySim

== pySim "classic"

* originally developed in 2009 by Sylvain Munaut
* Linux/FOSS tool for configuration of SuperSIM/MagicSIM
* replicated what proprietary vendor tools did to configure IMSI, Ki, ...
* those ultra-cheap Chinese SIM cards at the time (from the SIM card cloning days of COMP128v1) shipped with binary-only windows programs

== pySim "classic" fundamentals

* library code
** _transport_ - how to exchange APDUs with card
** _commands_ - encoding of ISO7816-4 / TS 51.011 / ETS 102 221 commands
** _cards_ - classes implementing various different programmable SIM cards
* executable programs
** `pySim-prog` for card programming
** `pySim-read` for card reading

== pySim transports

* transport - how to exchange APDUs with card
** _serial_ transport for serial readers (UART Rx+Tx connected to I/O)
** _pcsc_ transport for pcsc-lite supported readers (e.g. USB CCID)
* some more exotic options added later on by Vadim Yanitskiy:
** _calypso_ for using L1CTL to talk to OsmocomBB on Ti Calypso phone
** _modem_atcmd_ for using AT+CSIM commands talking to a cellular modem

== advanced pySim "classic" features

* batch programming of cards, using CSV file input
* even with ability to execute external tool for driving motorized card readers

== Limitations of pySim "classic"

* supports only very few files/parameters that can be programmed
** IMSI, MSISDN, Ki and SMSP was sufficient for making classic GSM SIMs work on your own network
** today, people want to configure VoLTE/IMS parameters, 5G SUCI stuff, etc.
* adding more fields to pySim-classic makes it unusable: Who wants to specify 250 different fields as command line arguments?

== Usability of pySim "classic"

* it was designed to program a batch of cards with defaults and random keys
* there are all kinds of 'automagic' bits happening if not all fields are specified
* pySim at least originally assumed that you program a full card, and not that you perform incremental small changes
* no user manual, very limited RADME and wiki instructions

== the road to pySim-shell

What to do about the user interface?

* GUIs are bad, they are not really scriptable/automatizable
* GUIs are hard to portable, test automatically, etc.
* people setting up FOSS cellular programs most likely use bash/zsh/tcsh on daily basis, so they are familiar with the concept of interactive shells / CLIs
* so let's do a CLI / shell / REPL!

== pySim-shell and `cmd2`

* uses `cmd2` python module for the shell
** extremely powerful framework for building CLI/shell type interfaces
** allows us to focus on implementing logic (like individual commands), not the shell itself
* supports output redirection to local files
* supports piping output through external programs ("| grep foo")
* supports `argparse` to define commands and their options
* allows to dynamically register and unregister commands
* allows full control over tab completion items. 

== pySim-shell UI fundamentals

* some global commands for things like PIN/CHV verification, enabling, disabling, unblcoking
* you `select` your way through the card filesystem, like 'cd' on bash/zsh/...
** tab-completion for [known] file names

.Selecting a file
----
pySIM-shell (MF/ADF.USIM)> select EF.IMSI
{
    "file_descriptor": {
        "shareable": true,
        "file_type": "working_ef",
        "structure": "transparent"
    },
    "file_identifier": "6F07",
    "proprietary_info": {
        "proprietary_D0": "20",
        "proprietary_D2": "0F"
    },
    "life_cycle_status_int": "operational_activated",
    "security_attrib_ref_expanded": "6F0603",
    "file_size": 9,
    "short_file_id": "38"
}
pySIM-shell (MF/ADF.USIM/EF.IMSI)>
----

== pySim-shell reading/writing

* context-sensitive 'action' commands depend on the type of file selected
** `read_binary`, `update_binary`, ... for transparent EF
** `read_record`, `update_record`, ... for linear fixed EF

.Reading and updating a transparent EF
----
pySIM-shell (MF/ADF.USIM/EF.IMSI)> read_binary
089910070000400300
pySIM-shell (MF/ADF.USIM/EF.IMSI)> update_binary 089910070000400301
pySIM-shell (MF/ADF.USIM/EF.IMSI)> read_binary
089910070000400301
----

== pySim-shell decoding/encoding

* working with hex-encoded binary data is not very convenient
* each specified file in the filesystem is represented by a python class
** class can provide encoder + decoder methods to go from binary to JSON and vice-versa
* some shell commands combine reading a file from the card + decoding / encoding
** `read_binary_decoded`, `write_binary_decoded`, ... for transparent EF
** `read_record_decoded`, `write_record_decoded`, ... for linear fixed EF

.Reading and updating a decoded transparent EF
----
pySIM-shell (MF/ADF.USIM/EF.IMSI)> read_binary_decoded
{
    "imsi": "901700000043000"
}
pySIM-shell (MF/ADF.USIM/EF.IMSI)> update_binary_decoded '{"imsi": "901700000043999"}'
pySIM-shell (MF/ADF.USIM/EF.IMSI)> read_binary_decoded
{
    "imsi": "901700000043999"
}
----

== pySim-shell editing of single parameters

* many files contain a lot of parameters
* entering all of them as a single line in `update_binary_decoded` becomes rather cumbersome
* `edit_binary_decoded` and `edit_record_decoded` to the rescue
** reads data from the card
** decodes it to JSON
** spawns whatever is your favorite text editor
** you do your editing, save + exit
** re-encodes the modified JSON
** writes the result to the card

== pySim-shell `export`

* one-shot command to iterate over all [known] files below the current DF and dump them
* output format is structured in a way that you can use it as a pySim-shell script
* results in easy method to backup and restore a card

NOTE:: only the ETSI/3GPP standard files that are covered by pySim classes are exported.  There may be any
number of proprietary additional files on the card.  Particularly K/OP/OPc or OTA keys are not exported this
way

== pySim-shell advanced cmd2 usage

* `run_script`: execute any text file as script inside the pySim-shell
* `run_pyscript`: execute a python script with access to all of the state / classes / methods
* `ipy`: run an interactive IPython session
* `py`: run an interactive python interpreter
** `card` represents the `pySim.cards.Card` instance
** `rs` represents the `pySim.filesystem.RuntimeState`

.Using interactive python shell with access to pySim-shell state
----
pySIM-shell (MF)> py
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.

End with `Ctrl-D` (Unix) / `Ctrl-Z` (Windows), `quit()`, `exit()`.
Non-Python commands can be issued with: app("your command")

>>> rs.selected_file
<pySim.filesystem.CardMF object at 0x7f97952f3070>
>>> rs.selected_file.desc
'Master File (directory root)'
----


== Other recent pySim developments

* introduction of `sphinx` for generating documentation
* adding API documentation at least for all of the *new* classes/methods that are used to build pySim-shell,
  and which you need to extend to cover more files
* automatic generation of pySim-shell command reference from `argparse` instances

== Call for Contributions

* pySim-shell is already very usable, but needs your help!
** most files still need encoder + decoder methods to be implemented
** we need automatic testing of encoders + decoders
** we want JSON schema for the JSON representation of each file
** we want to have batch programming in pySim-shell

So if you know a bit of python and care about SIM cards, please send your patches!

== Further Reading

* https://osmocom.org/projects/pysim/wiki[pySim project page on osmocom.org]
* https://git.osmocom.org/pysim/about/[pySim source code / git repository]
* https://media.ccc.de/v/36c3-10737-sim_card_technology_from_a-z[Video of talk "SIM card technology from A-Z"]


== EOF

End of File
personal git repositories of Harald Welte. Your mileage may vary