diff options
authorHarald Welte <>2021-04-09 19:56:35 +0200
committerHarald Welte <>2021-04-09 19:56:35 +0200
commitb0e4869b542657d48d6b17841e1ee39ff1a76006 (patch)
parent6ae6069a16141206e0067a1c8c320154cd9baa21 (diff)
pySim-shell talk for OsmoDevCall
1 files changed, 242 insertions, 0 deletions
diff --git a/2021/osmodevcall-pysim_shell/pysim_shell.adoc b/2021/osmodevcall-pysim_shell/pysim_shell.adoc
new file mode 100644
index 0000000..d6c9286
--- /dev/null
+++ b/2021/osmodevcall-pysim_shell/pysim_shell.adoc
@@ -0,0 +1,242 @@
+pySim-shell - next genreation SIM configuration tool
+:author: Harald Welte <>
+:copyright: 2021 by Harald Welte (License: CC-BY-SA)
+:backend: slidy
+:max-width: 45em
+== why program your own SIM cards?
+* because you want to operate your own cellular network (2G, 3G, 4G, 5G)
+** whether FOSS (Osmocom CNI, srsLTE, nextepc/open5gs, free5gc, OpenBTS, yateBTS, ...)
+** but also proprietary (e.g. Amarisoft)
+* 2G can use random 3rd party cards, but it's not really convenient in most cases
+* Anything >= 3G requires you to know the card-specific secrete keys
+** in reality that translates to having your own programmable SIM cards
+== some smartcard terminology
+* card filesystem
+** *MF* (Master File): The root directory
+** *DF* (Dedicated File): A subdirectory
+** *ADF* (Application Dedicated File): Directory of an application (like USIM, ISIM)
+** *EF* (Entry File): A regulare file
+*** *Transparent EF*: An unstructured file
+*** *Linear Fixed EF*: An file consisting of fixed-length records
+== specs vs. proprietary
+* SIM cards are fully specified by a combination of ISU, ETSI and 3GPP specs
+* this covers only the operation after the card has been issued, e.g.
+** reading and writing files accessible without PIN auth, after PIN auth
+** performing GSM authentication or UMTS AKA
+* it does _not_ cover how the card is issued/provisioned
+** secret key material (Ki / K / OP / OPc) is not readable from the card
+** it is an implementation detail on how the card manufacturer writes those to the card
+* this leads to the need for card-specific support / code in software like pySim
+== pySim "classic"
+* originally developed in 2009 by Sylvain Munaut
+* Linux/FOSS tool for configuration of SuperSIM/MagicSIM
+* replicated what proprietary vendor tools did to configure IMSI, Ki, ...
+* those ultra-cheap Chinese SIM cards at the time (from the SIM card cloning days of COMP128v1) shipped with binary-only windows programs
+== pySim "classic" fundamentals
+* library code
+** _transport_ - how to exchange APDUs with card
+** _commands_ - encoding of ISO7816-4 / TS 51.011 / ETS 102 221 commands
+** _cards_ - classes implementing various different programmable SIM cards
+* executable programs
+** `pySim-prog` for card programming
+** `pySim-read` for card reading
+== pySim transports
+* transport - how to exchange APDUs with card
+** _serial_ transport for serial readers (UART Rx+Tx connected to I/O)
+** _pcsc_ transport for pcsc-lite supported readers (e.g. USB CCID)
+* some more exotic options added later on by Vadim Yanitskiy:
+** _calypso_ for using L1CTL to talk to OsmocomBB on Ti Calypso phone
+** _modem_atcmd_ for using AT+CSIM commands talking to a cellular modem
+== advanced pySim "classic" features
+* batch programming of cards, using CSV file input
+* even with ability to execute external tool for driving motorized card readers
+== Limitations of pySim "classic"
+* supports only very few files/parameters that can be programmed
+** IMSI, MSISDN, Ki and SMSP was sufficient for making classic GSM SIMs work on your own network
+** today, people want to configure VoLTE/IMS parameters, 5G SUCI stuff, etc.
+* adding more fields to pySim-classic makes it unusable: Who wants to specify 250 different fields as command line arguments?
+== Usability of pySim "classic"
+* it was designed to program a batch of cards with defaults and random keys
+* there are all kinds of 'automagic' bits happening if not all fields are specified
+* pySim at least originally assumed that you program a full card, and not that you perform incremental small changes
+* no user manual, very limited RADME and wiki instructions
+== the road to pySim-shell
+What to do about the user interface?
+* GUIs are bad, they are not really scriptable/automatizable
+* GUIs are hard to portable, test automatically, etc.
+* people setting up FOSS cellular programs most likely use bash/zsh/tcsh on daily basis, so they are familiar with the concept of interactive shells / CLIs
+* so let's do a CLI / shell / REPL!
+== pySim-shell and `cmd2`
+* uses `cmd2` python module for the shell
+** extremely powerful framework for building CLI/shell type interfaces
+** allows us to focus on implementing logic (like individual commands), not the shell itself
+* supports output redirection to local files
+* supports piping output through external programs ("| grep foo")
+* supports `argparse` to define commands and their options
+* allows to dynamically register and unregister commands
+* allows full control over tab completion items.
+== pySim-shell UI fundamentals
+* some global commands for things like PIN/CHV verification, enabling, disabling, unblcoking
+* you `select` your way through the card filesystem, like 'cd' on bash/zsh/...
+** tab-completion for [known] file names
+.Selecting a file
+pySIM-shell (MF/ADF.USIM)> select EF.IMSI
+ "file_descriptor": {
+ "shareable": true,
+ "file_type": "working_ef",
+ "structure": "transparent"
+ },
+ "file_identifier": "6F07",
+ "proprietary_info": {
+ "proprietary_D0": "20",
+ "proprietary_D2": "0F"
+ },
+ "life_cycle_status_int": "operational_activated",
+ "security_attrib_ref_expanded": "6F0603",
+ "file_size": 9,
+ "short_file_id": "38"
+== pySim-shell reading/writing
+* context-sensitive 'action' commands depend on the type of file selected
+** `read_binary`, `update_binary`, ... for transparent EF
+** `read_record`, `update_record`, ... for linear fixed EF
+.Reading and updating a transparent EF
+pySIM-shell (MF/ADF.USIM/EF.IMSI)> read_binary
+pySIM-shell (MF/ADF.USIM/EF.IMSI)> update_binary 089910070000400301
+pySIM-shell (MF/ADF.USIM/EF.IMSI)> read_binary
+== pySim-shell decoding/encoding
+* working with hex-encoded binary data is not very convenient
+* each specified file in the filesystem is represented by a python class
+** class can provide encoder + decoder methods to go from binary to JSON and vice-versa
+* some shell commands combine reading a file from the card + decoding / encoding
+** `read_binary_decoded`, `write_binary_decoded`, ... for transparent EF
+** `read_record_decoded`, `write_record_decoded`, ... for linear fixed EF
+.Reading and updating a decoded transparent EF
+pySIM-shell (MF/ADF.USIM/EF.IMSI)> read_binary_decoded
+ "imsi": "901700000043000"
+pySIM-shell (MF/ADF.USIM/EF.IMSI)> update_binary_decoded '{"imsi": "901700000043999"}'
+pySIM-shell (MF/ADF.USIM/EF.IMSI)> read_binary_decoded
+ "imsi": "901700000043999"
+== pySim-shell editing of single parameters
+* many files contain a lot of parameters
+* entering all of them as a single line in `update_binary_decoded` becomes rather cumbersome
+* `edit_binary_decoded` and `edit_record_decoded` to the rescue
+** reads data from the card
+** decodes it to JSON
+** spawns whatever is your favorite text editor
+** you do your editing, save + exit
+** re-encodes the modified JSON
+** writes the result to the card
+== pySim-shell `export`
+* one-shot command to iterate over all [known] files below the current DF and dump them
+* output format is structured in a way that you can use it as a pySim-shell script
+* results in easy method to backup and restore a card
+NOTE:: only the ETSI/3GPP standard files that are covered by pySim classes are exported. There may be any
+number of proprietary additional files on the card. Particularly K/OP/OPc or OTA keys are not exported this
+== pySim-shell advanced cmd2 usage
+* `run_script`: execute any text file as script inside the pySim-shell
+* `run_pyscript`: execute a python script with access to all of the state / classes / methods
+* `ipy`: run an interactive IPython session
+* `py`: run an interactive python interpreter
+** `card` represents the `` instance
+** `rs` represents the `pySim.filesystem.RuntimeState`
+.Using interactive python shell with access to pySim-shell state
+pySIM-shell (MF)> py
+Python 3.9.2 (default, Feb 28 2021, 17:03:44)
+[GCC 10.2.1 20210110] on linux
+Type "help", "copyright", "credits" or "license" for more information.
+End with `Ctrl-D` (Unix) / `Ctrl-Z` (Windows), `quit()`, `exit()`.
+Non-Python commands can be issued with: app("your command")
+>>> rs.selected_file
+<pySim.filesystem.CardMF object at 0x7f97952f3070>
+>>> rs.selected_file.desc
+'Master File (directory root)'
+== Other recent pySim developments
+* introduction of `sphinx` for generating documentation
+* adding API documentation at least for all of the *new* classes/methods that are used to build pySim-shell,
+ and which you need to extend to cover more files
+* automatic generation of pySim-shell command reference from `argparse` instances
+== Call for Contributions
+* pySim-shell is already very usable, but needs your help!
+** most files still need encoder + decoder methods to be implemented
+** we need automatic testing of encoders + decoders
+** we want JSON schema for the JSON representation of each file
+** we want to have batch programming in pySim-shell
+So if you know a bit of python and care about SIM cards, please send your patches!
+== Further Reading
+*[pySim project page on]
+*[pySim source code / git repository]
+*[Video of talk "SIM card technology from A-Z"]
+== EOF
+End of File
personal git repositories of Harald Welte. Your mileage may vary