Reading and writing INI and unix configs
- Relational pipes
- Principles
- Roadmap
- FAQ
- Specification
- Implementation
- Examples
- License
- Screenshots
- Download
- Support & contact
INI is very common simple text format used for configuration files. It extends classic key=value config files and adds one more level – sections – for structuring.
Unfortunatelly, there is no single INI standard and we have to deal with various INI dialects.
Some organizations specify their own INI dialect, but often INI files lacking any formal specification are used.
Our tools support various specifics and dialects through the --parser-option options.
Like any other CLI options, they are supported in our Bash completion scripts.
Classic (unix) key=value config files can be considered a dialect or a subset of the INI format and thus be processed by INI input and output filters. It is like INI without any sections or INI having only global properties (key=value at the beginning of the file). Java .properties and MANIFEST.MF files are similar case. They also can be processed by our tools.
Simple INI example
This is a simple INI file:
; this entry is a „global property“ – does not belong to any section:
color=blue
[first-section]
x = 10
y = 20
z = 30
some = value
other = VALUE
[second-section]
x = 100
y = 200
z = 300
We read such INI file using this command:
cat ini-simple.ini | relpipe-in-ini | relpipe-out-tabular
and get:
ini:
╭──────────────────┬──────────────┬────────────────╮
│ section (string) │ key (string) │ value (string) │
├──────────────────┼──────────────┼────────────────┤
│ │ color │ blue │
│ first-section │ x │ 10 │
│ first-section │ y │ 20 │
│ first-section │ z │ 30 │
│ first-section │ some │ value │
│ first-section │ other │ VALUE │
│ second-section │ x │ 100 │
│ second-section │ y │ 200 │
│ second-section │ z │ 300 │
╰──────────────────┴──────────────┴────────────────╯
Record count: 9
Of course, we can do more that listing the entries as a table. We can convert them to other formats or e.g. run some command on each entry. Or modify certain values or add/remove entries and save as new INI file.
More complex INI example
The INI format is just seemingly simple. There is much more than section/key/value.
Our parser is by default configured to eat as much INI as possible.
It can be tuned by --parser-option CLI options.
Sometimes, such tuning is necessary, because some INI features are conflicting – some INI dialects are mutually exclusive.
(please note that syntax highlighting does not support advanced INI features and is sometimes broken)
global=value
[section][section-tag]
; section tags are currently supported by the parser (do not generate error)
; but do not generate any output
section-tag-support=partial
[quoting]
quoted = "some text in quotes"
apostrophed = 'some text in apostrophes'
unquoted = values are trimmed
we-need = " quotes or apostrophes"
to-keep = 'the leading/trailing whitespace '
[multiline]
quoted = "first line
second line"
apostrophed = 'first line
second line'
; unlike quoted/apostrophed strings, there is no line end inside the value:
backslashed = first line \
continues here
[comments] ; we can comment also secions
ini = "true" ; and even values
unix = true
; classic INI comment
# unix-style comments are also supported
but = be aware that ; this is part of the value, not a comment
; we need '' or "" to put comment on the same line as the value
[subkeys]
; we may specify subkeys in [] brackets:
a[x] = AX
a[y] = AY
; because this feature is quite uncommon, it is disabled by default
; and whole key+subkey becomes a key
; but we can turn on the subkey support by the CLI option: --enable-sub-keys true
; and get separete key and sub_key attributes on the output
[escaping]
escaped-multiline = first line\nsecond line
backslash = back\\slash
quotes = "quoted text with \"quotes\" inside"
tab = how\tvery
We read such INI file using this command:
cat ini-simple.ini | relpipe-in-ini --enable-sub-keys true | relpipe-out-tabular
and get:
ini:
╭──────────────────┬─────────────────────┬──────────────────┬──────────────────────────────────────────────────────────╮
│ section (string) │ key (string) │ sub_key (string) │ value (string) │
├──────────────────┼─────────────────────┼──────────────────┼──────────────────────────────────────────────────────────┤
│ │ global │ │ value │
│ section │ section-tag-support │ │ partial │
│ quoting │ quoted │ │ some text in quotes │
│ quoting │ apostrophed │ │ some text in apostrophes │
│ quoting │ unquoted │ │ values are trimmed │
│ quoting │ we-need │ │ quotes or apostrophes │
│ quoting │ to-keep │ │ the leading/trailing whitespace │
│ multiline │ quoted │ │ first line↲second line │
│ multiline │ apostrophed │ │ first line↲second line │
│ multiline │ backslashed │ │ first line continues here │
│ comments │ ini │ │ true │
│ comments │ unix │ │ true │
│ comments │ but │ │ be aware that ; this is part of the value, not a comment │
│ subkeys │ a │ x │ AX │
│ subkeys │ a │ y │ AY │
│ escaping │ escaped-multiline │ │ first line↲second line │
│ escaping │ backslash │ │ back\slash │
│ escaping │ quotes │ │ quoted text with "quotes" inside │
│ escaping │ tab │ │ how↹very │
╰──────────────────┴─────────────────────┴──────────────────┴──────────────────────────────────────────────────────────╯
Record count: 19
If we omit the --enable-sub-keys true option, the sub_key attribute disappears
and we get a[x] and a[y] in the key attribute
i.e. brackets are not interpreted in any special way – they are considered just part of the key.
See Bash completion for complete list of options and dialects.
Keeping comments and whitespace while modifying the INI files
Comments and empty lines are ignored by default and does not generate any output.
This is desired behavior in most cases.
But sometimes we want to pass them through.
We may e.g. generate some XHTML report or other user-friendly output containing the comments,
or we may want to modify existing INI without losing original comments and formatting.
The tool offers the --enable-comments and --enable-whitespace options.
They allows lossless or almost lossless round-trip conversion from INI to relations and back.
Besides that, we have also the --enable-event-numbers and --enable-line-numbers options.
But they are useful mostly for debugging purposes.
If we enable all that options, we get this output:
ini:
╭────────────────┬─────────────────┬──────────────────┬─────────────────────┬──────────────────┬──────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────┬─────────────────────╮
│ line (integer) │ event (integer) │ section (string) │ key (string) │ sub_key (string) │ value (string) │ comment (string) │ whitespace (string) │
├────────────────┼─────────────────┼──────────────────┼─────────────────────┼──────────────────┼──────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┼─────────────────────┤
│ 1 │ 1 │ │ global │ │ value │ │ │
│ 2 │ 2 │ │ │ │ │ │ ↲ │
│ 4 │ 4 │ section │ │ │ │ section tags are currently supported by the parser (do not generate error) │ │
│ 5 │ 5 │ section │ │ │ │ but do not generate any output │ │
│ 6 │ 6 │ section │ section-tag-support │ │ partial │ │ │
│ 7 │ 7 │ section │ │ │ │ │ ↲ │
│ 9 │ 9 │ quoting │ │ │ │ │ ↲ │
│ 10 │ 10 │ quoting │ quoted │ │ some text in quotes │ │ │
│ 11 │ 11 │ quoting │ apostrophed │ │ some text in apostrophes │ │ │
│ 12 │ 12 │ quoting │ │ │ │ │ ↲ │
│ 13 │ 13 │ quoting │ unquoted │ │ values are trimmed │ │ │
│ 14 │ 14 │ quoting │ we-need │ │ quotes or apostrophes │ │ │
│ 15 │ 15 │ quoting │ to-keep │ │ the leading/trailing whitespace │ │ │
│ 16 │ 16 │ quoting │ │ │ │ │ ↲ │
│ 18 │ 18 │ multiline │ │ │ │ │ ↲ │
│ 19 │ 19 │ multiline │ quoted │ │ first line↲second line │ │ │
│ 21 │ 20 │ multiline │ │ │ │ │ ↲ │
│ 22 │ 21 │ multiline │ apostrophed │ │ first line↲second line │ │ │
│ 24 │ 22 │ multiline │ │ │ │ │ ↲ │
│ 25 │ 23 │ multiline │ │ │ │ unlike quoted/apostrophed strings, there is no line end inside the value: │ │
│ 26 │ 24 │ multiline │ backslashed │ │ first line continues here │ │ │
│ 28 │ 25 │ multiline │ │ │ │ │ ↲ │
│ 29 │ 26 │ comments │ │ │ │ we can comment also secions │ │
│ 30 │ 27 │ comments │ │ │ │ │ ↲ │
│ 31 │ 28 │ comments │ ini │ │ true │ and even values │ │
│ 32 │ 29 │ comments │ unix │ │ true │ │ │
│ 33 │ 30 │ comments │ │ │ │ │ ↲ │
│ 34 │ 31 │ comments │ │ │ │ classic INI comment │ │
│ 35 │ 32 │ comments │ │ │ │ unix-style comments are also supported │ │
│ 36 │ 33 │ comments │ │ │ │ │ ↲ │
│ 37 │ 34 │ comments │ but │ │ be aware that ; this is part of the value, not a comment │ │ │
│ 38 │ 35 │ comments │ │ │ │ we need '' or "" to put comment on the same line as the value │ │
│ 39 │ 36 │ comments │ │ │ │ │ ↲ │
│ 41 │ 38 │ subkeys │ │ │ │ │ ↲ │
│ 42 │ 39 │ subkeys │ │ │ │ we may specify subkeys in [] brackets: │ │
│ 43 │ 40 │ subkeys │ a │ x │ AX │ │ │
│ 44 │ 41 │ subkeys │ a │ y │ AY │ │ │
│ 45 │ 42 │ subkeys │ │ │ │ │ ↲ │
│ 46 │ 43 │ subkeys │ │ │ │ because this feature is quite uncommon, it is disabled by default │ │
│ 47 │ 44 │ subkeys │ │ │ │ and whole key+subkey becomes a key │ │
│ 48 │ 45 │ subkeys │ │ │ │ but we can turn on the subkey support by the CLI option: --enable-sub-keys true │ │
│ 49 │ 46 │ subkeys │ │ │ │ and get separete key and sub_key attributes on the output │ │
│ 50 │ 47 │ subkeys │ │ │ │ │ ↲ │
│ 52 │ 49 │ escaping │ │ │ │ │ ↲ │
│ 53 │ 50 │ escaping │ escaped-multiline │ │ first line↲second line │ │ │
│ 54 │ 51 │ escaping │ backslash │ │ back\slash │ │ │
│ 55 │ 52 │ escaping │ quotes │ │ quoted text with "quotes" inside │ │ │
│ 56 │ 53 │ escaping │ tab │ │ how↹very │ │ │
│ 57 │ 54 │ escaping │ │ │ │ │ ↲↲ │
╰────────────────┴─────────────────┴──────────────────┴─────────────────────┴──────────────────┴──────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────┴─────────────────────╯
Record count: 49
So we can take the simple INI file (see above) and pass it through a filter
that changes (on-the-fly) the x value in the first-section:
cat ini-simple.ini \
| relpipe-in-ini --enable-comments true \
| relpipe-tr-scheme \
--relation 'ini' \
--for-each '
(if (and (string= $section "first-section") (string= $key "x"))
(set! $value "256"))' \
| relpipe-out-ini
and get an INI file that contains original comments and modified values:
; this entry is a „global property“ – does not belong to any section:
color = blue
[first-section]
x = 256
y = 20
z = 30
some = value
other = VALUE
[second-section]
x = 100
y = 200
z = 300
Both the comment at the beginning of the file
and the x value in second-section
were kept intact.
cat ini-simple.ini \
| relpipe-in-ini --enable-comments true \
| relpipe-tr-awk \
--relation '.*' \
--for-each '(section == "first-section" && key == "x") { value = "256"; }; record();' \
| relpipe-out-ini
This AWK filter will generate the same output.
And we can simply add comments just by adding e.g. comment = "the script changed this value"; after the value = "256";
which results in:
x = 256 ; the script changed this value
This way we can automatically modify our configuration files.
Generating INI files
The relpipe-out-ini tool serves not only for recreating already existing INI files but also for creating new ones.
Data may come from various sources – CSV editor, script, SQL query etc. – and may come in diferent shapes.
While on the INI side we use the term „dialect“, on the relational side we use the term „style“.
So in relpipe-out-ini we can chose from several styles:
-
standard: expects same structure as generated byrelpipe-in-ini; can generate also comments and whitespace -
literal: relation name → section name; attribute name → key; attribute value → value; multiple records → duplicate section names -
section-first: likeliteral, just section names are taken from the first attribute -
dropped: this relation is ignored/skipped -
auto: usestandardstyle if the relation haskeyandvalueattributes; otherwise useliteralstyle
Each relation might be processed in different way – use the --relation and regular expression describing the relation(s) name.
We may for example run:
relpipe-in-x11 --list-input-devices true \
| relpipe-tr-awk \
--relation '.*' \
--output-attribute 'type' string \
--output-attribute 'id' integer \
--output-attribute 'name' string \
--where 'type == "mouse" || type == "keyboard"' \
| relpipe-out-ini --style section-first
and get INI like this:
[keyboard]
id = 6
name = Power Button
[keyboard]
id = 7
name = Video Bus
[keyboard]
id = 8
name = Power Button
[mouse]
id = 10
name = Logitech USB Trackball
[keyboard]
id = 16
name = AT Translated Set 2 keyboard
[keyboard]
id = 12
name = ZSA Technology Labs Inc ErgoDox EZ Shine Keyboard
[mouse]
id = 9
name = 3Dconnexion 3Dconnexion Universal Receiver Mouse
This way, we can generate INI files to be loaded as configuration files by other programs.
We can also use relpipe-out-ini for displaying data rather vertically than horizontally (like in common relpipe-out-tabular).
This is useful for data with long values or many attributes.
The relpipe-out-recfile tool can also server this purpose.
Generating XHTML report from several INI files
INI format is used by many programs like Mercurial, Git, KDE, Gnome… and also by ODBC. Its configuration is spread across several INI files (system-wide driver configuration and connections and user-specific connections). We can collect all these INI files, use their names as relation names and generate a nice XHTML report:
for file in /etc/odbcinst.ini /etc/odbc.ini ~/.odbc.ini ; do
cat "$file" | relpipe-in-ini --relation "$(basename $file)";
done | relpipe-out-xhtml > odbc-report.xhtml
n.b. It might contain also sensitive information like passwords.
We may use some relpipe-tr-* filter to remove such records or replace such values by ********.
Java .properties and MANIFEST.MF
Java has built-in support for simple key=value format – so called .properties files.
They are often used for storing flat data where more advanced format (like XML) is not necessary.
This format supports comments and some escaping sequences.
Escaping of non-ASCII characters was mandatory but since Java 9 the UTF-8 encoding is used by default (no need to call native2ascii).
In Relational pipes we consider .properties files to be a dialect of the INI format so it is supported in relpipe-in-ini and relpipe-out-ini.
The same applies to the MANIFEST.MF which is another key=value format used in the Java world.
Thus we can read and write e.g. Java resource bundles (localization), Java EE or Spring application configs or MANIFEST.MF containing OSGi metadata and other information.
# read .properties or MANIFEST.MF:
cat example.properties | relpipe-in-ini --parser-option dialect java-properties | …
cat MANIFEST.MF | relpipe-in-ini --parser-option dialect java-manifest-mf | …
# write .properties or MANIFEST.MF:
… | relpipe-out-ini --writer-option dialect java-properties > example.properties
… | relpipe-out-ini --writer-option dialect java-manifest-mf > MANIFEST.MF
We can pass such data through various transformation (SQL, AWK, Scheme, XPath etc.) and do conversions to/from various formats (CSV, INI, XML, XHTML, YAML, Recfile, ASN.1 etc.) or convert .properties to MANIFEST.MF and vice versa.
Classic unix key=value config files
Many unix or GNU/Linux programs use simple key=value configuration files with # comments
and some basic escaping (like \\ → \, \n → new line etc.)
It is very similar to Java .properties mentioned above.
The configuration is often spread across many small files heavily stuffed with comments.
If we want to collect just the valid configuration entries from all the files,
we may somehow misuse the INI input filter and the fact that given files contain no sections – turn the filenames into section names
and feed the result to the relpipe-in-ini:
head -n -0 /etc/sysctl.d/*.conf \
| sed -E 's/==> (.*) <==/[\1]/g' \
| relpipe-in-ini --relation 'sysctl' \
| relpipe-out-tabular
This gives us nice overview of our system configuration:
sysctl:
╭─────────────────────────────────────────┬────────────────────────────────────┬────────────────╮
│ section (string) │ key (string) │ value (string) │
├─────────────────────────────────────────┼────────────────────────────────────┼────────────────┤
│ /etc/sysctl.d/uhd-usrp2.conf │ net.core.rmem_max │ 50000000 │
│ /etc/sysctl.d/uhd-usrp2.conf │ net.core.wmem_max │ 1048576 │
│ /etc/sysctl.d/10-console-messages.conf │ kernel.printk │ 4 4 1 7 │
│ /etc/sysctl.d/10-ipv6-privacy.conf │ net.ipv6.conf.all.use_tempaddr │ 2 │
│ /etc/sysctl.d/10-ipv6-privacy.conf │ net.ipv6.conf.default.use_tempaddr │ 2 │
│ /etc/sysctl.d/10-kernel-hardening.conf │ kernel.kptr_restrict │ 1 │
│ /etc/sysctl.d/10-link-restrictions.conf │ fs.protected_hardlinks │ 1 │
│ /etc/sysctl.d/10-link-restrictions.conf │ fs.protected_symlinks │ 1 │
│ /etc/sysctl.d/10-magic-sysrq.conf │ kernel.sysrq │ 176 │
│ /etc/sysctl.d/10-network-security.conf │ net.ipv4.conf.default.rp_filter │ 1 │
│ /etc/sysctl.d/10-network-security.conf │ net.ipv4.conf.all.rp_filter │ 1 │
│ /etc/sysctl.d/10-network-security.conf │ net.ipv4.tcp_syncookies │ 1 │
│ /etc/sysctl.d/10-ptrace.conf │ kernel.yama.ptrace_scope │ 1 │
│ /etc/sysctl.d/10-zeropage.conf │ vm.mmap_min_addr │ 65536 │
│ /etc/sysctl.d/99-sysctl.conf │ kernel.sysrq │ 1 │
╰─────────────────────────────────────────┴────────────────────────────────────┴────────────────╯
Record count: 15
We can also do this:
find /etc/sysctl.d/ -name '*.conf' \
| while read f; do echo "[$f]"; cat "$f"; done \
| relpipe-in-ini \
| relpipe-out-tabular
Or create a reusable function:
list-config() {
head -n -0 "$@" \
| sed -E 's/==> (.*) <==/[\1]/g' \
| relpipe-in-ini --relation "configs" \
| relpipe-out-tabular --write-types false;
}
and call it this way:
list-config /etc/sysctl.d/*.conf
It is a bit dirty hack (what if the file name, key or value contains <== or […]?) but it will work quite well.
And like with Java .properties and MANIFEST-MF, we can use the relpipe-out-ini to generate the configuration files.
Relational pipes, open standard and free software © 2018-2022 GlobalCode