Recfile is the native format of GNU Recutils. Recfiles are text files that contain records of various types. They are human-editable and serve as simple databases. Relational pipes support input and output in this format since v0.11.
We can convert any relational data to the recfile format by using relpipe-out-recfile
– e.g. our fstab
will look like this:
%rec: fstab
%type: dump int
%type: pass int
scheme: UUID
device: 29758270-fd25-4a6c-a7bb-9a18302816af
mount_point: /
type: ext4
options: relatime,user_xattr,errors=remount-ro
dump: 0
pass: 1
scheme:
device: /dev/sr0
mount_point: /media/cdrom0
type: udf,iso9660
options: user,noauto
dump: 0
pass: 0
scheme:
device: /dev/sde
mount_point: /mnt/data
type: ext4
options: relatime,user_xattr,errors=remount-ro
dump: 0
pass: 2
scheme: UUID
device: a2b5f230-a795-4f6f-a39b-9b57686c86d5
mount_point: /home
type: btrfs
options: relatime
dump: 0
pass: 2
scheme:
device: /dev/mapper/sdf_crypt
mount_point: /mnt/private
type: xfs
options: relatime
dump: 0
pass: 2
Then we can edit this data (e.g. in GNU Emacs which has mode for this format) or store it in a version control system like Mercurial or Git.
Because it is a text format (like XML, which is also supported and good for this purpose),
we can efficiently track changes in data across versions, do diff
or (with some care) even patch
.
And we can use whole GNU Recutils toolchain while working with such data.
Obligatory example of filtering our fstab
:
relpipe-in-fstab | relpipe-out-recfile | recsel -e "type = 'btrfs' || type = 'xfs'"
Will give us a recfile:
scheme: UUID
device: a2b5f230-a795-4f6f-a39b-9b57686c86d5
mount_point: /home
type: btrfs
options: relatime
dump: 0
pass: 2
scheme:
device: /dev/mapper/sdf_crypt
mount_point: /mnt/private
type: xfs
options: relatime
dump: 0
pass: 2
And we can convert it back to the relational format using relpipe-in-recfile
:
relpipe-in-fstab \
| relpipe-out-recfile \
| recsel -e "type = 'btrfs' || type = 'xfs'" \
| relpipe-in-recfile \
| relpipe-out-tabular
and print as a table in our terminal:
recfile:
╭─────────────────┬──────────────────────────────────────┬──────────────────────┬───────────────┬──────────────────┬───────────────┬───────────────╮
│ scheme (string) │ device (string) │ mount_point (string) │ type (string) │ options (string) │ dump (string) │ pass (string) │
├─────────────────┼──────────────────────────────────────┼──────────────────────┼───────────────┼──────────────────┼───────────────┼───────────────┤
│ UUID │ a2b5f230-a795-4f6f-a39b-9b57686c86d5 │ /home │ btrfs │ relatime │ 0 │ 2 │
│ │ /dev/mapper/sdf_crypt │ /mnt/private │ xfs │ relatime │ 0 │ 2 │
╰─────────────────┴──────────────────────────────────────┴──────────────────────┴───────────────┴──────────────────┴───────────────┴───────────────╯
Record count: 2
n.b. in the v0.11 the conversion to recfiles and back is not 100% lossless (unlike XML) because Relational pipes support only three data types (string, unsigned integer and boolean) in this version; this will be improved in later releases (more data types are planned before v1.0)
Because some web browsers or tools can store the original URL in extended attributes while downloading a file,
we can use recsel
to find files downloaded from some particular domain:
find -print0 | relpipe-in-filesystem \
--file path \
--file size \
--file type \
--xattr xdg.origin.url --as url \
| relpipe-out-recfile \
| recsel -e 'url ~ "^https?://([^/]*\.)?archive\.org/"'
Relational pipes can be also used together with SQL-DK (in 2019-03-05 development version) to pipe data from big relational databases like PostgreSQL or MariaDB to other formats like recfiles. Having a script:
sql-dk --db "$postgresql_db" --sql "
SELECT
table_schema,
table_name,
table_type
FROM information_schema.tables
WHERE table_schema IN ('pg_catalog', 'information_schema')
LIMIT 3;
SELECT
typName,
typCategory,
typByVal
FROM pg_type
LIMIT 4" \
--relation tables \
--relation types \
--formatter xml \
| relpipe-in-xml
Download: examples/sql-dk_pg_1.sh
We can convert result sets from any SQL queries to relational format and then work with such data without connection to the original database. Thus we can cache (materialize) the results locally in a file and use them even offline. Or we can run the SQL query each time and have fresh data:
sql-dk_pg_1.sh | relpipe-out-recfile
Will result in:
%rec: tables
table_schema: pg_catalog
table_name: pg_type
table_type: BASE TABLE
table_schema: pg_catalog
table_name: pg_roles
table_type: VIEW
table_schema: pg_catalog
table_name: pg_group
table_type: VIEW
%rec: types
%type: typbyval bool
typname: bool
typcategory: B
typbyval: true
typname: bytea
typcategory: U
typbyval: false
typname: char
typcategory: S
typbyval: true
typname: name
typcategory: S
typbyval: false
Or we can view the data in classic tabular way using relpipe-out-tabular
:
tables:
╭───────────────────────┬─────────────────────┬─────────────────────╮
│ table_schema (string) │ table_name (string) │ table_type (string) │
├───────────────────────┼─────────────────────┼─────────────────────┤
│ pg_catalog │ pg_type │ BASE TABLE │
│ pg_catalog │ pg_roles │ VIEW │
│ pg_catalog │ pg_group │ VIEW │
╰───────────────────────┴─────────────────────┴─────────────────────╯
Record count: 3
types:
╭──────────────────┬──────────────────────┬────────────────────╮
│ typname (string) │ typcategory (string) │ typbyval (boolean) │
├──────────────────┼──────────────────────┼────────────────────┤
│ bool │ B │ true │
│ bytea │ U │ false │
│ char │ S │ true │
│ name │ S │ false │
╰──────────────────┴──────────────────────┴────────────────────╯
Record count: 4
Materialized (or fresh) data from the database can be further transformed
using relpipe-tr-*
commands like grep, sed, cut, guile,
or (through the recfile conversion) by the recsel
command from GNU Recutils.
The relpipe-in-recfile
will help with conversion of recfiles to various formats like XHTML,
pretty-printing or with xargs-like processing
(using relpipe-out-nullbyte
and regular xargs
or read_nullbyte
function
as described in the Writing an output filter in Bash example).
Thus we can have data-driven Bash scripts based on our recfiles.
Relational pipes, open standard and free software © 2018-2022 GlobalCode