Release v0.18
- Relational pipes
- Principles
- Roadmap
- FAQ
- Specification
- Implementation
- Examples
- License
- Screenshots
- Download
- Support & contact
We are pleased to introduce you the new development version of Relational pipes. This release brings several new tools of each type (inputs, outputs, transformations and streamlet) and some smaller improvements.
- New input tools for reading various formats: read ASN.1, YAML and JSON; see details below
- New XMLTable-like input tools for converting arbitrary tree structures to relations: read ASN.1, INI, MIME, YAML, JSON, CBOR and HTML; see details below
- New input tool and streamlet for reading barcodes and QR codes: see details below
- New input and output tool for the INI format: see details below
- New input and output tool for interacting with X11: see details below
- New XPath transformation: see details below
- Data types support on CSV input and output: strings, integers and booleans; see details below
- The grep, cut and sed transformations have new CLI interface: see details below
See the (new) examples and (new) screenshots pages for details.
Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. Any suggestions, ideas and bug reports are welcome in our mail box.
New standard input modules
These tools allows serialization of arbitrary relational data
and then reading resulting file or stream back to the relational data.
An of course, we can read data created or generated somewhere else.
Since this release, we can newly serialize our relations to YAML and ASN.1, store, edit or transfer these files or streams and then deserialize them and pipe them through relational transformations or outputs.
Formats like XML or Recfiles were already supported in previous releases.
Full lossless round-trip is now possible with: XML, YAML, ASN.1 and Recfile formats.
Partial lossless round-trip is possible with CSV – these input and output tools are limited to a single relation.
Lossy round-trip is possible with relpipe-out-nullbyte and relpipe-in-cli –
the nullbyte stream may contain only a single relation and its data types need to be passed as CLI arguments to the input tool.
Other formats (e.g. JACK, INI or X11) may also have corresponding inputs and outputs, but they produce and consume some specific structures – they are not universal and usable with arbitrary relational data.
YAML and JSON
YAML is a text format for serializing tree structures into text (like XML).
It can be sometimes tricky, but basic scenarios are intuitive and straightforward.
We can use YAML to describe relational data and load them using the relpipe-in-yaml input filter
and we can produce YAML files using the relpipe-out-yaml output filter.
More details in the example: Reading and writing YAML
ASN.1
There are actually more ASN.1 applications around us than most people realize. ASN.1 is almost everywhere – in cryptography (X.509 / TLS / HTTP, S/MIME etc.), LDAP, payment cards, telecommunications (including GSM, GPRS, EDGE, UMTS, LTE, 5G), Kerberos authentication etc. So it is very useful to be able to access these data. Because ASN.1 is abstract, there are several ways how such data can be serialized into a stream of octets (bytes). Most common way is BER (Basic Encoding Rules), so we support BER (and thus its variants DER and CER) in the first version of our tools. Later other encoding might be supported and we could also parse the ASN.1 syntax (the schema language).
ASN.1 BER output was added several releases ago.
Now we have also ASN.1 BER input (relpipe-in-asn1) so we can read data in this format generated earlier.
Of course, BER data can be generated or read also by other ASN.1 capable software.
However, maybe more interesting than relpipe-in-asn1 is relpipe-in-asn1table (see below) that reads arbitrary ASN.1 BER data (not only serialized relational data).
Reading and writing INI
INI is very common simple text format used for configuration files. It extends classic key=value config files and adds one more level – sections – for structuring. In this release we got input and output filter for INI and similar formats (Java .properties, MANIFEST.MF, key=value configs). Because comments and whitespace are also supported, we can do (almost) loss-less conversions and transformations (e.g. change value of an entry).
More details in the example: Reading and writing INI and unix configs
Reading barcodes and QR codes
Either ubiquitous 1D barcodes (procudct labels, ISBN in form of EAN-13 etc.) or 2D barcodes (QR containing hyperlinks, vCards etc.)
encode numbers, texts or other data into images that can be printed and scanned or photographed later.
Now we can interact with this technology – read barcodes using the
relpipe-in-barcode tool or barcode-reader streamlet.
More details in the example: Reading barcodes and QR
New XMLTable-like input modules: ASN.1, INI, MIME, YAML, JSON, CBOR, HTML
More details in the examples:
- Reading and querying JSON, YAML, CBOR, HTML, MIME, INI, ASN.1 and XML in a uniform way
- Exploring content of X.509 certificates
X11 input and output modules
The X Window System or X11 comes from 1984 and is still widely used (despite we have some other options like Wayland).
This protocol and set of libraries and interfaces gives us GUI (graphical user interface), manages our displays, windows, keyboards and mice.
Since this release of Relational pipes we can interact with this wonderful technology through the relpipe-in-x11 and relpipe-out-x11 tools.
In the following example we will show how to acquire information about the input devices, screens and windows or capture and emit X11 events (key presses and mouse movements).
More details in the example: Exploring X11 windows and devices and emulating mouse movements and keystrokes
XPath transformation
We got a new powerful language for filtering and transformations: XPath.
It is now part of the toolset consisting of SQL, AWK, Scheme and others.
However XPath is originally a language designed for XML, in Relational pipes we can use it for relational data coming from various sources, not only XML,
and also for data that violates the rules of normal forms.
We can process quite complex tree structures entangled in records but we can also write simple and intuitive expressions like x = "a" or y = 123.
More details in the example: Filtering and transforming relational data with XPath
Data types in CSV + generic relpipe-tr-infertypes tool
CSV input and output filters now support data types i.e. CSV can now carry not only text strings, but also booleans and integers (or more types in future releases).
We also have a new tool relpipe-tr-infertypes that can automatically recognize the types in data that came from CSV or any other source.
More details in the example: CSV and data types
Feature overview
Data types
- boolean ~ since v0.8
- variable-length signed integer (SLEB128) ~ since v0.15
- string in UTF-8 ~ since v0.8
Inputs
- Recfile ~ since v0.11
- XML ~ since v0.9
- XMLTable ~ since v0.13
- CSV ~ since v0.9
- file system ~ since v0.9
- CLI ~ since v0.8
- fstab ~ since v0.8
- SQL script ~ since v0.14
- JACK ~ since v0.16
- barcode
- X11
- ASN.1 BER
- ASN.1 BER Table
- INI
- INITable
- MIMETable
- YAML
- YAMLTable
- JSON
- JSONTable
- CBORTable
- HTMLTable
Transformations
- sql: filtering and transformations using the SQL language ~ since v0.13
- xpath: filtering and transformations using the XPath language
- awk: filtering and transformations using the classic AWK tool and language ~ since v0.12
- scheme: filtering and transformations defined in the Scheme language using GNU Guile ~ since v0.10
- grep: regular expression filter, removes unwanted records from the relation ~ since v0.8
- cut: regular expression attribute cutter (removes or duplicates attributes and can also DROP whole relation) ~ since v0.8
- sed: regular expression replacer ~ since v0.8
- validator: just a pass-through filter that crashes on invalid data ~ since v0.8
- infertypes: derive data types from attribute values
- python: highly experimental ~ since v0.8
Streamlets
- xpath (example, unstable) ~ since v0.15
- hash (example, unstable) ~ since v0.15
- jar_info (example, unstable) ~ since v0.15
- mime_type (example, unstable) ~ since v0.15
- exiftool (example, unstable) ~ since v0.15
- pid (example, unstable) ~ since v0.15
- cloc (example, unstable) ~ since v0.15
- exiv2 (example, unstable) ~ since v0.15
- inode (example, unstable) ~ since v0.15
- lines_count (example, unstable) ~ since v0.15
- pdftotext (example, unstable) ~ since v0.15
- pdfinfo (example, unstable) ~ since v0.15
- tesseract (example, unstable) ~ since v0.15
- barcode (example, unstable)
Outputs
- ASN.1 BER ~ since v0.11
- Recfile ~ since v0.11
- CSV ~ since v0.9
- tabular ~ since v0.8
- XML ~ since v0.8
- nullbyte ~ since v0.8
- GUI in Qt ~ since v0.8
- ODS (LibreOffice) ~ since v0.8
- JACK ~ since v0.17
- X11
- INI
- YAML
New examples
- Reading and querying JSON, YAML, CBOR, HTML, MIME, INI, ASN.1 and XML in a uniform way
- Filtering and transforming relational data with XPath
- CSV and data types
- Exploring X11 windows and devices and emulating mouse movements and keystrokes
- Exploring content of X.509 certificates
- Running SQL JOINs on multiple CSV files
- Reading and writing YAML
- Reading and writing INI and unix configs
- Reading barcodes and QR
New screenshots
Backward incompatible changes
In relpipe-tr-sql and relpipe-in-sql the --list-data-sources option has now boolean parameter
i.e. it is not the presence of the option but the true or false value what matters.
We can also list the data sources while we simultaneously run some SQL.
The relpipe-tr-grep, relpipe-tr-cut and relpipe-tr-sed
were the last tools with the obsolete positional CLI interface.
In this release, they were updated and now share the same style of CLI arguments as other tools.
Besides consistent and predictable user interface, these tools are now more powerful because they can transforma more relations at once.
Instead of e.g.
relpipe-tr-grep 'a' 'b' 'c'
we now write:
relpipe-tr-grep --relation 'a' --attribute 'b' --value 'c'
Bash completion helps us while writing such commands and result is much more readable than original cryptic version.
See updated examples:
- Renaming VG in /etc/fstab using relpipe-tr-sed
- Using relpipe-tr-sed with groups and backreferences
- Filtering /etc/fstab using relpipe-tr-grep
- Doing projection and restriction using cut and grep
- Reading apt (Debian package system) results
- Generating statistics and charts using XMLTable
- Streamlets preview
- Running SQL JOINs on multiple CSV files
- Finding runnable JARs
Installation
Instalation was tested on Debian GNU/Linux 10.2. The process should be similar on other distributions.
# Install dependencies as root:
if [[ "$USER" == "root" ]]; then
RELPIPE_DEPENDENCIES=(g++ make cmake mercurial pkg-config)
RELPIPE_DEPENDENCIES+=(libxerces-c-dev) # needed only for relpipe-in-xml module
RELPIPE_DEPENDENCIES+=(guile-2.2-dev) # needed only for relpipe-tr-scheme module
RELPIPE_DEPENDENCIES+=(gawk) # needed only for relpipe-tr-awk module
RELPIPE_DEPENDENCIES+=(libxml++2.6-dev) # needed only for relpipe-in-xmltable module
RELPIPE_DEPENDENCIES+=(unixodbc-dev) # needed only for relpipe-tr-sql module
RELPIPE_DEPENDENCIES+=(libsqliteodbc) # needed only for relpipe-tr-sql module if used with SQLite
RELPIPE_DEPENDENCIES+=(odbc-postgresql) # needed only for relpipe-tr-sql module if used with PostgreSQL
RELPIPE_DEPENDENCIES+=(libjack-jackd2-dev) # needed only for relpipe-in-jack and relpipe-out-jack modules; Jack1 should also work
RELPIPE_DEPENDENCIES+=(libxi-dev libxtst-dev) # needed only for relpipe-in-x11 and relpipe-out-x11
RELPIPE_DEPENDENCIES+=(libzbar-dev libmagick++-dev) # needed only for relpipe-in-barcode and the barcode-reader streamlet
RELPIPE_DEPENDENCIES+=(libgumbo-dev) # needed only for relpipe-in-htmltable
RELPIPE_DEPENDENCIES+=(libyaml-dev) # needed only for relpipe-in-yaml, relpipe-in-yamltable, relpipe-in-json and relpipe-in-jsontable
RELPIPE_DEPENDENCIES+=(libcbor-dev) # needed only for relpipe-in-cbortable
RELPIPE_DEPENDENCIES+=(libvmime-dev) # needed only for relpipe-in-mimetable
apt install "${RELPIPE_DEPENDENCIES[@]}"
exit;
fi
# Run rest of installation as a non-root user:
export RELPIPE_VERSION=${RELPIPE_VERSION:-"v0.18"}
export RELPIPE_SRC=~/src
export RELPIPE_BUILD=~/build
export RELPIPE_INSTALL=~/install
export PKG_CONFIG_PATH="$RELPIPE_INSTALL/lib/pkgconfig/:$PKG_CONFIG_PATH"
export PATH="$RELPIPE_INSTALL/bin:$PATH"
rm -rf "$RELPIPE_BUILD"/relpipe-*
mkdir -p "$RELPIPE_SRC" "$RELPIPE_BUILD" "$RELPIPE_INSTALL"
# Helper functions:
relpipe_download() { for m in "$@"; do cd "$RELPIPE_SRC" && ([[ -d "relpipe-$m.cpp" ]] && hg pull -R "relpipe-$m.cpp" && hg update -R "relpipe-$m.cpp" "$RELPIPE_VERSION" || hg clone -u "$RELPIPE_VERSION" https://hg.globalcode.info/relpipe/relpipe-$m.cpp) || break; done; }
relpipe_install() { for m in "$@"; do cd "$RELPIPE_BUILD" && mkdir -p relpipe-$m.cpp && cd relpipe-$m.cpp && cmake -DCMAKE_INSTALL_PREFIX:PATH="$RELPIPE_INSTALL" "$RELPIPE_SRC/relpipe-$m.cpp" && make && make install || break; done; }
RELPIPE_LIBS=(lib-common lib-reader lib-writer lib-cli lib-xmlwriter)
RELPIPE_TOOLS=(in-cli in-fstab in-xml in-xmltable in-csv in-filesystem in-recfile out-nullbyte out-ods out-tabular out-xml out-csv out-asn1 out-recfile tr-cut tr-grep tr-sed tr-validator tr-scheme tr-awk tr-sql in-jack out-jack)
RELPIPE_TOOLS+=(in-asn1 in-asn1table in-barcode in-cbortable in-htmltable in-ini in-initable in-mimetable in-x11 in-yaml in-yamltable out-ini out-x11 out-yaml tr-infertypes tr-xpath) # since v0.18
# RELPIPE_TOOLS+=(out-gui.qt tr-python) # not enabled by default
# Download all sources:
relpipe_download "${RELPIPE_LIBS[@]}" "${RELPIPE_TOOLS[@]}"
# Optional: At this point, we have all dependencies and sources downloaded, so we can disconnect this computer from the internet in order to verify that our build process is sane, deterministic and does not depend on any external resources.
# Build and install libraries:
relpipe_install "${RELPIPE_LIBS[@]}"
# Build and install tools:
relpipe_install "${RELPIPE_TOOLS[@]}"
# Load Bash completion scripts:
for c in "$RELPIPE_SRC"/relpipe-*/bash-completion.sh ; do . "$c"; done
# Enable streamlet examples:
export RELPIPE_IN_FILESYSTEM_STREAMLET_PATH="$RELPIPE_SRC"/relpipe-in-filesystem.cpp/streamlet-examples/
# Clean-up:
unset -f relpipe_install
unset -f relpipe_download
unset -v RELPIPE_VERSION
unset -v RELPIPE_SRC
unset -v RELPIPE_BUILD
unset -v RELPIPE_INSTALL
# List mice, keyboards and some fstab entries:
(relpipe-in-fstab ; relpipe-in-x11 2>/dev/null) \
| relpipe-tr-xpath \
--relation 'fstab' --where 'pass > 1' \
--relation 'x11_input_device' --where 'type = "keyboard" or type = "mouse"' \
| relpipe-out-tabular
Download: examples/release-v0.18.sh
Relational pipes are modular thus you can download and install only parts you need (the libraries are needed always).
Tools out-gui.qt and tr-python require additional libraries and are not built by default.
Relational pipes, open standard and free software © 2018-2022 GlobalCode