Release v0.17
- Relational pipes
- Principles
- Roadmap
- FAQ
- Specification
- Implementation
- Examples
- License
- Screenshots
- Download
- Support & contact
We are pleased to introduce you the new development version of Relational pipes. This release improves and standardizes the CLI interface of several tools, brings a new output module + several smaller improvements.
Note: there is also v0.17.1 (the only difference are two fixes in relpipe-out-gui).
-
The
relpipe-in-climodule has changed CLI interface: see details below -
The
relpipe-in-csvmodule has changed CLI interface: see details below -
The
relpipe-out-csvmodule has new CLI interface: see details below -
The
relpipe-out-tabularmodule has new CLI interface: see details below -
The
relpipe-tr-guilewas renamed torelpipe-tr-scheme: see details below -
New tool
relpipe-out-jackfor generating MIDI through JACK: see details below
See the examples and screenshots pages for details.
Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. Any suggestions, ideas and bug reports are welcome in our mail box.
New CLI interfaces
Several earlier tools have non-standard CLI interface and does not conform with principle „Use --long-options“.
Posistional parameters might be shorter but they are also cryptic and less flexible.
In this release the relpipe-in-cli, relpipe-in-csv tools dropped their old CLI interface
and got a new one based on named parameters and long options.
The relpipe-out-csv and relpipe-out-tabular had no parameters in previous versions and now have standard CLI interface.
In relpipe-out-tabular we can now optionally omit several parts:
--write-record-count false,
--write-types false,
--write-relation-name false
and thus simplify the output if we do not need given information.
╭───────┬────────╮ │ first │ second │ ├───────┼────────┤ │ 1 │ abc │ │ 2 │ def │ ╰───────┴────────╯
These options may be set globally or for particular relation using the --relation option with pattern parameter.
Thus we may e.g. print some relations with data types and some without them.
Remaining tools that will be transformed in the same way: relpipe-tr-grep, relpipe-tr-cut, relpipe-tr-sed
and will get standard CLI interface.
JACK (MIDI) output module
A powerful audio system called JACK allows us to build pipelines consisting of audio interfaces, players, recorders, filters and effects… and route sound streams (both PCM and MIDI) through them. MIDI messages can come from keyboards or other hardware MIDI controllers or from MIDI players and other software.
In the v0.16 release, the relpipe-in-jack was introduced.
This tools allowed capturing the MIDI events from JACK and sending them to the relational pipeline.
Now, in v0.17, we introduce the counterpart tool relpipe-out-jack that reads relational data
and sends SysEx messages, MIDI notes and other events to the JACK pipeline.
We can join the JACK graph with relpipe-out-jack command.
It does not produce anything on STDOUT, it sends MIDI events to JACK instead, so no other output data are needed.
Besides capturing and playing the MIDI, we can use the relpipe-in-jack and relpipe-out-jack
to tak a snapshot of the JACK graph (connections between ports) and recreate it later.
Or establish any new connections or break existing ones.
Either through CLI commands of relpipe-in-jack or through relational data (list of connection and disconnection commands)
sent to relpipe-out-jack.
Both commands have also the --required-connections and --connect-to parameters,
so they can react on reaching particular connection count or automatically connect their port to another one specified on the CLI.
The bash-completion suggests available port names.
More details in the examples: Working with JACK ports and Generating and sending MIDI messages.
Feature overview
Data types
- boolean ~ since v0.8
- variable-length signed integer (SLEB128) ~ since v0.15
- string in UTF-8 ~ since v0.8
Inputs
- Recfile ~ since v0.11
- XML ~ since v0.9
- XMLTable ~ since v0.13
- CSV ~ since v0.9
- file system ~ since v0.9
- CLI ~ since v0.8
- fstab ~ since v0.8
- SQL script ~ since v0.14
- JACK ~ since v0.16
Transformations
- sql: filtering and transformations using the SQL language ~ since v0.13
- awk: filtering and transformations using the classic AWK tool and language ~ since v0.12
- scheme: filtering and transformations defined in the Scheme language using GNU Guile ~ since v0.10
- grep: regular expression filter, removes unwanted records from the relation ~ since v0.8
- cut: regular expression attribute cutter (removes or duplicates attributes and can also DROP whole relation) ~ since v0.8
- sed: regular expression replacer ~ since v0.8
- validator: just a pass-through filter that crashes on invalid data ~ since v0.8
- python: highly experimental ~ since v0.8
Streamlets
- xpath (example, unstable) ~ since v0.15
- hash (example, unstable) ~ since v0.15
- jar_info (example, unstable) ~ since v0.15
- mime_type (example, unstable) ~ since v0.15
- exiftool (example, unstable) ~ since v0.15
- pid (example, unstable) ~ since v0.15
- cloc (example, unstable) ~ since v0.15
- exiv2 (example, unstable) ~ since v0.15
- inode (example, unstable) ~ since v0.15
- lines_count (example, unstable) ~ since v0.15
- pdftotext (example, unstable) ~ since v0.15
- pdfinfo (example, unstable) ~ since v0.15
- tesseract (example, unstable) ~ since v0.15
Outputs
- ASN.1 BER ~ since v0.11
- Recfile ~ since v0.11
- CSV ~ since v0.9
- tabular ~ since v0.8
- XML ~ since v0.8
- nullbyte ~ since v0.8
- GUI in Qt ~ since v0.8
- ODS (LibreOffice) ~ since v0.8
- JACK
New examples
Backward incompatible changes
The relpipe-tr-guile has been renamed to relpipe-tr-scheme, so it is necessary to modify existing scripts.
However the CLI interface has not been changed, thus it can be simply updated e.g. by:
sed s/relpipe-tr-guile/relpipe-tr-scheme/g -i script.sh
The reason behind this rename is that Scheme is a programming language and Guile is one of implementations of this language.
And we prefer depending on interfaces rather than on particular implementations.
Similar situation is relpipe-tr-sql: this module is also named after the language (SQL), not after the default implementation (SQLite)
and this module can use any SQL engine.
The CLI interface of the relpipe-in-cli tool has been standardized and made consistent with other tools.
Now, it uses named parameters instead of cryptic positional ones.
The format of the STDIN stream remains unchanged (values separated by \0 byte).
# Old syntax before v0.17:
relpipe-in-cli generate "relation_from_cli" 3 \
"a" "integer" \
"b" "string" \
"c" "boolean" \
"1" "Hello" "true" \
"2" "World!" "false"
# New syntax:
relpipe-in-cli \
--relation "relation_from_cli" \
--attribute "a" integer \
--attribute "b" string \
--attribute "c" boolean \
--record "1" "Hello" "true" \
--record "2" "World!" "false"
We can repeat these parameters and generate several relations (that was not possibel in the older version).
One relation may be filled with values from STDIN: --records-on-stdin true (replaces the old relpipe-in-cli generate-from-stdin syntax).
And the last relation may be filled with values from remaining CLI arguments: --records "1" "Hello" "true" "2" "World!" "false".
The demo (relpipe-in-cli demo) has been removed (sample data can be easily generated by setting the CLI arguments).
The CLI interface of the relpipe-in-csv tool has also been standardized and made consistent with other tools.
If we have attribute names on the first line of the CSV file (which is quite common), the usage is same as in previous versions:
cat file.csv | relpipe-in-csv
If we have only data in the CSV file (no headers) and thus want to specify them on CLI, the syntax is different:
# Old syntax before v0.17:
cat file.csv | relpipe-in-csv "my_relation" "a" integer "b" boolean
# New syntax:
cat file.csv | relpipe-in-csv \
--relation "my_relation" \
--attribute "a" integer \
--attribute "b" boolean
Bash-completion scripts for all tools are provided.
Future versions should allow optionally storing the attribute data types in the CSV header and parsing them back (together with the attribute names).
Installation
Instalation was tested on Debian GNU/Linux 10.2. The process should be similar on other distributions.
# Install dependencies as root:
apt install g++ make cmake mercurial pkg-config
apt install libxerces-c-dev # needed only for relpipe-in-xml module
apt install guile-2.2-dev # needed only for relpipe-tr-scheme module
apt install gawk # needed only for relpipe-tr-awk module
apt install libxml++2.6-dev # needed only for relpipe-in-xmltable module
apt install unixodbc-dev # needed only for relpipe-tr-sql module
apt install libsqliteodbc # needed only for relpipe-tr-sql module if used with SQLite
apt install odbc-postgresql # needed only for relpipe-tr-sql module if used with PostgreSQL
apt install libjack-jackd2-dev # needed only for relpipe-in-jack and relpipe-out-jack modules; Jack1 should also work
# Run rest of installation as a non-root user:
export RELPIPE_VERSION="v0.17"
export RELPIPE_SRC=~/src
export RELPIPE_BUILD=~/build
export RELPIPE_INSTALL=~/install
export PKG_CONFIG_PATH="$RELPIPE_INSTALL/lib/pkgconfig/:$PKG_CONFIG_PATH"
export PATH="$RELPIPE_INSTALL/bin:$PATH"
rm -rf "$RELPIPE_BUILD"/relpipe-*
mkdir -p "$RELPIPE_SRC" "$RELPIPE_BUILD" "$RELPIPE_INSTALL"
# Helper functions:
relpipe_download() { for m in "$@"; do cd "$RELPIPE_SRC" && ([[ -d "relpipe-$m.cpp" ]] && hg pull -R "relpipe-$m.cpp" && hg update -R "relpipe-$m.cpp" "$RELPIPE_VERSION" || hg clone -u "$RELPIPE_VERSION" https://hg.globalcode.info/relpipe/relpipe-$m.cpp) || break; done; }
relpipe_install() { for m in "$@"; do cd "$RELPIPE_BUILD" && mkdir -p relpipe-$m.cpp && cd relpipe-$m.cpp && cmake -DCMAKE_INSTALL_PREFIX:PATH="$RELPIPE_INSTALL" "$RELPIPE_SRC/relpipe-$m.cpp" && make && make install || break; done; }
# Download all sources:
relpipe_download lib-common lib-reader lib-writer lib-cli lib-xmlwriter in-cli in-fstab in-xml in-xmltable in-csv in-filesystem in-recfile out-gui.qt out-nullbyte out-ods out-tabular out-xml out-csv out-asn1 out-recfile tr-cut tr-grep tr-python tr-sed tr-validator tr-scheme tr-awk tr-sql in-jack out-jack
# Optional: At this point, we have all dependencies and sources downloaded, so we can disconnect this computer from the internet in order to verify that our build process is sane, deterministic and does not depend on any external resources.
# Build and install libraries:
relpipe_install lib-common lib-reader lib-writer lib-cli lib-xmlwriter
# Build and install tools:
relpipe_install in-fstab in-cli in-fstab in-xml in-xmltable in-csv in-recfile tr-cut tr-grep tr-sed tr-scheme tr-awk tr-sql out-nullbyte out-ods out-tabular out-xml out-csv out-asn1 out-recfile in-filesystem in-jack out-jack
# Load Bash completion scripts:
for c in "$RELPIPE_SRC"/relpipe-*/bash-completion.sh ; do . "$c"; done
# Enable streamlet examples:
export RELPIPE_IN_FILESYSTEM_STREAMLET_PATH="$RELPIPE_SRC"/relpipe-in-filesystem.cpp/streamlet-examples/
# Clean-up:
unset -f relpipe_install
unset -f relpipe_download
unset -v RELPIPE_VERSION
unset -v RELPIPE_SRC
unset -v RELPIPE_BUILD
unset -v RELPIPE_INSTALL
# List available JACK ports (the JACK daemon should be running):
relpipe-in-jack --list-ports true --list-midi-messages false | relpipe-out-tabular
Download: examples/release-v0.17.sh
Relational pipes are modular thus you can download and install only parts you need (the libraries are needed always).
Tools out-gui.qt and tr-python require additional libraries and are not built by default.
Relational pipes, open standard and free software © 2018-2022 GlobalCode