Release v0.11
- Relational pipes
- Principles
- Roadmap
- FAQ
- Specification
- Implementation
- Examples
- License
- Screenshots
- Download
- Support & contact
We are pleased to introduce you the new development version of Relational pipes. This release brings Recfile input and output and ASN.1 BER output:
- Recfile format support: Recfile is the native format of GNU Recutils. Recfiles are text files that contain records of various types. They are human-editable and serve as simple databases. Relational pipes now contain tools for converting relational data streams to and from the recfile format. This format is good for editing by hand or for storing in some version control system. GNU Recutils and Relational pipes can be used together for data processing or e.g. for data-driven scripting.
- ASN.1 BER output: Abstract Syntax Notation One (ASN.1) is traditional and well established language for describing interfaces and data structures in particular. This format/language is widespread and actively used in telecommunications, cryptography and other industries. BER is one of ASN.1 encodings. This release of Relational pipes bring basic support for output in the BER format. Now it can serve as a preview (such data can be opened in dumpasn1 or wireshark) or a template (the source code can be modified to generate BER data in some particular format) and help with interaction with ASN.1 native speakers.
See the examples and screenshots pages for details.
Please note that this is still a development release and thus the API (libraries, CLI arguments, formats) might and will change. Any suggestions, ideas and bug reports are welcome in our mail box.
Data types
- boolean ~ since v0.8
- variable unsigned integer (prototype) ~ since v0.8
- string in UTF-8 ~ since v0.8
Inputs
- Recfile
- XML ~ since v0.9
- CSV ~ since v0.9
- file system ~ since v0.9
- CLI ~ since v0.8
- fstab ~ since v0.8
Transformations
- guile: filtering and transformations defined in the Scheme language using GNU Guile ~ since v0.10
- grep: regular expression filter, removes unwanted records from the relation ~ since v0.8
- cut: regular expression attribute cutter (removes or duplicates attributes and can also DROP whole relation) ~ since v0.8
- sed: regular expression replacer ~ since v0.8
- validator: just a pass-through filter that crashes on invalid data ~ since v0.8
- python: highly experimental ~ since v0.8
Outputs
- ASN.1 BER
- Recfile
- CSV ~ since v0.9
- tabular ~ since v0.8
- XML ~ since v0.8
- nullbyte ~ since v0.8
- GUI in Qt ~ since v0.8
- ODS (LibreOffice) ~ since v0.8
Instalation was tested on Debian GNU/Linux 9.6. The process should be similar on other distributions.
# Install dependencies as root:
su -c "apt install g++ make cmake mercurial pkg-config"
su -c "apt install libxerces-c-dev" # needed only for relpipe-in-xml module
su -c "apt install guile-2.2-dev" # needed only for relpipe-tr-guile module; guile-2.0-dev also works but requires a patch (see below)
# Run rest of installation as a non-root user:
export RELPIPE_VERSION="v0.11"
export RELPIPE_SRC=~/src
export RELPIPE_BUILD=~/build
export RELPIPE_INSTALL=~/install
export PKG_CONFIG_PATH="$RELPIPE_INSTALL/lib/pkgconfig/:$PKG_CONFIG_PATH"
export PATH="$RELPIPE_INSTALL/bin:$PATH"
rm -rf "$RELPIPE_BUILD"/relpipe-*
mkdir -p "$RELPIPE_SRC" "$RELPIPE_BUILD" "$RELPIPE_INSTALL"
# Helper functions:
relpipe_download() { for m in "$@"; do cd "$RELPIPE_SRC" && ([[ -d "relpipe-$m.cpp" ]] && hg pull -R "relpipe-$m.cpp" && hg update -R "relpipe-$m.cpp" "$RELPIPE_VERSION" || hg clone -u "$RELPIPE_VERSION" https://hg.globalcode.info/relpipe/relpipe-$m.cpp) || break; done; }
relpipe_install() { for m in "$@"; do cd "$RELPIPE_BUILD" && mkdir -p relpipe-$m.cpp && cd relpipe-$m.cpp && cmake -DCMAKE_INSTALL_PREFIX:PATH="$RELPIPE_INSTALL" "$RELPIPE_SRC/relpipe-$m.cpp" && make && make install || break; done; }
# Download all sources:
relpipe_download lib-protocol lib-reader lib-writer lib-cli lib-xmlwriter in-cli in-fstab in-xml in-csv in-filesystem in-recfile out-gui.qt out-nullbyte out-ods out-tabular out-xml out-csv out-asn1 out-recfile tr-cut tr-grep tr-python tr-sed tr-validator tr-guile
# Optional: At this point, we have all dependencies and sources downloaded, so we can disconnect this computer from the internet in order to verify that our build process is sane, deterministic and does not depend on any external resources.
# Build and install libraries:
relpipe_install lib-protocol lib-reader lib-writer lib-cli lib-xmlwriter
# Build and install tools:
relpipe_install in-fstab in-cli in-fstab in-xml in-csv in-recfile tr-cut tr-grep tr-sed tr-guile out-nullbyte out-ods out-tabular out-xml out-csv out-asn1 out-recfile
# relpipe_install in-filesystem # requires GCC 8 or patching (see below)
# Clean-up:
unset -f relpipe_install
unset -f relpipe_download
unset -v RELPIPE_VERSION
unset -v RELPIPE_SRC
unset -v RELPIPE_BUILD
unset -v RELPIPE_INSTALL
# Filter your your fstab using Guile Scheme and view it like on an 80s green screen terminal!
relpipe-in-fstab | relpipe-tr-guile --relation 'fstab' --where '(or (= $pass 1) (string= $type "swap") )' | relpipe-out-tabular
Download: examples/release-v0.11.sh
Relational pipes are modular thus you can download and install only parts you need (the libraries are needed always).
Tools out-gui.qt and tr-python require additional libraries and are not built by default.
The module relpipe-in-filesystem uses C++ filesystem API which is supported since GCC 8.
This module can be compiled and seems usable even with GCC 6, but requires some patching (switch to the experimental API):
sed 's@#include <filesystem>@#include <experimental/filesystem>@g' -i "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FileAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/XattrAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FilesystemCommand.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/AttributeFinder.h
sed 's@std::filesystem@std::experimental::filesystem@g' -i "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FileAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/XattrAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FilesystemCommand.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/AttributeFinder.h
sed 's/.*PROPERTY CXX_STANDARD.*/#\0/g' -i "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/CMakeLists.txt
The module relpipe-tr-guile uses GNU Guile 2.2 but can also work with 2.0.
In such case, it requires this patch:
sed 's/guile-2\.2/guile-2.0/g' -i "$RELPIPE_SRC"/relpipe-tr-guile.cpp/src/CMakeLists.txt
Relational pipes, open standard and free software © 2018-2022 GlobalCode