Release v0.9
- Relational pipes
- Principles
- Roadmap
- FAQ
- Specification
- Implementation
- Examples
- License
- Screenshots
- Download
- Support & contact
We are pleased to introduce you the new version of Relational pipes. It brings several important features:
- XML input and output: offers lossless bidirectional conversion between relational format and text. Can be used for manual modifications or reviews. XML form of relational data can be generated or consumed by XQuery or XSLT processors or other XML tools.
-
CSV input and output:
CSV is a standard format defined in RFC 4180.
It is very simple, lacks data types and can contain only a single relation.
Despite this, it is very useful when integrating Relational pipes with other tools.
At the end of our pipeline, we can put
relpipe-out-csv, feed the relation to e.g. GNU R and compute some statistics or generate some nice graphs. -
Filesystem input: this input filter reads file paths (usually generated by the
findcommand) and fetches metadata of given files. Supports various name/path formats, size, owner, symlinks… and extended attributes (xattr). The output can be customized using CLI arguments. This tool can be used to catalogue disk contents, search files or to collect data for some statistics like: How much space does files in particular formats occupy? or From which domains I download files most often? Other use case is: save the output ofrelpipe-in-filesystemin a file and use it as an index of our files (even on offline media).
See the examples page for details.
Data types
- boolean ~ since v0.8
- variable unsigned integer (prototype) ~ since v0.8
- string in UTF-8 ~ since v0.8
Inputs
- XML
- CSV
- file system
- CLI ~ since v0.8
- fstab ~ since v0.8
Transformations
- grep: regular expression filter, removes unwanted records from the relation ~ since v0.8
- cut: regular expression attribute cutter (removes or duplicates attributes and can also DROP whole relation) ~ since v0.8
- sed: regular expression replacer ~ since v0.8
- validator: just a pass-through filter that crashes on invalid data ~ since v0.8
- python: highly experimental ~ since v0.8
Outputs
- CSV
- tabular ~ since v0.8
- XML ~ since v0.8
- nullbyte ~ since v0.8
- GUI in Qt ~ since v0.8
- ODS (LibreOffice) ~ since v0.8
Instalation was tested on Debian GNU/Linux 9.6. The process should be similar on other distributions.
# Install dependencies as root:
su -c "apt install g++ make cmake mercurial pkg-config"
su -c "apt install libxerces-c-dev" # needed only for relpipe-in-xml module
# Run rest of installation as a non-root user:
export RELPIPE_VERSION="v0.9"
export RELPIPE_SRC=~/src
export RELPIPE_BUILD=~/build
export RELPIPE_INSTALL=~/install
export PKG_CONFIG_PATH="$RELPIPE_INSTALL/lib/pkgconfig/:$PKG_CONFIG_PATH"
export PATH="$RELPIPE_INSTALL/bin:$PATH"
rm -rf "$RELPIPE_BUILD"/relpipe-*
mkdir -p "$RELPIPE_SRC" "$RELPIPE_BUILD" "$RELPIPE_INSTALL"
# Helper functions:
relpipe_download() { for m in "$@"; do cd "$RELPIPE_SRC" && ([[ -d "relpipe-$m.cpp" ]] && hg pull -R "relpipe-$m.cpp" && hg update -R "relpipe-$m.cpp" "$RELPIPE_VERSION" || hg clone -u "$RELPIPE_VERSION" https://hg.globalcode.info/relpipe/relpipe-$m.cpp) || break; done; }
relpipe_install() { for m in "$@"; do cd "$RELPIPE_BUILD" && mkdir -p relpipe-$m.cpp && cd relpipe-$m.cpp && cmake -DCMAKE_INSTALL_PREFIX:PATH="$RELPIPE_INSTALL" "$RELPIPE_SRC/relpipe-$m.cpp" && make && make install || break; done; }
# Download all sources:
relpipe_download lib-protocol lib-reader lib-writer lib-cli lib-xmlwriter in-cli in-fstab in-xml in-csv in-filesystem out-gui.qt out-nullbyte out-ods out-tabular out-xml out-csv tr-cut tr-grep tr-python tr-sed tr-validator
# Build and install libraries:
relpipe_install lib-protocol lib-reader lib-writer lib-cli lib-xmlwriter
# Build and install tools:
relpipe_install in-fstab in-cli in-fstab in-xml in-csv tr-cut tr-grep tr-sed out-nullbyte out-ods out-tabular out-xml out-csv
# relpipe_install in-filesystem # requires GCC 8 or patching (see below)
# Clean-up:
unset -f relpipe_install
unset -f relpipe_download
unset -v RELPIPE_VERSION
unset -v RELPIPE_SRC
unset -v RELPIPE_BUILD
unset -v RELPIPE_INSTALL
# View your fstab like on an 80s green screen terminal!
relpipe-in-fstab | relpipe-out-tabular
Download: examples/release-v0.9.sh
Relational pipes are modular thus you can download and install only parts you need (the libraries are needed always).
Tools out-gui.qt and tr-python require additional libraries and are not built by default.
The module relpipe-in-filesystem uses C++ filesystem API which is supported since GCC 8.
This module can be compiled and seems usable even with GCC 6, but requires some patching (switch to the experimental API):
sed 's@#include <filesystem>@#include <experimental/filesystem>@g' -i "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FileAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/XattrAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FilesystemCommand.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/AttributeFinder.h
sed 's@std::filesystem@std::experimental::filesystem@g' -i "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FileAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/XattrAttributeFinder.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/FilesystemCommand.h "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/AttributeFinder.h
sed 's/.*PROPERTY CXX_STANDARD.*/#\0/g' -i "$RELPIPE_SRC"/relpipe-in-filesystem.cpp/src/CMakeLists.txt
Relational pipes, open standard and free software © 2018-2022 GlobalCode