Specification
- Relational pipes
- Principles
- Roadmap
- FAQ
- Specification
- Implementation
- Examples
- License
- Screenshots
- Download
- Support & contact
Currently only fragments of the specification are published and the incompatible changes might (and will) come before the v1.0.0 release. Please stay tuned for this stable version which will deliver specification such complete and precise that independent implementation of the format will be possible.
Relational pipes data format
Stream structure
Relational pipes data is a stream / sequence of data parts. Type of a part is designated by its first octet. Parts have variable length and specific internal structure (see grammar below).
| id | name | description |
| 0x1D | relation | start of the relation |
| 0x1E | record | start of the record |
| 0x2B | magic | optional magic number |
Relational pipes binary stream grammar in a pseudo-notation:
magic =
"2b",
"0601040182c65d0804";
(* 2b0601040182c65d0804 i.e. OID 1.3.6.1.4.1.41821.8.4 in ASN.1/BER *)
(* since v0.19 *)
type-id =
"01" | (* boolean *)
"02" | (* integer *)
"03" ; (* string *)
boolean = "00" | "01"; (* false, true *)
integer = ?LEB128?; (* signed variable-length integer *)
string-length = integer;
string = string-length, ?UTF-8?; (* string-length = UTF-8 octet count *)
relation-name = string;
attribute-count = integer;
attribute-value = boolean | integer | string; (* matching attribute-type *)
attribute-name = string;
attribute-type = type-id;
attribute-definition = attribute-name, attribute-type;
relation-header =
"1d",
relation-name,
attribute-count, (* at least one *)
{ attribute-definition }; (* n = attribute-count *)
record =
"1e",
{ attribute-value }; (* n = attribute-count *)
relation =
relation-header, { record };
relational-stream =
{ relation | magic };
n.b. future implementation of null values and sparse data will slightly change this grammar
Data types
Currently, there are only three data types.
| typeId | code | description |
| 0x01 | boolean | logical value, true/false |
| 0x02 | integer | signed integer number of arbitrary length |
| 0x03 | string | character string in UTF-8 |
Libraries
relpipe-lib-writer
Wraps an output stream (usually STDOUT), accepts method calls (relations, attributes) and generates Relational pipes data on the stream.
relpipe-lib-reader
Wraps an input stream (usually STDIN). The caller creates and sets handlers (zero or more) using addHandler() and then calls process() method.
During this method call, the reader reads the input and calls the handlers.
Handlers receive relations and attributes.
Tools
relpipe-in-cli
A tool that generates a single relation. If we want more relations in a single stream, we just call this command multiple times:
(relpipe-in-cli ... ; relpipe-in-cli ... ; relpipe-in-cli ... ) | relpipe-out-tabular
Or concatenate several files or do a combination of both files and commands.
This command accept these arguments:
- relation name
- attribute count
- names of attributes
- types of attributes
- attribute values
These data might be passed as CLI arguments on the command line or as null-byte (\0) separated list of values on STDIN.
Both ways can be combined e.g. pass relation name and metadata as CLI arguments and the data on STDIN.
The tool simply starts with CLI arguments (if any) and continues with values from STDIN (if any).
This tool is a good entry point to the Relational pipes world because it requires no programming and construction of the argument list or \0 separated list can be done in any language or environment.
Tools like perl or tr can convert almost any data to this form and pass it to relpipe-in-cli.
relpipe-in-csv
A tool that parses a CSV (comma-separated values) input and generates a single relation from it. Values might be "quoted". If quoted value contains a quote literal, it is escaped by doubling. Encoding must be UTF-8. Line ends might be LF or CRLF.
relpipe-in-fstab
A tool that parses fstab (or mtab) file containing a list of devices, mount-points and their options and generates a single relation from it.
If executed on TTY, it reads data from the default location: /etc/fstab.
If executed with STDIN attached to a file or another command (in a pipe), it reads data from this stream.
relpipe-in-xml
A tool that reads XML data in the format generated by relpipe-out-xml
and converts it back to the relational format.
Can be used together with XSLT procesor XQuery engine or other XML generator in two basic scenarios:
- as an input filter: to convert other formats to relational data
- as a transformation: in pipeline:
relpipe-out-xml | some-xml-processor | relpipe-in-xml
XQuery is very powerful language that can do various transformations, filtering and even JOIN and UNION operations. So its power is comparable to relational databases. In future Relational pipes releases, there will be also an SQL transformation tool, where these operations will be defined in classic SQL syntax.
relpipe-in-filesystem
A tool that reads \0 separated list of file paths and generates relation
with metadata of given files.
Can read basic metadata like file path, name, size, owner…
and also extended attributes (xattr).
relpipe-out-tabular
A tool that formats relational data as tables with unicode borders and ANSI colors. Good for viewing relational data in a terminal or redirecting such view to a file or clipboard (to be pasted anywhere, where fixed-width font is used).
relpipe-out-gui
A tool that views relational data in a GUI window. Relations are displayed as panels with tables. Particular implementation might offer also additional features like chart drawing or simple statistics (sum, max, min, avg, percentiles etc.)
relpipe-out-nullbyte
A tool that converts relational data to a list of null-byte (\0) separated values.
Makes sense only for a single relation because boundaries between relations will be lost if there are more of them.
Also attribute names and types are lost by default.
Is suitable for passing a single relation to xargs --null --max-args=X (where X is the attribute count)
or other command that accepts values separated by a null-byte.
relpipe-out-xml
A tool that converts relational data to its XML representation. Is useful for further processing in other tools (e.g. XSLT procesor that generates XHTML org other XML-based format) or for storage of relational data in a text form (good for version control systems, diff, manual editation or review).
relpipe-out-ods
A tool that converts relational data to its ODS (OpenDocument) representation resp. its flat variant (a single XML file instead of a ZIP containing many files). This OpenDocument output can be opened in tools like LibreOffice and further processed (add charts, calculations etc.).
relpipe-out-csv
A tool that convers relational data into the CSV format (comma-separated values). Makes sense only for a single relation because boundaries between relations will be lost if there are more of them.
relpipe-tr-validator
A tool that behaves like cat, dd or pv commands i.e. reads data from STDIN and outputs the same data on STDOUT.
But compared to these tools, data are parsed and then converted back to the Relational pipes format.
Thus if the input was not in this format, the process fails (exit code != 0).
Errors (if any) are reported on STDERR.
relpipe-tr-sed
A tool that modifies attribute values according to given regular expression and replacement string.
Works with given relations and attributes only (others stay untouched).
Relation and attribute name are also specified as a regular expression.
Thus single relpipe-tr-sed run may modify multiple relations or attributes.
relpipe-tr-grep
A tool that drops records (does restriction) according to their match to given regular expression.
Relation and attribute name are also specified as a regular expression.
Thus single relpipe-tr-grep run may modify multiple relations or do match on multiple attributes.
relpipe-tr-cut
A tool that drops or multiplies attributes or changes their order (does projection) according to their name match to given regular expression.
Relation name is also specified as a regular expression and there might be multiple regexes specifying desired attributes.
Thus single relpipe-tr-cut run may modify multiple relations or pick multiple attributes from them.
Relational pipes, open standard and free software © 2018-2023 GlobalCode