Using custom version of SQLite (LD_PRELOAD)
- Relational pipes
- Principles
- Roadmap
- FAQ
- Specification
- Implementation
- Examples
- License
- Screenshots
- Download
- Support & contact
One of reasons why we prefer shared libraries (.so) rather than static linking,
is that shared libraries are much more hacker-friendly and allow the user switching to a newer or modified library without recompiling the program.
n.b. This method is obsolete since v0.16 that does not use SQLite library directly
and uses arbitrary database driver (including SQLite one) through an abstraction layer (ODBC).
This article is still valid as an example of the LD_PRELOAD hack and can be used with older versions of Relational pipes.
Since v0.16 we can easily replace whole ODBC driver (and thus use also different version of the SQLite),
there is no need for LD_PRELOAD hacking
– we can just configure desired driver (the .so file) in the INI file or ad-hoc in the connection string.
By default, relpipe-tr-sql links to the SQLite library available in our distribution (e.g. 3.22).
As we can check:
$ ldd $(which relpipe-tr-sql) | grep sqlite
libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f4c73888000)
But what if we want to use some new features like window functions that are available in later (3.25) library versions? Or what if we want to add our own custom modifications to this library? Do we have to recompile the Relational pipes tools?
No, we can just plug the new/modified library in and use it instead of the distribution one.
Download and compile SQLite library
The build process of SQLite should be straightforward:
# Switch to the user for such experiments (optional but recommended):
su - hacker
# Create directories, download and extract sources:
mkdir -p ~/src/sqlite
cd ~/src/sqlite
wget https://www.sqlite.org/2019/sqlite-autoconf-3300100.tar.gz
tar xvzf sqlite-autoconf-3300100.tar.gz
cd sqlite-autoconf-3300100/
# Optional: do some changes to the source code
# Build SQLite:
./configure
make
# Test it:
echo "SELECT 'hello world'" | ./sqlite3
The desired shared libraries are located in the .libs directory.
Use this library in the Relational pipes tools
We have already build/installed relpipe-tr-sql which is linked to the SQLite library available in our distribution.
Then switching to a custom version of the library is very easy – we just need to set an environment variable.
export LD_PRELOAD=~/src/sqlite/sqlite-autoconf-3300100/.libs/libsqlite3.so.0
And then relpipe-tr-sql will use the newer library version as we can check:
$ ldd $(which relpipe-tr-sql) | grep sqlite
/home/hacker/src/sqlite/sqlite-autoconf-3300100/.libs/libsqlite3.so.0 (0x00007f9979578000)
Now we can use new features like window functions:
relpipe-in-fstab \
| relpipe-tr-sql \
--relation "fs_types" "
SELECT
mount_point,
type,
count(*) OVER (PARTITION BY type) AS same_type_count
FROM fstab" \
| relpipe-out-tabular
And get result like this one:
fstab:
╭──────────────────────┬───────────────┬──────────────────────────╮
│ mount_point (string) │ type (string) │ same_type_count (string) │
├──────────────────────┼───────────────┼──────────────────────────┤
│ /home │ btrfs │ 1 │
│ / │ ext4 │ 2 │
│ /mnt/data │ ext4 │ 2 │
│ /media/cdrom0 │ udf,iso9660 │ 1 │
│ /mnt/private │ xfs │ 1 │
╰──────────────────────┴───────────────┴──────────────────────────╯
Record count: 5
That would not be possible with older versions of the SQLite library – as we can check by unsetting the LD_PRELOAD variable:
unset LD_PRELOAD
Which returns us to the previous state where SQLite from our distribution was used. And then calling the same SQL query leads to an error.
The LD_PRELOAD hack can be used with any other software – it is not specific to Relational pipes.
Another example is the Spacenav Hack which bridges/translates two APIs of a library.
n.b. if we do export LD_PRELOAD it will affect all programs started from given shell session
and if we even put it in our .bashrc, it will affect all Bash sessions started later and programs started from them.
Which might not be a desired behavior. So sometimes it is better to set the LD_PRELOAD variable only for a single command, not globally.
This can be done through a custom wrapper script or an alias:
alias relpipe-tr-sql='LD_PRELOAD=~/src/sqlite/sqlite-autoconf-3300100/.libs/libsqlite3.so.0 relpipe-tr-sql'
We can safely put this this line into our .bashrc without affecting any other software.
ABI compatibility
The prerequisite for such easy library swapping is the compatibility of the ABI (application binary interface). It means that we can change the library internals (the SQL language features in this case) but we must retain the compiled representation of the C functions compatible so the both parts (the library and the program) will still fit together. We can not e.g. remove a C function. And we should also not do any incompatible changes on the semantic level (although it could still link together, it would lead to unwanted results).
In case of libraries that follow the Semantic versioning (as required by Sane software manifesto) for their ABI, it is easy to say which versions are compatible and which would require recompiling the program or even changing its source code. If the patch or minor version numbers were changed, the libraries could be swapped this way. If the major version was changed, it would be probably necessary to also modify the software that uses this library.
Relational pipes, open standard and free software © 2018-2022 GlobalCode