dbhub.io development setup

Here is how to setup a development environment for dbhub.io.

dbhub.io project on Github

First I tried Docker which was a PITA this time, see issue 211, but ended up with a manual installation on a fresh Alpine Linux image, based von 3.18 edge.

Built and tested with dbhub.io master branch up to commit 5c9e1ab.

Following steps in order to run on the root shell. (initial ALpine Linux login is root without password)

Setting up Alpine Linux and installing packages:

### Used ISO: alpine-extended-3.8.0-x86_64.iso
### https://dl-cdn.alpinelinux.org/alpine/edge/releases/x86_64/alpine-extended-3.8.0-x86_64.iso
setup-alpine
setup-apkrepos
apk -U upgrade
apk add nano
nano /etc/apk/repositories
### Uncomment community. Mirrors depending on what you choose in setup-apkrepos
https://ftp.halifax.rwth-aachen.de/alpine/edge/main
http://ftp.halifax.rwth-aachen.de/alpine/edge/main
http://ftp.halifax.rwth-aachen.de/alpine/edge/community
###
apk update
apk add ca-certificates curl file git go libc-dev make memcached minio openrc openssl openssl-dev postgresql yarn
apk add rabbitmq-server --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing

Add base services to default run level:

rc-update add memcached default
rc-update add minio default
rc-update add postgresql default
rc-update add rabbitmq-server default
mkdir /etc/rabbitmq
rabbitmq-plugins enable rabbitmq_management
rabbitmq-plugins enable rabbitmq_top

Add dbhub user, create required folders and set permissions:

addgroup dbhub
adduser -D -S -s /bin/ash -G dbhub dbhub
mkdir -p /var/log/dbhub /home/dbhub/.dbhub/disk_cache
chown -R dbhub:dbhub -p /var/log/dbhub /home/dbhub/.dbhub/disk_cache
chown -R dbhub:dbhub /var/log/dbhub /home/dbhub/.dbhub/disk_cache
chmod -R 700 /var/log/dbhub /home/dbhub/.dbhub/disk_cache

Edit minio service config and change password if desired: (optional)

nano /etc/conf.d/minio
### Set minio password
MINIO_ROOT_USER="minio"
MINIO_ROOT_PASSWORD="minio123"
###

Initialize postgresql and rabbitmq servers:
SomeDefaultRandomString01 can be changed as desired, has a minimum length tough.

su - postgres -c 'mkdir -p /var/lib/postgresql/15/data'
su - postgres -c 'chmod 0700 /var/lib/postgresql/15/data'
su - postgres -c 'initdb -D /var/lib/postgresql/15/data'
su - postgres -c 'mkdir /run/postresql'
mkdir /run/postgresql
chown postgres:postgres /run/postgresql
su - postgres -c 'pg_ctl start -D /var/lib/postgresql/15/data'
mkdir -p /var/log/postgresql
chown -R postgres:postgres /var/log/postgresql
echo 'SomeDefaultRandomString01' > /var/lib/rabbitmq/.erlang.cookie
chmod 400 /var/lib/rabbitmq/.erlang.cookie
chown -R rabbitmq:rabbitmq /var/lib/rabbitmq/
chown -R rabbitmq:rabbitmq /var/log/rabbitmq/
chown -R minio:minio /var/lib/minio/
createuser -U postgres -d dbhub
createdb -U postgres -O dbhub dbhub

Add required variables to system environment:

nano /etc/profile
### Add env variables
export DBHUB_SOURCE=/dbhub.io
export GOBIN=/usr/local/bin
export CONFIG_FILE=/home/dbhub/.dbhub/config.toml
###

Create sqlite build script, set executable and run:

nano build_sqlite.sh
### Add
echo "Downloading SQLite source code" && \
mkdir /sqlite && \
cd /sqlite && \
TARBALL=$(curl -s https://sqlite.org/download.html | awk '/<!--/,/-->/ {print}' | grep 'sqlite-autoconf' | cut -d ',' -f 3) && \
SHA3=$(curl -s https://sqlite.org/download.html | awk '/<!--/,/-->/ {print}' | grep 'sqlite-autoconf' | cut -d ',' -f 5) && \
curl -LsS -o sqlite.tar.gz https://sqlite.org/${TARBALL} && \
VERIFY=$(openssl dgst -sha3-256 sqlite.tar.gz | cut -d ' ' -f 2) && \
if [ "$SHA3" != "$VERIFY" ]; then exit 1 ; fi && \
if [ ! -f sqlite.tar.gz ]; then echo "Downloading the SQLite source code did not work" ; exit 3 ; fi && \
echo "Compiling local SQLite" && \
tar xfz sqlite.tar.gz && \
cd sqlite-autoconf-* || exit 4 && \
CPPFLAGS="-DSQLITE_ENABLE_COLUMN_METADATA=1 -DSQLITE_MAX_VARIABLE_NUMBER=250000 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_GEOPOLY=1 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS=1 -DSQLITE_ENABLE_FTS5=1 -DSQLITE_ENABLE_STAT4=1 -DSQLITE_ENABLE_JSON1=1 -DSQLITE_SOUNDEX=1 -DSQLITE_ENABLE_MATH_FUNCTIONS=1 -DSQLITE_MAX_ATTACHED=125 -DSQLITE_ENABLE_MEMORY_MANAGEMENT=1 -DSQLITE_ENABLE_SNAPSHOT=1" ./configure --prefix=/sqlite --enable-dynamic-extensions=no && \
make -j2 && \
make install && \
cd .. && \
rm -rf sqlite-autoconf-* && \
echo "/sqlite/lib:/lib:/usr/local/lib:/usr/lib" > /etc/ld-musl-x86_64.path
###
chmod +x build_dbhub.sh
ash build_sqlite.sh

Create dbhub build script, set executable and run:

nano build_dbhub.sh
### Add
cd /
git clone --branch master --depth 5 https://github.com/sqlitebrowser/dbhub.io
export DBHUB_SOURCE=/dbhub.io
cd ${DBHUB_SOURCE}
yarn
yarn run babel ${DBHUB_SOURCE}/webui/jsx --out-dir ${DBHUB_SOURCE}/webui/js --presets babel-preset-react-app/prod
yarn run webpack -c ${DBHUB_SOURCE}/webui/webpack.config.js
cd ${DBHUB_SOURCE}/api
PKG_CONFIG_PATH=/sqlite/lib/pkgconfig go build -gcflags "all=-N -l" -buildvcs=false -o /usr/local/bin/dbhub-api .
cd ${DBHUB_SOURCE}/db4s
PKG_CONFIG_PATH=/sqlite/lib/pkgconfig go build -gcflags "all=-N -l" -buildvcs=false -o /usr/local/bin/dbhub-db4s .
cd ${DBHUB_SOURCE}/live
PKG_CONFIG_PATH=/sqlite/lib/pkgconfig go build -gcflags "all=-N -l" -buildvcs=false -o /usr/local/bin/dbhub-live .
cd ${DBHUB_SOURCE}/standalone/analysis
PKG_CONFIG_PATH=/sqlite/lib/pkgconfig go build -gcflags "all=-N -l" -buildvcs=false -o /usr/local/bin/dbhub-analysis .
ln -f -s /usr/local/bin/dbhub-analysis  /etc/periodic/15min/
cd ${DBHUB_SOURCE}/webui
PKG_CONFIG_PATH=/sqlite/lib/pkgconfig go build -gcflags "all=-N -l" -buildvcs=false -o /usr/local/bin/dbhub-webui .
###
chmod +x build_dbhub.sh
ash build_dbhub.sh

Install GO debugger:

GOBIN=/usr/local/bin go install github.com/go-delve/delve/cmd/dlv@latest

Create script for easy remote debugging:

nano debug.sh
### Add
# Kill the existing running daemons
rc-service dbhub-webui stop
rc-service dbhub-api stop
rc-service dbhub-db4s stop
rc-service dbhub-live stop
pkill dlv

if [ -c /dev/console ]; then
  chmod o+w /dev/console
fi

su - dbhub -c 'nohup dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec /usr/local/bin/dbhub-webui >>/var/log/dbhub/output.log 2>&1 &'
su - dbhub -c 'nohup dlv --listen=:2346 --headless=true --api-version=2 --accept-multiclient exec /usr/local/bin/dbhub-api >>/var/log/dbhub/output.log 2>&1 &'
su - dbhub -c 'nohup dlv --listen=:2347 --headless=true --api-version=2 --accept-multiclient exec /usr/local/bin/dbhub-db4s >>/var/log/dbhub/output.log 2>&1 &'
su - dbhub -c 'nohup dlv --listen=:2348 --headless=true --api-version=2 --accept-multiclient exec /usr/local/bin/dbhub-live node1 /tmp/node1>>/var/log/dbhub/output.log 2>&1 &'
# Delay long enough for the DBHub.io daemons to start
sleep 1

See example launch.json for VSC below. How to setup remote Go debugging in VSC see:

Go in Visual Studio Code
Remote debugging

Copy dbhub configuration into users folder where web application is looking for:

cp /dbhub.io/docker/config.toml /home/dbhub/.dbhub/config.toml
chown dbhub:dbhub /home/dbhub/.dbhub/config.toml

Edit minio password in web application config: (optional, must match to the one set above)

nano /home/dbhub/.dbhub/config.toml
### Change minio secret to match service config
[minio]
secret = "minio123"
###

Add non-root user and doas (sudo alternative) package (optional):

### Add a non-root user
adduser dev
adduser dev wheel
apk add doas
nano /etc/doas.d/doas.conf
### Add group
permit persist :wheel
###

Reboot. Services memcached, minio, rabbitmq-server and postgresql should start:

reboot

Create openrc init scripts to run all 4 dbhub services:

nano /etc/init.d/dbhub-webui
### Add
#!/sbin/openrc-run

supervisor=supervise-daemon
name="dbhub-webui"
description="DBHub WebUI"
command="/usr/local/bin/dbhub-webui"
command_user="dbhub"
command_background="true"
output_log="/var/log/dbhub/output.log"
error_log="/var/log/dbhub/output.log"
pidfile="/var/run/dbhub-webui.pid"

depend() {
        need net
        after memcached minio rabbitmq-server postgresql
}

start_pre()
{
    checkpath --directory --owner dbhub:dbhub --mode 0700 /var/log/dbhub
}

stop() {
        ebegin "Stopping dbhub-webui"
        start-stop-daemon --stop --quiet --pidfile=$pidfile --exec "timeout 1m pkill dbhub-webui"
        eend $?
}
###

nano /etc/init.d/dbhub-api
### Add
#!/sbin/openrc-run

supervisor=supervise-daemon
name="dbhub-api"
description="DBHub API"
command="/usr/local/bin/dbhub-api"
command_user="dbhub"
command_background="true"
output_log="/var/log/dbhub/output.log"
error_log="/var/log/dbhub/output.log"
pidfile="/var/run/dbhub-api.pid"

depend() {
        need net
        after memcached minio rabbitmq-server postgresql dbhub-webui
}

start_pre()
{
    checkpath --directory --owner dbhub:dbhub --mode 0700 /var/log/dbhub
}

stop() {
        ebegin "Stopping dbhub-api"
        start-stop-daemon --stop --quiet --pidfile /var/run/dbhub-api.pid --exec "timeout 1m pkill dbhub-api"
        eend $?
}
###

nano /etc/init.d/dbhub-db4s
### Add
#!/sbin/openrc-run

supervisor=supervise-daemon
name="dbhub-db4s"
description="DBHub DB4S"
command="/usr/local/bin/dbhub-db4s"
command_user="dbhub"
command_background="true"
output_log="/var/log/dbhub/output.log"
error_log="/var/log/dbhub/output.log"
pidfile="/var/run/dbhub-db4s.pid"

depend() {
        need net
        after memcached minio rabbitmq-server postgresql dbhub-webui dbhub-api
}

start_pre()
{
    checkpath --directory --owner dbhub:dbhub --mode 0700 /var/log/dbhub
}

stop() {
        ebegin "Stopping dbhub-db4s"
        start-stop-daemon --stop --quiet --pidfile /var/run/dbhub-db4s.pid --exec "timeout 1m pkill dbhub-db4s"
        eend $?
}
###

nano /etc/init.d/dbhub-live
### Add
#!/sbin/openrc-run

supervisor=supervise-daemon
name="dbhub-live"
description="DBHub Live"
command="/usr/local/bin/dbhub-live"
command_args="node1 /tmp/node1"
command_user="dbhub"
command_background="true"
output_log="/var/log/dbhub/output.log"
error_log="/var/log/dbhub/output.log"
pidfile="/var/run/dbhub-live.pid"

depend() {
        need net
        after memcached minio rabbitmq-server postgresql dbhub-webui dbhub-api dbhub-db4s
}

start_pre()
{
    checkpath --directory --owner dbhub:dbhub --mode 0700 /var/log/dbhub
}

stop() {
        ebegin "Stopping dbhub-live"
        start-stop-daemon --stop --quiet --pidfile /var/run/dbhub-live.pid --exec "timeout 1m pkill dbhub-live"
        eend $?
}
###

Make service init scripts executable and add to default runlevel:

chmod +x /etc/init.d/dbhub-webui
chmod +x /etc/init.d/dbhub-api
chmod +x /etc/init.d/dbhub-live
chmod +x /etc/init.d/dbhub-db4s

rc-update add dbhub-webui default
rc-update add dbhub-api default
rc-update add dbhub-db4s default
rc-update add dbhub-live default

Finally reboot:

reboot

For additional information see also the dbhub.io wiki on github and the repo README's.

Example launch.json for remote debugging in VSC:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "dbhub-webui",
            "type": "go",
            "request": "attach",
            "debugAdapter": "dlv-dap",
            "mode": "remote",
            "remotePath": "/dbhub.io",
            "port": 2345,
            "host": "192.168.178.57",
            "substitutePath": [
                { "from": "${workspaceFolder}", "to": "/dbhub.io" }
            ]
        },
        {
            "name": "dbhub-api",
            "type": "go",
            "request": "attach",
            "debugAdapter": "dlv-dap",
            "mode": "remote",
            "remotePath": "/dbhub.io",
            "port": 2346,
            "host": "192.168.178.57",
            "substitutePath": [
                { "from": "${workspaceFolder}", "to": "/dbhub.io" }
            ]
        },
        {
            "name": "dbhub-db4s",
            "type": "go",
            "request": "attach",
            "debugAdapter": "dlv-dap",
            "mode": "remote",
            "remotePath": "/dbhub.io",
            "port": 2347,
            "host": "192.168.178.57",
            "substitutePath": [
                { "from": "${workspaceFolder}", "to": "/dbhub.io" }
            ]
        },
        {
            "name": "dbhub-live",
            "type": "go",
            "request": "attach",
            "debugAdapter": "dlv-dap",
            "mode": "remote",
            "remotePath": "/dbhub.io",
            "port": 2348,
            "host": "192.168.178.57",
            "substitutePath": [
                { "from": "${workspaceFolder}", "to": "/dbhub.io" }
            ]
        }
    ]
}

Change the remote server IP as required.

👈 Home