YugabyteDB on Jelastic in the Hidora cloud 🇨🇭

Jelastic is a DevOps PaaS, running containers with easy scaling and pay-per-use charging. If you want to test YugabyteDB (open-source PostgreSQL-compatible distributed SQL database) you may encounter some specificity. The main issue you may encounter comes from a bug in OpenVZ/Virtuozzo, the container and virtualization software used by Jelastic, which doesn’t expose the number of CPU correctly. I propose a workaround to make it run but check the related issues ans support before doing that in production.

I’m running this on Hidora, a Swiss cloud providing an easy Pay-As-You-Use hosting Platform-as-a-Service, with great user experience and support. You should be able to run this on the free trial

I connect to the console https://app.hidora.com/

I’ll create a new environment

Create New Environment

There’s a choice of existing containers. To install the YugabyteDB image, I choose “Custom Container Image”

Custom Container Image

Of course YugabyteDB can be deployed in Docker Engine or Kubernetes. I’ve demo this at the Geneva DevOps meetup but to show the basics here I’m installing it as a VPS container (Virtual Private Server)

I choose the Environment Name yb-demo

I choose the container image with Select Image: yugabytedb/yugabyte

The image doesn’t start the database because there are multiple possibilites to start and sacle out a YugabyteDB database. The entrypoint command can be set in the Container Layer Settings (Menu: 🔧More)

Container Layer Settings

CMD / Entry Point

There is the CMD / Entry Point I use for this demo:

bash -c '
yum install -y gcc && echo -e "#define _GNU_SOURCE\n#include <unistd.h>\nint sched_getcpu (void) { return sysconf(_SC_NPROCESSORS_CONF)-1 ; };\n" > sched_getcpu.c && gcc -shared -o /tmp/sched_getcpu.so -fPIC sched_getcpu.c ; export LD_PRELOAD=/tmp/sched_getcpu.so && ulimit -n 1048576 && /home/yugabyte/bin/yugabyted start --daemon=false --base_dir /mnt/disk0 --data_dir=/mnt/disk1 --master_webserver_port=80 --ysql_port 5433 --listen $HOSTNAME $(echo --join $MASTER_HOST | grep -v ${HOSTNAME%%-*})
'

This deserves a little explanation. The goal is to run yugabyted start to start the first node but also be able to scale by adding more containers with the same command

• I use MASTER_HOST to --join the first node, except when we are on the first node. (this is the reason for grep -v to remove this when the $MASTER_HOST is the current $HOSTNAME)
• I set ulimit higher
• I workaround an issue with OpenVZ/Virtuozzo not counting CPU correctly — and OpenVZ is used by Jelastic PaaS. See:

percpu_rwlock assert sched_getcpu() within the range of visible CPUs is broken on OpenVZ · Issue…

⚠ Please check the issue before using this in production, there may be some performance issues with this workaround. It seems that it is not considered as a LXC bug, but many software had to workaround this, like OpenJDK or Puppet. I’ve put the ink in this blog post. The git issue above will track our solution for YugabyteDB.

When starting with yugabyted the first 3 nodes must be started one after the others as they increase the cluster Replication Factor. This is why I start with one node:

Stateless horizontal scaling

You may choose Stateful scaling which starts the containers one after the other like this:

Stateful horizontal scaling

For this demo, to see the nodes starting one by one and the cluster going from Replication Factor RF=1 to RF=3 I’ve used:

Stateful horizontal scaling with 1 node

This also depends on your persistence. Here, by default in Hidora, the persistent storage is NFS which is not useful for YugabyteDB because it is a shared-nothing architecture. I’m using local storage:

Of course, for production you need to think about the data protection you want, with a combination of Replication Factor, cross-cluster replication, persistent volumes…

So, this starts the node(s). As defined in my yugabyted start command, the logs are visible in the directory set above by --base-dir

Application Servers Config — Disks

yugabyted.log

Note that starting the container, and installing gcc for the workaround above can take time. Restart the container if it didn’t came up in the time expected:

Restart Container

For example, you will find this when a node cannot find the master

[yugabyted start] 2022-06-27 05:35:16,775 ERROR:  | 18.5s | Unable to add master node133619-yb-demo.hidora.com to existing cluster at node133617.
For more information, check the logs in /mnt/disk0/logs
[yugabyted start] 2022-06-27 05:35:16,775 INFO:  | 18.5s | Shutting down...

But that’s when you have started more than one node. For the moment, I have only one yb-master and one yb-tserver

The console is visible though the load balancer http://yb-demo.hidora.com/

master console though load balancer

Note that don’t really need the load balancer here because each master will get the info from the leader. Click on it to see it.

Now, you can scale the cluster. You need at least 3 nodes to get a Replication Factor RF=3 protection:

Environment Settings

Horizontal Scaling

The console shows the two masters:

2 masters console

One more time:

Here is Replication Factor 3:

RF=3 console

You are ready to test, connecting to any node on port 5433 (as set by ysql_port 5433) with any PostgreSQL driver. There’s also a Cassandra endpoint but I keep things simple here.

Now, you are ready to test resilience by stopping a node, or rolling upgrades:

image version

With Replication Factor 3, one node can be down and the application continues.

The PostgreSQL endpoint is defined by --ysql_port and I used the default 5433. You can choose a port opened to a load balancer or use the YugabyteDB Smart Drivers where you provide one endpoint only and will use the others when opening new connections

For a lab, you can make all IPs public and open the ports, or get them redirected:

So, here is an example of connection with psql :

psql postgres://node133598-yb-demo.hidora.com:11328

You have (mostly) all PostgreSQL features, seeing the distributed database as one logically but distributed and replicated to many nodes. Those nodes are visible from yb_servers()

select * from yb_servers()

This is the cluster-info that is used by the Smart Drivers to balance the connections dynamically

You can easily check the load of your cluster:

Statistics

And, as you have seen that you can add or remove node, why not testing auto-scaling:

auto-scaling rules

When adding nodes, data is automatically rebalanced, as well as Raft leaders. But do not automatically remove nodes: if the quorum is not there the database will stop. There’s a need to inform the database before so that it re-balances data before the nodes are down.

If you are testing YugabyteDB on Hidora or any Jelastic PaaS, I’m Developer Advocate and love to get feedback. So please let me know here or on my dev.to blog, in the Yugabyte Community slack channel, or on Twitter: