Replication manager overview

Replication manager (repman) is an orchestrator for high availability on top of MariaDB 10.x and MySQL and Percona Server 5.7 GTID replication topologies. It supports both interactive and automated failover of the master server. It verifies the integrity of the slave servers before promoting one of them as the replacement master and it also protects the slaves by automatically setting them into read-only mode.

Replication manager cluster has at least 2 nodes, one should be master node and the rest are slaves. It is advised to use at least 3 nodes cluster to keep the cluster tolerant to failures when losing or stopping a slave. Replication manager support for replication tree or relay slaves architecture, in case of master death one of the slaves under the relay is promoted as a master. It also support Maxscale, ProxySQL, Haproxy, etc for routing read/write request to databases.

SlapOS overview

SlapOS provides a compact and unified solution to automate provisioning, orchestration, monitoring and preventive maintenance of software services that can be deployed in data centers, edge nodes, user terminals or embedded systems without the need for virtualization.

SlapOS is composed of two types of servers:

• SlapOS Nodes which run on a minimal GNU/Linux distribution with installed slapos-node package. It consists of SlapGrid daemon which manage Buildout for bootstrapping applications and supervisord to control running processes. Depending on the server ressources, a SlapOS Node can host many Mariadb and Replication Manager application instances.
• SlapOS Master is the orchestrator. It informs each node which software should be build or deployed and which instances of which software should be run. The Node also return to the master the state of services that are deployed as well as information usefull to connect to services. This documentation explain how to request replication manager service with Rapid.Space.

Our goal is to provide an easier way to automatically deploy a full replication manager cluster, with one or more mariadb backups instances then, allow users to request replicated databases into deployed cluster. To do that, we will have to create a replication manager software release for building the software with all required dependencies and provide instances profiles for deploying the cluster and requesting database shared instances.

Software integration and new orchestrator

To provision the cluster, replication manager can generate database and proxy configurations based tags provided and hardware resources description. The prov-orchestrator parameter supplied to repman indicate on which platform the cluster is going to be provisioned. It's useful for generating configuration files according the node platform that will run database and proxies services. Untill now, local, onpremise, OpenSVC and Kubernate orchestrator was supported but none of them where compatible with SlapOS. Instead of regenerating multiple configurations and scripts managed by repman, the easier solution was to add a new orchestrator to repman called slapos for support deployment into Computer Partition. Cluster configuration was updated with new information about the computer partition where the service is deployed, like slapos-proxysql-partitions, proxysql-servers-ipv6, slapos-db-partitions.

We only support Mariadb as database server for clusters. Event if replication manager can support multiple proxies for a  single cluster, we are deploying for now a single ProxySQL per cluster to keep things less complex. The proxy is the entry point to access databases. Initialy, proxies was configured only with IPv4 network, but for slapos support we added support for IPv6 so that, a service deployed in another server can access the database using IPv6 network.

Replication Manager Software Release

The first step of integration is to build Replication manager component, which is mainly writen in go language. By using the golang environment provided by SlapOS, we simply have to write the component buildout profile which extends golang environment and run go build with flags required to build replication manager from sources.

Once we was able to build the component, we create the software release for automating installation of repman and needed dependencies. There is a number of components available to use from the SlapOS repository, but we had to add few more required for replication manager, like: restic (for database backup), sysbench and proxysql.

With the software release, SlapOS node is able to build the entirely software and required components, generate instance configurations and different services, then run them using supervisord. The software release for replication manager was added here.

First release of repman

The software release version 1.0.259 was released on rapid.space, and can be used for deploying clusters. First of all, we should build the SR into servers which will host the cluster. For example, if we want to deploy a cluster with one master and 3 backups (all instances allocated in different servers for good resiliency),  we need to setup 5 slapos node servers. The procedure to add a software release to SlapOS Master and push compiled binary to cache is available here. This other documentation explain how to supply a software release from command line.

Once the software release is built on servers, we can request cluster instance from rapid.space UI or by using command line by running following commands:

$ slapos console
2022-08-01 10:07:57 slapos[16518] INFO IPython not available - using plain Python shell

>>> import json
>>> repman_comp_guid = "COMP-XXX"
>>> json_parameter = """{
...     "repman-cluster-dict": {
...             "cluster1": {
...                     "database-amount": 4,
...                     "-sla-0-computer_guid": "COMP-XXX",
...                     "-sla-1-computer_guid": "COMP-XXX",
...                     "-sla-2-computer_guid": "COMP-XXX",
...                     "-sla-3-computer_guid": "COMP-XXX",
...                     "tags": [
...                             "gtidstrict", "bind", "pkg", "innodb", "noquerycache",
...                             "slow", "pfs", "linux", "readonly", "diskmonitor",
...                             "sqlerror", "compressbinlog", "bm4ci", "mroonga",
...                             "utctime", "readcommitted", "nohandshake", "ssl"
...                     ],
...                     "proxysql-user": "external",
...                     "proxy-tags": ["pkg", "masterslave", "linux", "noreadwritesplit", "ssl"],
...                     "innodb-file-per-table": 1,
...                     "use-ipv6": true,
...                     "autorejoin": true,
...                     "autoseed": true,
...                     "failover-mode": "manual",
...                     "failover-limit": 5,
...                     "failover-falsepositive-heartbeat": true,
...                     "failover-falsepositive-heartbeat-timeout": 3,
...                     "failover-falsepositive-ping-counter": 5,
...                     "failover-max-slave-delay": 30,
...                     "failover-readonly-state": true,
...                     "failover-restart-unsafe": false,
...                     "failover-time-limit": 0,
...                     "switchover-at-equal-gtid": false,
...                     "switchover-slave-wait-catch": true,
...                     "switchover-wait-kill": 5000,
...                     "switchover-wait-trx": 10,
...                     "switchover-wait-write-query": 10
...             }
...     }
... }"""
>>> request(
  'https://lab.nexedi.com/nexedi/slapos/raw/1.0.259/software/repman/software.cfg',
  'repmanClusterProd',
  partition_parameter_kw=json.loads(json_parameter),
  software_type='default',
  filter_kw={'computer_guid': repman_comp_guid}
)

This will request a cluster called cluster1 with 4 mariadb instances and one ProxySQL. The variable repman_comp_guid is the computer GUID of replication manager instance, it's the root instance which request mariadb instances. Parameters -sla-XX-computer_guid specify where the mariadb XX will be allocated. Details about all parameters are described in the file instance-repman-input-schema.json.

After few minutes, the cluster will be deployed and connection parameters will be published to rapid.space panel UI.

Request a database instance

Our deployed cluster has no database yet. To add a database to the cluster, we need to request a shared instance which is also called Slave Instance. The slave instance Parameters are sent to the cluster, these parameters are then used to create the database while redeploying the cluster instance. The advantage of requesting a slave instance to create the database is that, different users can create their own databases in the same cluster. Users will only see databases they have created without know any informations about instance of others.

A few minutes after the request, the instance is ready, database was created on master and replicated to slaves. Published connections parameters show URLs for connecting to database, both IPv6 and IPv4 url required SSL connection as database account creation includes a REQUIRE SSL clause in the GRANT statement. IPv6 url allow to access to database from internet.

Setup Wordpress with repman database

Now we have a replicated database deployed, we want to use it with our Wordpress service. Our wordpress instance and replication manager database are on different network so we will use IPv6 for configuration.

After we have validated database information, wordpress will finish configuration and install the database.