We had poor experience working with kubernetes-native mysql and redis databases:

• Hard to setup and finetune
• pmm agent setup on k8 is a nightmare
• Hard to setup backup
• blackbox like behavior that is not appropriate for production
• Based on ceph-on-k8s with poor performance (this might change once we change to ceph-on-baremetal)
• /healthcheck is sensitive to response times and might restart mysql if there is some load

So we built “Docker Apps with Persistence” (DAP) environment to run DCS (Docker Compose Stack) apps, that has HA, backups, works fast and easy to maintain: https://gitlab.k8.catchmedia.com/cmsys/dap

Setup is done on three ubuntu 22.04 servers with keepalived for set of IP numbers. Mysql and Redis servers run as docker containers on ext4:

• Primary nodes of mysql are put onto dap1, secondary on dap2, with master-master replication
• Primary nodes of redis are put onto dap3, secondary on dap2
• cold rsync-based backup is always done on dap2 (/opt/sys/bin/dap_backup.sh)

New mysql or redis database are started in gitops way, adding app parameters to GIT.

TBD: Implement automatic replica rebuilding using:

1. https://forums.percona.com/t/backup-mysql-using-lvm-snapshot/16778
2. https://wiki.catchmedia.com/doku.php?id=system:servers:lxc:lls

1. Clone https://gitlab.k8.catchmedia.com/cmsys/dap.git
2. Copy/update new app in apps dir
3. commit
4. deploy: root@dap1:/opt/dap# ./run_cicd.sh

Allow root to login:

root@dap1:/opt/mysql_reports_nabp# dcch
bash-4.2# mysql
CREATE USER 'root'@'10.%' IDENTIFIED BY "delek777";
GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.%' WITH GRANT OPTION;

Rebuilding mysql replica (example for db mysql_reports_nabp):

root@dap1:# cd /opt/mysql_reports_nabp/
root@dap1:/opt/mysql_reports_nabp# /opt/dap/rebuild_mysql_slave.sh 

To restart mysql docker container (example for db mysql_reports_nabp, assuming dap1 is the master):

root@dap1:# cd /opt/mysql_reports_nabp/
root@dap1:/opt/mysql_reports_nabp# dcdown
...
root@dap1:/opt/mysql_reports_nabp# dcup

To get virtual IP/port for a docker container (example for db mysql_cmts_dev, assuming dap1 is the master):

root@dap1:# cd /opt/mysql_cmts_dev/
root@dap1:/opt/mysql_cmts_dev# dcii
container [mysql_cmts_dev] on [10.3.5.91:24220]
Virtual IP:port is [10.3.13.220:14220]

# Create placeholder on a docker-enabled server, for example /opt/mysql_test
root@dap2:~# mkdir /opt/mysql_test

# Rsync from a snapshot on bas1 (for example mysql_reports_nqa)
root@dap2:/opt/mysql_test# rsync -raH root@bas1:/home/dap/snapshots/20240318_220002/mysqls/opt/mysql_reports_nqa/* .

# Adjust the copy, for port and names
root@dap2:/opt/mysql_test# read port < port; echo 34000 > port
root@dap2:/opt/mysql_test# rpl port=$port port=34000 conf.d/default.cnf
root@dap2:/opt/mysql_test# mkdir binlog; chown -R 999.999 mysql binlog
root@dap2:/opt/mysql_test# rpl mysql_reports_nqa mysql_test docker-compose.yml 

# Start the docker container
root@dap2:/opt/mysql_test# dcup

# connect to it and verify the db is visible
root@dap2:/opt/mysql_test# dcmy