f1xska

<h1>DAP - Docker Apps with Persistence</h1>
<div class="level1">

<p>
We had poor experience working with kubernetes-native mysql and redis databases:
</p>
<ul>
<li class="level1"> Hard to setup and finetune</li>
<li class="level1"> pmm agent setup on k8 is a nightmare</li>
<li class="level1"> Hard to setup backup</li>
<li class="level1"> blackbox like behavior that is not appropriate for production</li>
<li class="level1"> Based on ceph-on-k8s with poor performance (this might change once we change to ceph-on-baremetal)</li>
<li class="level1"> /healthcheck is sensitive to response times and might restart mysql if there is some load</li>
</ul>

<p>
So we built "Docker Apps with Persistence" (DAP) environment to run DCS (Docker Compose Stack) apps, that has HA, backups, works fast and easy to maintain:
<a href="https://gitlab.k8.catchmedia.com/cmsys/dap" class="urlextern" title="https://gitlab.k8.catchmedia.com/cmsys/dap" rel="ugc nofollow">https://gitlab.k8.catchmedia.com/cmsys/dap</a>
</p>

<p>
Setup is done on three ubuntu 22.04 servers with keepalived for set of IP numbers. Mysql and Redis servers run as docker containers on ext4:
</p>
<ul>
<li class="level1"> Primary nodes of mysql are put onto dap1, secondary on dap2, with master-master replication</li>
<li class="level1"> Primary nodes of redis are put onto dap3, secondary on dap2</li>
<li class="level1"> cold rsync-based backup is always done on dap2 (/opt/sys/bin/dap_backup.sh)</li>
</ul>

<p>
New mysql or redis database are started in gitops way, adding app parameters to GIT.
</p>

<p>
TBD:
Implement automatic replica rebuilding using:
</p>
<ol>
<li class="level1"> <a href="https://forums.percona.com/t/backup-mysql-using-lvm-snapshot/16778" class="urlextern" title="https://forums.percona.com/t/backup-mysql-using-lvm-snapshot/16778" rel="ugc nofollow">https://forums.percona.com/t/backup-mysql-using-lvm-snapshot/16778</a></li>
<li class="level1"> <a href="https://wiki.catchmedia.com/doku.php?id=system:servers:lxc:lls" class="urlextern" title="https://wiki.catchmedia.com/doku.php?id=system:servers:lxc:lls" rel="ugc nofollow">https://wiki.catchmedia.com/doku.php?id=system:servers:lxc:lls</a></li>
</ol>

</div>

<h2>Technicals</h2>
<div class="level2">

</div>

<h3>How to add new mysql db to DAP</h3>
<div class="level3">
<ol>
<li class="level1"> Clone <a href="https://gitlab.k8.catchmedia.com/cmsys/dap.git" class="urlextern" title="https://gitlab.k8.catchmedia.com/cmsys/dap.git" rel="ugc nofollow">https://gitlab.k8.catchmedia.com/cmsys/dap.git</a></li>
<li class="level1"> Copy/update new app in apps dir</li>
<li class="level1"> commit</li>
<li class="level1"> deploy: <code>root@dap1:/opt/dap# ./run_cicd.sh</code></li>
</ol>

</div>

<h3>Change root account password</h3>
<div class="level3">

<p>
Allow root to login:
</p>
<pre class="code">root@dap1:/opt/mysql_reports_nabp# dcch
bash-4.2# mysql
CREATE USER 'root'@'10.%' IDENTIFIED BY "delek777";
GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.%' WITH GRANT OPTION;

</pre><p> </p>

</div>

<h3>rebuild mysql replica</h3>
<div class="level3">

<p>
Rebuilding mysql replica (example for db mysql_reports_nabp):
</p>
<pre class="code">root@dap1:# cd /opt/mysql_reports_nabp/
root@dap1:/opt/mysql_reports_nabp# /opt/dap/rebuild_mysql_slave.sh

</pre><p> </p>

</div>

<h3>restart mysql</h3>
<div class="level3">

<p>
To restart mysql docker container (example for db mysql_reports_nabp, assuming dap1 is the master):
</p>
<pre class="code">root@dap1:# cd /opt/mysql_reports_nabp/
root@dap1:/opt/mysql_reports_nabp# dcdown
...
root@dap1:/opt/mysql_reports_nabp# dcup

</pre><p> </p>

</div>

<h3>To get virtual IP/port for a docker container</h3>
<div class="level3">

<p>
To get virtual IP/port for a docker container (example for db mysql_cmts_dev, assuming dap1 is the master):
</p>
<pre class="code">root@dap1:# cd /opt/mysql_cmts_dev/
root@dap1:/opt/mysql_cmts_dev# dcii
container [mysql_cmts_dev] on [10.3.5.91:24220]
Virtual IP:port is [10.3.13.220:14220]

</pre><p> </p>

</div>

<h3>To restore backup from BAS server</h3>
<div class="level3">
<pre class="code"># Create placeholder on a docker-enabled server, for example /opt/mysql_test
root@dap2:~# mkdir /opt/mysql_test

# Rsync from a snapshot on bas1 (for example mysql_reports_nqa)
root@dap2:/opt/mysql_test# rsync -raH root@bas1:/home/dap/snapshots/20240318_220002/mysqls/opt/mysql_reports_nqa/* .

# Adjust the copy, for port and names
root@dap2:/opt/mysql_test# read port < port; echo 34000 > port
root@dap2:/opt/mysql_test# rpl port=$port port=34000 conf.d/default.cnf
root@dap2:/opt/mysql_test# mkdir binlog; chown -R 999.999 mysql binlog
root@dap2:/opt/mysql_test# rpl mysql_reports_nqa mysql_test docker-compose.yml

# Start the docker container
root@dap2:/opt/mysql_test# dcup

# connect to it and verify the db is visible
root@dap2:/opt/mysql_test# dcmy

</pre><p> </p>

</div>

Enable Complex Tables Editor Height px Turn supplementary image paste support on

Edit summary: Minor changes