f1xska - topic

dap

anonymous@undisclosed.example.com (Anonymous) — Tue, 31 Dec 2024 08:55:16 +0000

2026/05/14 14:51	current
Line 1:	Line 1:
	+	====== DAP - Docker Apps with Persistence ======

	+	We had poor experience working with kubernetes-native mysql and redis databases:
	+	* Hard to setup and finetune
	+	* pmm agent setup on k8 is a nightmare
	+	* Hard to setup backup
	+	* blackbox like behavior that is not appropriate for production
	+	* Based on ceph-on-k8s with poor performance (this might change once we change to ceph-on-baremetal)
	+	* /healthcheck is sensitive to response times and might restart mysql if there is some load
	+
	+	So we built "Docker Apps with Persistence" (DAP) environment to run DCS (Docker Compose Stack) apps, that has HA, backups, works fast and easy to maintain:
	+	https://gitlab.k8.catchmedia.com/cmsys/dap
	+
	+	Setup is done on three ubuntu 22.04 servers with keepalived for set of IP numbers. Mysql and Redis servers run as docker containers on ext4:
	+	* Primary nodes of mysql are put onto dap1, secondary on dap2, with master-master replication
	+	* Primary nodes of redis are put onto dap3, secondary on dap2
	+	* cold rsync-based backup is always done on dap2 (/opt/sys/bin/dap_backup.sh)
	+
	+	New mysql or redis database are started in gitops way, adding app parameters to GIT.
	+
	+	TBD:
	+	Implement automatic replica rebuilding using:
	+	- https://forums.percona.com/t/backup-mysql-using-lvm-snapshot/16778
	+	- https://wiki.catchmedia.com/doku.php?id=system:servers:lxc:lls
	+
	+	===== Technicals =====
	+
	+	==== How to add new mysql db to DAP ====
	+
	+	- Clone https://gitlab.k8.catchmedia.com/cmsys/dap.git
	+	- Copy/update new app in apps dir
	+	- commit
	+	- deploy: ''root@dap1:/opt/dap# ./run_cicd.sh''
	+
	+	==== Change root account password ====
	+
	+	Allow root to login:
	+	<code>
	+	root@dap1:/opt/mysql_reports_nabp# dcch
	+	bash-4.2# mysql
	+	CREATE USER 'root'@'10.%' IDENTIFIED BY "delek777";
	+	GRANT ALL PRIVILEGES ON . TO 'root'@'10.%' WITH GRANT OPTION;
	+	</code>
	+
	+	==== rebuild mysql replica ====
	+
	+	Rebuilding mysql replica (example for db mysql_reports_nabp):
	+	<code>
	+	root@dap1:# cd /opt/mysql_reports_nabp/
	+	root@dap1:/opt/mysql_reports_nabp# /opt/dap/rebuild_mysql_slave.sh
	+	</code>
	+
	+	==== restart mysql ====
	+
	+	To restart mysql docker container (example for db mysql_reports_nabp, assuming dap1 is the master):
	+	<code>
	+	root@dap1:# cd /opt/mysql_reports_nabp/
	+	root@dap1:/opt/mysql_reports_nabp# dcdown
	+	...
	+	root@dap1:/opt/mysql_reports_nabp# dcup
	+	</code>
	+
	+	==== To get virtual IP/port for a docker container ====
	+
	+	To get virtual IP/port for a docker container (example for db mysql_cmts_dev, assuming dap1 is the master):
	+	<code>
	+	root@dap1:# cd /opt/mysql_cmts_dev/
	+	root@dap1:/opt/mysql_cmts_dev# dcii
	+	container [mysql_cmts_dev] on [10.3.5.91:24220]
	+	Virtual IP:port is [10.3.13.220:14220]
	+	</code>
	+
	+	==== To restore backup from BAS server ====
	+
	+	<code>
	+	# Create placeholder on a docker-enabled server, for example /opt/mysql_test
	+	root@dap2:~# mkdir /opt/mysql_test
	+
	+	# Rsync from a snapshot on bas1 (for example mysql_reports_nqa)
	+	root@dap2:/opt/mysql_test# rsync -raH root@bas1:/home/dap/snapshots/20240318_220002/mysqls/opt/mysql_reports_nqa/* .
	+
	+	# Adjust the copy, for port and names
	+	root@dap2:/opt/mysql_test# read port < port; echo 34000 > port
	+	root@dap2:/opt/mysql_test# rpl port=$port port=34000 conf.d/default.cnf
	+	root@dap2:/opt/mysql_test# mkdir binlog; chown -R 999.999 mysql binlog
	+	root@dap2:/opt/mysql_test# rpl mysql_reports_nqa mysql_test docker-compose.yml
	+
	+	# Start the docker container
	+	root@dap2:/opt/mysql_test# dcup
	+
	+	# connect to it and verify the db is visible
	+	root@dap2:/opt/mysql_test# dcmy
	+
	+	</code>

gendoc

anonymous@undisclosed.example.com (Anonymous) — Tue, 31 Dec 2024 08:55:16 +0000

2026/05/14 14:51	current
Line 1:	Line 1:
	+	====== Gendoc - Generic Docker Compose Stack apps (DCS) ======
	+
	+	gendoc is CatchMedia specific set of conventions and aliases to manage Docker Compose Stack applications (DCS).
	+
	+	See also: [[dap\|Docker Apps with Persistency]] - DCS+keepalived+haproxy
	+
	+	We try to use gendoc/dcs instead of k8 because k8-based apps are notoriously harder to setup, manage and backup. The complexity price for HA and scalability just does not pay off for most applications we use.
	+
	+	===== Conventions =====
	+
	+	* aliases entity is installed
	+	* DCS app has /opt/DCS_NAME/docker-compose.yml file
	+	* The DCS might require dedicated IP using macvlan. If dedicated outbound source IP is needed, then Firewall->NAT->Outbound tuning on FW might be necessary
	+	* docker-compose.yml might have additional custom fields:<code>
	+	x-ska:
	+	start_on_boot: "no"
	+	on_demand_port: 25565
	+	</code>
	+	* for DCS with persistency, /opt/DCS_NAME should be separate LVM volume, mounted on boot. Same for production-grade apps, to protect from disk full situations.
	+	* systemd service gendoc starts DCS apps with start_on_boot=yes
	+
	+	===== Aliases =====
	+
	+	Aliases are actually scripts that sit in /opt/aliases/bin (should be in PATH). They must be run from the home directory of DCS - /opt/DCS_NAME, for example /opt/pamail
	+
	+	* ''dcup'' - start DCS app
	+	* ''dcdown'' - stop DCS app
	+	* ''dclog'' - see runtime log
	+	* ''dcch'' - try to run bash or sh inside of the container (if container has shell in it). If you must get in as root, use this cmd (example for mysql_padb_a1): ''docker exec -u 0 -it mysql_padb_a1 /bin/bash -l''
	+	* ''dcii'' - see information on the DCS
	+	* ''dcmy'' - for mysql containers - connect to mysql server using mysql CLI
	+	* ''dcmyv'' - for mysql containers - connect to mysql server using mysql CLI over Virtual IP
	+	* ''dcrc'' - for redis containers - connect to redis server using redis CLI
	+	* ''docps'' - See list of DCS running: docps

idrac

anonymous@undisclosed.example.com (Anonymous) — Tue, 31 Dec 2024 08:55:16 +0000

2026/05/14 14:51	current
Line 1:	Line 1:
	+	Boot from ISO Dell R630: Remote File Share→Image File Path (must be samba):\\
	+	\\
	+	''//\\samba4tmp.gendoc.catchmedia.com\iso\ubuntu-20.04.6-live-server-amd64.iso//''
	+

k8

anonymous@undisclosed.example.com (Anonymous) — Tue, 31 Dec 2024 09:00:07 +0000

2024/12/24 16:17		current
Line 1:		Line 1:
-	====== k8 - kubernetes ======	+	====== k8 notes of Arie ======

-	Useful aliases used in CM
	<code>		<code>
		+	#Bash into container
		+	k exec -it wiki-dokuwiki-9f9fd859d-2wfx6 bash
		+
		+	### Useful aliases ###
	kush [-n ns] pod_name # get shell into pod		kush [-n ns] pod_name # get shell into pod
	kud pod_name # kubectl describe pod		kud pod_name # kubectl describe pod
	kug pod pod_name # kubectl get pod_name		kug pod pod_name # kubectl get pod_name
	ktail pod_name # get logs for the pod		ktail pod_name # get logs for the pod
-	</code>

-	[[private:arie:k8\|Memos of arie on k8]]	+	# fetch chart to analyse values.yaml
		+	helm fetch gitlab/gitlab --untar
		+
		+	# debug certs
		+	kug orders,challenges -o wide
		+
		+	# Delete PV that has finalizers
		+	pv=pvc-daec7eed-fc8e-4281-a15b-6b370df54804
		+	echo '{"metadata": {"finalizers": null}}' \| kubectl patch pv $pv --patch-file=/dev/stdin
		+	# Delete POD that got stuck (and no finalizers to delete)
		+	kubectl delete pods fluent-bit-xczr2 --grace-period=0 --force
		+
		+	#copy file into pod
		+	kubectl cp /mnt/common/tmp/run dev/page2-dp-77f87664bb-4l8mc:/tmp/run
		+
		+	# Delete namespace that got stuck
		+	namespaceToDelete=brocha
		+	kubectl get namespace "$namespaceToDelete" -o json \| \
		+	jq 'del(.spec.finalizers)' \| \
		+	kubectl replace --raw /api/v1/namespaces/$namespaceToDelete/finalize -f -
		+
		+	# drain and evict pods from a node
		+	kubectl drain --ignore-daemonsets --delete-emptydir-data k8i03
		+	#untaint
		+	kubectl uncordon k8i03
		+
		+	# port forwarding of localport 40090 to port 9090 of the svc
		+	kubectl port-forward -n monitoring svc/prometheus-k8s 40090:9090 --address='0.0.0.0'
		+
		+	# see nodes labels
		+	kug nodes --show-labels
		+	# remove label worker from a node:
		+	kubectl label node ip-10-0-7-187.ec2.internal worker-
		+	# add control-plane label
		+	kubectl label node k8r05 node-role.kubernetes.io/control-plane=
		+	kubectl label node k8p01 general=yes
		+
		+	# See cpu/mem usage by nodes
		+	kubectl top node
		+
		+	# Show pod by label
		+	kug pod -n monitoring -l app.kubernetes.io/name=kube-state-metrics
		+	# kill pods by label
		+	kudel pod -n nqa -l app=etljobs-sel
		+
		+	# watch hpa of gwpa@rnd
		+	kug hpa -n rnd gwpa-hpa --watch
		+
		+	# events on API of prod2 farm
		+	kug events -n prod2
		+	</code>

kafka

anonymous@undisclosed.example.com (Anonymous) — Tue, 31 Dec 2024 08:55:16 +0000

2026/05/14 14:51	current
Line 1:	Line 1:
	+	====== Apache Kafka ======

	+	We use [[https://kafka.apache.org/\|Apache Kafka]] for messaging queues.
	+
	+	==== How to see kafka configurations of a running consumer ====
	+
	+	<code>
	+	root@k8dc1:~# kush -n rnd cmkafka-0
	+	unset JMX_PORT
	+	# list topics
	+	/opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list
	+	# get details on topic play_placement
	+	/opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --describe --topic play_placement
	+	</code>
	+
	+	https://learn.conduktor.io/kafka/kafka-topics-cli-tutorial/
	+
	+	==== Increase amount of partitions ====
	+
	+	On CM farm:<code>
	+	kush -n dev sys-dp
	+	export KAFKA_ADDRESS=cmkafka2test:9092
	+	export REPLICATION_FACTOR=1
	+	/opt/bin/kafka_topics_set.py cm_events_batch_queue 10
	+	</code>