Kafka: deleting offset of a consumer group from a topic
After Kafka 2.8
./kafka-consumer-groups.sh \
--bootstrap-server <bootstrap-server-url> \
--delete-offsets \
--group <my-group> \
--topic <topic-name>
After Kafka 2.8
./kafka-consumer-groups.sh \
--bootstrap-server <bootstrap-server-url> \
--delete-offsets \
--group <my-group> \
--topic <topic-name>
After some time of work Zookeeper writes a huge transaction log and adding any new event in the log makes fsync to FS very long
Logs:
2023-09-21 06:36:22,085 [myid:] - WARN [SyncThread:3:o.a.z.s.p.FileTxnLog@394] - fsync-ing the write ahead log in SyncThread:3 took 1832ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide
2023-09-21 06:36:25,120 [myid:] - WARN [SyncThread:3:o.a.z.s.p.FileTxnLog@394] - fsync-ing the write ahead log in SyncThread:3 took 1072ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide
2023-09-21 06:36:39,506 [myid:] - WARN [SyncThread:3:o.a.z.s.p.FileTxnLog@394] - fsync-ing the write ahead log in SyncThread:3 took 1393ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide
To prevent size growth we can add two options to Zookeeper
In config file:
autopurge.snapRetainCount=7
autopurge.purgeInterval=24
Or via Env variables when running in Docker
ZOO_AUTOPURGE_PURGEINTERVAL: "24"
ZOO_AUTOPURGE_SNAPRETAINCOUNT: "7"
This options will keep only last 7 days in the transaction log
When you try to update/pull or get a diff for the Git repository, you can get a warning:
warning: Not a git repository. Use --no-index to compare two paths outside a working tree
usage: git diff --no-index [<options>] <path> <path>
It is because the new version of Git forbidden to run git commands in a directory that is not owned by the current user ( CVE-2022-24765 )
To fix, you need to change permission for the directory or set security options to git when running commands in CI/CD
git config --global --add safe.directory $(pwd)
In case you forgot the root password or can’t login to the CentOS, we can start OS by editing kernel boot options in Grub
When the system is booting press any key (left, right, top, down) on Grub screen
Select the kernel you need to run and press ‘e’ to open the editor
Read moreWhen building debezium-connect image on Gitlab runners had error
* getaddrinfo() thread failed to start
* Could not resolve host: packages.confluent.io
* Closing connection 0
curl: (6) getaddrinfo() thread failed to start
After running all commands in Container found the problem
Step 4/11 : RUN microdnf upgrade --refresh && microdnf -y install curl && microdnf clean all
---> Running in fdbcae0c9647
(microdnf:7): GLib-ERROR **: 11:03:33.774: file ../glib/gthread-posix.c: line 1340 (g_system_thread_new): error 'Operation not permitted' during 'pthread_create'
The command '/bin/sh -c microdnf upgrade --refresh && microdnf -y install curl && microdnf clean all' returned a non-zero code: 133
Bug on Fedora issue tracker https://bugzilla.redhat.com/show_bug.cgi?id=2019901
Fix by running a container with security options
podman run -it --rm --security-opt seccomp=unconfined registry.access.redhat.com/ubi9-beta/ubi-minimal
or add config to Runner
cat /etc/gitlab-runner/config.toml
....
security_opt = ["seccomp:unconfined"]
....
If it’s not enough and the internal queue grows very fast – change the Wiredtiger limit of concurrent transaction
db.adminCommand( { setParameter: 1,
wiredTigerConcurrentReadTransactions: xx } )
db.adminCommand( { setParameter: 1,
wiredTigerConcurrentWriteTransactions: xx } )
The newer version of the DKMS Nvidia driver won’t install on Ubuntu 22.04.
From logs :
cat /var/lib/dkms/nvidia/520.56.06/build/make.log
....
cc -Wp,-MMD,/var/lib/dkms/nvidia/520.56.06/build/nvidia/.nv-mmap.o.d -nostdinc -I./arch/x86/include -I./arch/x86/include/generated -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/compiler-version.h -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Wno-format-security -std=gnu11 -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=none -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mindirect-branch=thunk-extern -mindirect-branch-register -mindirect-branch-cs-prefix -mfunction-return=thunk-extern -fno-jump-tables -mharden-sls=all -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-address-of-packed-member -O2 -fno-allow-store-data-races -Wframe-larger-than=1024 -fstack-protector-strong -Wno-array-bounds -Wimplicit-fallthrough=5 -Wno-main -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -ftrivial-auto-var-init=zero -fzero-call-used-regs=used-gpr -pg -mrecord-mcount -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-maybe-uninitialized -Wno-alloc-size-larger-than -fno-strict-overflow -fno-stack-check -fconserve-stack -Werror=date-time -g -gdwarf-5 -I/var/lib/dkms/nvidia/520.56.06/build/common/inc -I/var/lib/dkms/nvidia/520.56.06/build -Wall -MD -Wno-cast-qual -Wno-error -Wno-format-extra-args -D__KERNEL__ -DMODULE -DNVRM -DNV_VERSION_STRING=\"520.56.06\" -Wno-unused-function -Wuninitialized -fno-strict-aliasing -ffreestanding -mno-red-zone -mcmodel=kernel -DNV_UVM_ENABLE -DNV_SPECTRE_V2=0 -DNV_KERNEL_INTERFACE_LAYER -I/var/lib/dkms/nvidia/520.56.06/build/nvidia -DNVIDIA_UNDEF_LEGACY_BIT_MACROS -UDEBUG -U_DEBUG -DNDEBUG -fsanitize=bounds -fsanitize=shift -fsanitize=bool -fsanitize=enum -DMODULE -DKBUILD_BASENAME='"nv_mmap"' -DKBUILD_MODNAME='"nvidia"' -D__KBUILD_MODNAME=kmod_nvidia -c -o /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-mmap.o /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-mmap.c ; ./tools/objtool/objtool --hacks=jump_label --hacks=noinstr --retpoline --rethunk --sls --stackval --static-call --uaccess --module /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-mmap.o
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-pci.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-dmabuf.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-acpi.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-cray.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-i2c.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-dma.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-mmap.o] Error 1
make[1]: *** [Makefile:1858: /var/lib/dkms/nvidia/520.56.06/build] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-6.0.0-060000-generic'
make: *** [Makefile:82: modules] Error 2
Error “unrecognized command-line option ‘-ftrivial-auto-var-init=zero’ looks like we need a newer version of GCC. Firstly try to install deps
sudo apt install build-essential
Read more Use an app named ‘safe’
safe copy secret/staging/user secret/prod/user
URL: https://github.com/starkandwayne/safe/tree/v1.6.1
Sometimes I need to have separate partitions on VPS in Hetzner.
The simplest way to create a custom disk structure – use Cloud-init script
With commands:
#cloud-config
# we need to disable the initial call to "growpart" as otherwise the first partition would consume
# all space on the dist
#
# The final disk layout is:
# #0 0 - 10GB - ext4 on /
# #1 10GB - 100% - lvm
#
# The reason for this approach is that when using CentOS on Hetzner you cannot use
# the "disk_setup" mechanism from cloud-init, it simply is ignored.
growpart:
mode: off
runcmd:
- printf "fix\n" | parted ---pretend-input-tty /dev/sda print # Fix parted error, first
- [ parted, "/dev/sda", mkpart, primary, ext2, "10GB", "100%" ] # create a new partition, starting at 10GB
- [ parted, "/dev/sda", set, "2", lvm, on ] # set LVM flag
- [ growpart, "/dev/sda", "1" ] # grow first partition to 10GB
- [ partx, --update, "/dev/sda" ] # reload partition table
- [ resize2fs, /dev/sda1 ] # resize first partition (/) to 10GB
- [ pvcreate, "/dev/sda2" ] # create PV on /dev/sda2 (100%-10GB)
- [ vgcreate, vg1, "/dev/sda2" ] # create VG, adding PV /dev/sda2
repo_update: true
repo_upgrade: all
packages:
- lvm2 # missing in the Hetzner image
In this example, we have 10Gb for / and empty disk /dev/sdb
For count rows in each PostgreSQL table, we need to create a function and run it for all schemas
create function
cnt_rows(schema text, tablename text) returns integer
as
$body$
declare
result integer;
query varchar;
begin
query := 'SELECT count(1) FROM ' || schema || '.' || tablename;
execute query into result;
return result;
end;
$body$
language plpgsql;
Read more After upgrading the kernel to a newer kernel, OS won’t boot with a stacked message
pci 0000 : 00 : 07.0 : DPC : RP PIO log size 0 is invalid
xhci_hcd 0000 : 00 : 0d.0 : can't setup : -12
xhci hcd 0000 : 00 : 0d.0 : init 0000 : 00 : 0d.0 fail , -12
thunderbolt 0000 : 00 : 0d.2 : failed to determine connection manager , aborting
xhci hcd 0000 : 00 : 14.0 : can't setup : -12
xhci_hcd 0000 : 00 : 14.0 : init 0000 : 00 : 14.0 fail , -12
1801_smbus . 0000 : 00 : 1f.4 : Transaction timeout
1801_smbus 0000 : 00 : 1f.4 : Failed terminating the transaction
1801_smbus 0000 : 00 : 1f.4 : SMBus is busy , can't use it !
i915 0000 : 00 : 02.0 : [ drm ] * ERROR * Scratch setup failed
i915 0000 : 00 : 02.0 : Device initialization failed ( -12 )
Read more