Author Archive

Kafka: deleting offset of a consumer group from a topic

After Kafka 2.8

./kafka-consumer-groups.sh \
  --bootstrap-server <bootstrap-server-url> \
  --delete-offsets \
  --group <my-group> \
  --topic <topic-name>

Prevent storing huge transaction logs in Zookeeper

After some time of work Zookeeper writes a huge transaction log and adding any new event in the log makes fsync to FS very long

Logs:

2023-09-21 06:36:22,085 [myid:] - WARN  [SyncThread:3:o.a.z.s.p.FileTxnLog@394] - fsync-ing the write ahead log in SyncThread:3 took 1832ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide
2023-09-21 06:36:25,120 [myid:] - WARN  [SyncThread:3:o.a.z.s.p.FileTxnLog@394] - fsync-ing the write ahead log in SyncThread:3 took 1072ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide
2023-09-21 06:36:39,506 [myid:] - WARN  [SyncThread:3:o.a.z.s.p.FileTxnLog@394] - fsync-ing the write ahead log in SyncThread:3 took 1393ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide

To prevent size growth we can add two options to Zookeeper

In config file:

autopurge.snapRetainCount=7
autopurge.purgeInterval=24

Or via Env variables when running in Docker

        ZOO_AUTOPURGE_PURGEINTERVAL: "24"
        ZOO_AUTOPURGE_SNAPRETAINCOUNT: "7"

This options will keep only last 7 days in the transaction log

Git: warning: Not a git repository

When you try to update/pull or get a diff for the Git repository, you can get a warning:

warning: Not a git repository. Use --no-index to compare two paths outside a working tree
usage: git diff --no-index [<options>] <path> <path>

It is because the new version of Git forbidden to run git commands in a directory that is not owned by the current user ( CVE-2022-24765 )

To fix, you need to change permission for the directory or set security options to git when running commands in CI/CD

git config --global --add safe.directory $(pwd)

Run CentOS 8 in rescue mode to change root password

In case you forgot the root password or can’t login to the CentOS, we can start OS by editing kernel boot options in Grub

When the system is booting press any key (left, right, top, down) on Grub screen

Select the kernel you need to run and press ‘e’ to open the editor

Read more

Debezium: getaddrinfo() thread failed to start

When building debezium-connect image on Gitlab runners had error

* getaddrinfo() thread failed to start
* Could not resolve host: packages.confluent.io
* Closing connection 0
curl: (6) getaddrinfo() thread failed to start

After running all commands in Container found the problem

Step 4/11 : RUN microdnf upgrade --refresh && microdnf -y install curl && microdnf clean all
 ---> Running in fdbcae0c9647

(microdnf:7): GLib-ERROR **: 11:03:33.774: file ../glib/gthread-posix.c: line 1340 (g_system_thread_new): error 'Operation not permitted' during 'pthread_create'
The command '/bin/sh -c microdnf upgrade --refresh && microdnf -y install curl && microdnf clean all' returned a non-zero code: 133

Bug on Fedora issue tracker https://bugzilla.redhat.com/show_bug.cgi?id=2019901

Fix by running a container with security options

podman run -it --rm --security-opt seccomp=unconfined registry.access.redhat.com/ubi9-beta/ubi-minimal

or add config to Runner

cat /etc/gitlab-runner/config.toml
....
security_opt = ["seccomp:unconfined"]
....

Mongo WiredTiger Concurrency Tickets Available

If it’s not enough and the internal queue grows very fast – change the Wiredtiger limit of concurrent transaction

db.adminCommand( { setParameter: 1,
   wiredTigerConcurrentReadTransactions: xx } )
db.adminCommand( { setParameter: 1,
   wiredTigerConcurrentWriteTransactions: xx } )

Can’t install nvidia/520.56.06 on Ubuntu 22.04

The newer version of the DKMS Nvidia driver won’t install on Ubuntu 22.04.

From logs :

cat /var/lib/dkms/nvidia/520.56.06/build/make.log
....
  cc -Wp,-MMD,/var/lib/dkms/nvidia/520.56.06/build/nvidia/.nv-mmap.o.d -nostdinc -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/compiler-version.h -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Wno-format-security -std=gnu11 -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=none -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mindirect-branch=thunk-extern -mindirect-branch-register -mindirect-branch-cs-prefix -mfunction-return=thunk-extern -fno-jump-tables -mharden-sls=all -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-address-of-packed-member -O2 -fno-allow-store-data-races -Wframe-larger-than=1024 -fstack-protector-strong -Wno-array-bounds -Wimplicit-fallthrough=5 -Wno-main -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -ftrivial-auto-var-init=zero -fzero-call-used-regs=used-gpr -pg -mrecord-mcount -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-maybe-uninitialized -Wno-alloc-size-larger-than -fno-strict-overflow -fno-stack-check -fconserve-stack -Werror=date-time -g -gdwarf-5 -I/var/lib/dkms/nvidia/520.56.06/build/common/inc -I/var/lib/dkms/nvidia/520.56.06/build -Wall -MD -Wno-cast-qual -Wno-error -Wno-format-extra-args -D__KERNEL__ -DMODULE -DNVRM -DNV_VERSION_STRING=\"520.56.06\" -Wno-unused-function -Wuninitialized -fno-strict-aliasing -ffreestanding -mno-red-zone -mcmodel=kernel -DNV_UVM_ENABLE -DNV_SPECTRE_V2=0 -DNV_KERNEL_INTERFACE_LAYER -I/var/lib/dkms/nvidia/520.56.06/build/nvidia -DNVIDIA_UNDEF_LEGACY_BIT_MACROS -UDEBUG -U_DEBUG -DNDEBUG  -fsanitize=bounds -fsanitize=shift -fsanitize=bool -fsanitize=enum  -DMODULE  -DKBUILD_BASENAME='"nv_mmap"' -DKBUILD_MODNAME='"nvidia"' -D__KBUILD_MODNAME=kmod_nvidia -c -o /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-mmap.o /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-mmap.c   ; ./tools/objtool/objtool  --hacks=jump_label  --hacks=noinstr     --retpoline  --rethunk  --sls  --stackval  --static-call  --uaccess   --module  /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-mmap.o
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-pci.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-dmabuf.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-acpi.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-cray.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-i2c.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-dma.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[2]: *** [scripts/Makefile.build:249: /var/lib/dkms/nvidia/520.56.06/build/nvidia/nv-mmap.o] Error 1
make[1]: *** [Makefile:1858: /var/lib/dkms/nvidia/520.56.06/build] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-6.0.0-060000-generic'
make: *** [Makefile:82: modules] Error 2

Error “unrecognized command-line option ‘-ftrivial-auto-var-init=zero’ looks like we need a newer version of GCC. Firstly try to install deps

sudo apt install build-essential
Read more

Clone secret path in Haspicorp Vault

Use an app named ‘safe’

safe copy secret/staging/user secret/prod/user

URL: https://github.com/starkandwayne/safe/tree/v1.6.1

Hetzner VPS: filesystem partitions

Sometimes I need to have separate partitions on VPS in Hetzner.

The simplest way to create a custom disk structure – use Cloud-init script

With commands:

#cloud-config

# we need to disable the initial call to "growpart" as otherwise the first partition would consume
# all space on the dist
#
# The final disk layout is:
# #0    0 - 10GB - ext4 on /
# #1 10GB - 100% - lvm
#
# The reason for this approach is that when using CentOS on Hetzner you cannot use
# the "disk_setup" mechanism from cloud-init, it simply is ignored.

growpart:
  mode: off

runcmd:
  - printf "fix\n" | parted ---pretend-input-tty /dev/sda print        # Fix parted error, first
  - [ parted, "/dev/sda", mkpart, primary, ext2, "10GB", "100%" ]  # create a new partition, starting at 10GB
  - [ parted, "/dev/sda", set, "2", lvm, on ]  # set LVM flag
  - [ growpart, "/dev/sda", "1" ]  # grow first partition to 10GB
  - [ partx, --update, "/dev/sda" ] # reload partition table
  - [ resize2fs, /dev/sda1 ] # resize first partition (/) to 10GB
  - [ pvcreate, "/dev/sda2" ] # create PV on /dev/sda2 (100%-10GB)
  - [ vgcreate, vg1, "/dev/sda2" ] # create VG, adding PV /dev/sda2

repo_update: true
repo_upgrade: all

packages:
  - lvm2  # missing in the Hetzner image

In this example, we have 10Gb for / and empty disk /dev/sdb

Postgresql, count rows in all tables

For count rows in each PostgreSQL table, we need to create a function and run it for all schemas

create function 
cnt_rows(schema text, tablename text) returns integer
as
$body$
declare
  result integer;
  query varchar;
begin
  query := 'SELECT count(1) FROM ' || schema || '.' || tablename;
  execute query into result;
  return result;
end;
$body$
language plpgsql;
Read more

i915 0000:00:02.0: Device initialization failed (-12)

After upgrading the kernel to a newer kernel, OS won’t boot with a stacked message

pci 0000 : 00 : 07.0 : DPC : RP PIO log size 0 is invalid 
xhci_hcd 0000 : 00 : 0d.0 : can't setup : -12 
xhci hcd 0000 : 00 : 0d.0 : init 0000 : 00 : 0d.0 fail , -12 
thunderbolt 0000 : 00 : 0d.2 : failed to determine connection manager , aborting 
xhci hcd 0000 : 00 : 14.0 : can't setup : -12 
xhci_hcd 0000 : 00 : 14.0 : init 0000 : 00 : 14.0 fail , -12 
1801_smbus . 0000 : 00 : 1f.4 : Transaction timeout 
1801_smbus 0000 : 00 : 1f.4 : Failed terminating the transaction 
1801_smbus 0000 : 00 : 1f.4 : SMBus is busy , can't use it ! 
i915 0000 : 00 : 02.0 : [ drm ] * ERROR * Scratch setup failed 
i915 0000 : 00 : 02.0 : Device initialization failed ( -12 )
Read more