After some time of work Zookeeper writes a huge transaction log and adding any new event in the log makes fsync to FS very long
Logs:
2023-09-21 06:36:22,085 [myid:] - WARN [SyncThread:3:o.a.z.s.p.FileTxnLog@394] - fsync-ing the write ahead log in SyncThread:3 took 1832ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide
2023-09-21 06:36:25,120 [myid:] - WARN [SyncThread:3:o.a.z.s.p.FileTxnLog@394] - fsync-ing the write ahead log in SyncThread:3 took 1072ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide
2023-09-21 06:36:39,506 [myid:] - WARN [SyncThread:3:o.a.z.s.p.FileTxnLog@394] - fsync-ing the write ahead log in SyncThread:3 took 1393ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide
To prevent size growth we can add two options to Zookeeper
In config file:
autopurge.snapRetainCount=7
autopurge.purgeInterval=24
Or via Env variables when running in Docker
ZOO_AUTOPURGE_PURGEINTERVAL: "24"
ZOO_AUTOPURGE_SNAPRETAINCOUNT: "7"
This options will keep only last 7 days in the transaction log