Щось корисне про IT

After some time of work Zookeeper writes a huge transaction log and adding any new event in the log makes fsync to FS very long

Logs:

2023-09-21 06:36:22,085 [myid:] - WARN  [SyncThread:3:o.a.z.s.p.FileTxnLog@394] - fsync-ing the write ahead log in SyncThread:3 took 1832ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide
2023-09-21 06:36:25,120 [myid:] - WARN  [SyncThread:3:o.a.z.s.p.FileTxnLog@394] - fsync-ing the write ahead log in SyncThread:3 took 1072ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide
2023-09-21 06:36:39,506 [myid:] - WARN  [SyncThread:3:o.a.z.s.p.FileTxnLog@394] - fsync-ing the write ahead log in SyncThread:3 took 1393ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide

To prevent size growth we can add two options to Zookeeper

In config file:

autopurge.snapRetainCount=7
autopurge.purgeInterval=24

Or via Env variables when running in Docker

        ZOO_AUTOPURGE_PURGEINTERVAL: "24"
        ZOO_AUTOPURGE_SNAPRETAINCOUNT: "7"

This options will keep only last 7 days in the transaction log