Flume: java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType


2017-08-09 12:29:09,754 (conf-file-poller-0) [ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:146)] Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType
	at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:235)
	at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:411)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:141)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 12 more


export JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre/
export JAVA_OPTS="$JAVA_OPTS -Dorg.apache.flume.log.rawdata=true -Dorg.apache.flume.log.printconfig=true "

Linux & Unix system


Забавна программа якщо треба перенести якийсь текст на телефон, а під рукою нема ніяких штатних месенджерів

echo "text_string" | qrencode -t UTF8 -o - 
████ ▄▄▄▄▄ █▀▄█▀▄█ ▄▄▄▄▄ ████
████ █   █ █▀▄█▀██ █   █ ████
████ █▄▄▄█ █▀▀ ▄ █ █▄▄▄█ ████
████▄▄▄▄▄▄▄█▄█ ▀ █▄▄▄▄▄▄▄████
████ ▄▄▄ ▀▄▄  ██ ▄█▄▀ ▀▄▀████
█████▀▄▄▄█▄█▀█▀▄█▀▄▀█  ▄█████
████▄▄▄▄█▄▄▄ ▀▄▄▀█ █▀▄█ ▀████
████ ▄▄▄▄▄ █▄ ▄█▄ ██▀▀ ▄█████
████ █   █ █ ▄ █▀▀ ▄▀ ▀▄▄████
████ █▄▄▄█ █ ▄▀▀▄▀▄█▀▀ ██████

Linux & Unix system

SSH client stuck on connect

This is probably an issue with D-Bus and systemd. If the dbus service is restarted for some reason, you will also need to restart systemd-logind
restart systemd-logind service:

systemctl restart systemd-logind

I had this same issue on CentOS 7, because the messagebus was restarted (which is how the D-Bus service is called on CentOS).


Linux bridge: MAC addresses and dynamic ports

Scenario: KVM virtualization host running several bridged guests. The host has a bridge interface br0 that starts out containing only eth0, and other interfaces are dynamically added and removed from the bridge as guests are started and stopped.
The problem is, the host seems to randomly suffer some loss of connectivity (from a few to 30-40 seconds) when some guest is started or stopped. Initially one might think of something related to STP, but it turns out that it is disabled (and even then, ports appearing or disappearing should not affect existing ports).

What happens here is that, when a new guest is started, a tap interface is created and enslaved to the bridge (the tap interface is usually connected to the guest’s own ethernet interface). This interface has a name like vnet5, for example. Minor but important detail, this tap interface has a more or less random MAC address. (Note that we’re talking of the host-side tap interface, not the guest-side virtual interface, whose MAC address is configurable in the guest’s configuration file or on the command line). It is theoretically possible to set the MAC address of this host-side tap interface, but it’s quite involved and basically nobody does that (and also virt-manager has no provision to do that).

Now, by default bridge interfaces in Linux use, for their MAC address, the lowest MAC address among the enslaved interfaces. So if the newly created interface has a lower MAC, the bridge changes its MAC address and uses that of the new interface. When the guest is stopped, the tap interface is removed from the bridge and destroyed, at which point the bridge’s MAC address has to change again. Depending on when and how often guests are started/stopped, the host’s bridge interface may be changing its MAC address quite often.

Of course, this may (and does) cause connectivity problems for traffic to/from the host (fortunately, traffic to/from the guests is not affected), because neighboring computers and network devices suddenly have to update their ARP caches with the new MAC address before they can talk to the host. Depending on the traffic patterns, this may take anything from a few seconds to several tens of seconds.


Mysql: wrong slave_master_info after upgrade to 5.7

After upgrade mysql-server from 5.6 to 5.7.10 something strange going on with multimaster replication.
When you add more then one channel replication tool not see second channel and server hangs on first channel.
This happens because mysql_upgrade tool have a bug. This bug create columns in table slave_master_info in wrong order.

Linux & Unix system

Apache Helix presentation

Video on youtube

Docs: http://helix.apache.org/0.7.1-docs/Tutorial.html
Zookeeper: http://zookeeper.apache.org/doc/r3.3.3/zookeeperStarted.html

Linux & Unix Security system

Add CA certifice from Let’s Encrypt to Java 8

For add cert to Java 8 you need use keytool

cd /tmp/ && \
curl -O https://letsencrypt.org/certs/letsencryptauthorityx1.der && \
sudo $JAVA_HOME/bin/keytool -noprompt -importcert -alias letsencryptauthorityx1 -keystore $JAVA_HOME/jre/lib/security/cacerts -file  letsencryptauthorityx1.der  -storepass changeit && \
curl -O https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.der && \
sudo $JAVA_HOME/bin/keytool -noprompt -importcert -alias lets-encrypt-x3-cross-signed -keystore $JAVA_HOME/jre/lib/security/cacerts -file lets-encrypt-x3-cross-signed.der -storepass changeit
Linux & Unix system

Syncthing: Debian run script

Rename syncthing.txt to syncthing
Put to /etc/init.d/
Change USER and DAEMON vars

 chmod 755 /etc/init.d/syncthing 

To run Syncthing :

sudo service syncthing start

For autostart

sudo update-rc.d syncthing defaults
Linux & Unix system

avconv: simple video converter in Linux

If you want convert video from big size to smartphone size use this Libav tool

apt-get install libav-tools

or Gentoo

USE="mp3 speex threads v4l jpeg2k aac amr cpudetection faac opus vdpau x264 webp  x265 xvid " CPU_FLAGS_X86="mmx mmxext sse sse2 sse3 sse4_1"  emerge -av libav

and command for convert

avconv -i original_video.mkv -b:v 600k -s 576x432 -strict experimental small_size.mp4
Linux & Unix Security system

Path MTU Discovery Black Hole

Эта проблема совсем не нова. Она описана в RFC 2923 в 2000 году. Но тем не менее, продолжает встречаться с завидным упорством у многих провайдеров. А ведь именно провайдер виноват в данной ситуации: не нужно блокировать ICMP тип 3 код 4. Причем слушаться «голоса разума» ( т. е. клиентов, понимающих в чем проблема) они обычно не хотят.

Решение проблемы с PMTU

Не будем звонить в техподдержку, а попробуем решить проблему, исходя из собственных средств.
Разработчики Linux, тоже знающие о ней, предусмотрели специальную опцию в iptables. Цитата из man iptables:

This target allows to alter the MSS value of TCP SYN packets, to control the maximum size for that connection (usually limiting it to your outgoing interface’s MTU minus 40 for IPv4 or 60 for IPv6, respectively). Of course, it can only be used in conjunction with -p tcp. It is only valid in the mangle table. This target is used to overcome criminally braindead ISPs or servers which block “ICMP Fragmentation Needed” or “ICMPv6 Packet Too Big” packets. The symptoms of this problem are that everything works fine from your Linux firewall/router, but machines behind it can never exchange large packets:
1) Web browsers connect, then hang with no data received.
2) Small mail works fine, but large emails hang.
3) ssh works fine, but scp hangs after initial handshaking.
Workaround: activate this option and add a rule to your firewall configuration like:

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
system web

Bind9: balancing by client ip (even/odd)

For some reason i need balancing client’s 50/50 for 2 datacenter’s
But, if client send GET request to datacenter1 (DC1) all following request need redirect to the same DC1. Only if DC1 is down – send to DC2
Simplest way – is share client’s by their ip (odd/even)
So we need two Bind9 services on one server.
First Bind9 will be answered that www.example.org is in DC1, second – in DC2
For example first named.conf:

zone "example.com" {
    type master;
    file "/etc/bind/db.example.com";