Spark Kafka Streaming to HBase
HBase Installation Step By Step Guide
Install Java
sudo apt update
sudo apt install default-jre -y
sudo apt install default-jdk -y
ls /usr/lib/jvm/
default-java java-1.8.0-openjdk-amd64 java-8-openjdk-amd64
echo '
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
export PATH=$PATH:$JAVA_HOME/bin
' | tee -a ~/.bashrc
source ~/.bashrc
Install Hadoop
wget http://www-us.apache.org/dist/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz
tar -xzvf hadoop-2.8.5.tar.gz
sudo mv hadoop-2.8.5 /usr/local/hadoop
echo '
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
' | tee -a ~/.bashrc
source ~/.bashrc
hadoop version
Hadoop 2.8.5
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 0b8464d75227fcee2c6e7f2410377b3d53d3d5f8
Compiled by jdu on 2018-09-10T03:32Z
Compiled with protoc 2.5.0
From source with checksum 9942ca5c745417c14e318835f420733
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.8.5.jar
Set JAVA_HOME in hadoop-env.sh
vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
Validate hadoop installation by running the example project:
mkdir ~/input
cp /usr/local/hadoop/etc/hadoop/*.xml ~/input
hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar grep ~/input ~/grep_example 'principal[.]*'
cat ~/grep_example/*
6 principal
1 principal.
The MapReduce task found one occurrence of the word principal followed by a period and six occurrences where it was not.
Hadoop Configurations
vim $HADOOP_HOME/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
vim $HADOOP_HOME/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.replication</name >
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>file:///home/hadoop/hadoopinfra/hdfs/namenode</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>file:///home/hadoop/hadoopinfra/hdfs/datanode</value>
</property>
</configuration>
vim $HADOOP_HOME/etc/hadoop/yarn-site.xml
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
cp $HADOOP_HOME/etc/hadoop/mapred-site.xml.template $HADOOP_HOME/etc/hadoop/mapred-site.xml
Verifying Hadoop installation
cd ~
hdfs namenode -format
start-dfs.sh
start-yarn.sh
curl icanhazip.com
35.227.87.79
Install Hbase
wget http://www-eu.apache.org/dist/hbase/2.1.0/hbase-2.1.0-bin.tar.gz
tar -zxvf hbase-2.1.0-bin.tar.gz
sudo mv hbase-2.1.0 /usr/local/HBase/
echo '
export HBASE_HOME=/usr/local/HBase
export PATH=$PATH:$HBASE_HOME/bin
' | tee -a ~/.bashrc
source ~/.bashrc
Edit JAVA_HOME in hbase-env.sh
vim /usr/local/HBase/conf/hbase-env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
HBase Configurations
vim /usr/local/HBase/conf/hbase-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:8030/hbase</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/hadoop/zookeeper</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
</configuration>
Start HBase
start-hbase.sh
hbase shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/HBase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
Version 2.1.0, re1673bb0bbfea21d6e5dba73e013b09b8b49b89b, Tue Jul 10 17:26:48 CST 2018
Took 0.0066 seconds
hbase(main):001:0>