第4章 Hadoop 2.6 Single Node Cluster 安裝指令

Hadoop Single Node Cluster是只以一台機器,建立hadoop環境,您仍然可以使用hadoop命令,只是無法發揮使用多台機器的威力。 因為只有一台伺服器,所以所有功能都在一台伺服器中,安裝步驟如下:
1 安裝JDK
2 設定 SSH 無密碼登入
3 下載安裝Hadoop
4 設定Hadoop環境變數
5 Hadoop組態檔設定
6 建立與格式化HDFS目錄
7 啟動Hadoop
8 開啟Hadoop Web介面



1.安裝JDK
java -version
sudo apt-get update
sudo apt-get install default-jdk
java -version
update-alternatives --display java

2.設定 SSH 無密碼登入

sudo apt-get install ssh
sudo apt-get install rsync
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
ll /home/hduser/.ssh
ll ~/.ssh
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
3.下載安裝Hadoop

wget http://ftp.twaren.net/Unix/Web/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz
sudo tar -zxvf hadoop-2.6.0.tar.gz
sudo mv hadoop-2.6.0 /usr/local/hadoop
ll /usr/local/hadoop
4.設定Hadoop環境變數

修改~/.bashrc
sudo gedit ~/.bashrc
輸入下列內容
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin 
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
讓~/.bashrc修改生效
source ~/.bashrc
5.修改Hadoop組態設定檔

Step1 改hadoop-env.sh
sudo gedit /usr/local/hadoop/etc/hadoop/hadoop-env.sh
輸入下列內容:
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
Step2 修改core-site.xml
sudo gedit /usr/local/hadoop/etc/hadoop/core-site.xml
在<configuration></configuration>之間,輸入下列內容:
<property>
   <name>fs.default.name</name>
   <value>hdfs://localhost:9000</value>
</property> 
Step3 修改yarn-site.xml
sudo gedit /usr/local/hadoop/etc/hadoop/yarn-site.xml
在<configuration></configuration>之間,輸入下列內容:
<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
Step4 修改mapred-site.xml
sudo cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml
sudo gedit /usr/local/hadoop/etc/hadoop/mapred-site.xml
在<configuration></configuration>之間,輸入下列內容:
<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>
Step5 修改hdfs-site.xml
sudo gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml
在<configuration></configuration>之間,輸入下列內容:
<property>
   <name>dfs.replication</name>
   <value>3</value>
</property>
<property>
   <name>dfs.namenode.name.dir</name>
   <value> file:/usr/local/hadoop/hadoop_data/hdfs/namenode</value>
</property>
<property>
   <name>dfs.datanode.data.dir</name>
   <value> file:/usr/local/hadoop/hadoop_data/hdfs/datanode</value>
 </property>

6.建立與格式化HDFS 目錄
sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/namenode
sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/datanode
sudo chown hduser:hduser -R /usr/local/hadoop
hadoop namenode -format
7.啟動Hadoop
啟動start-dfs.sh,再啟動 start-yarn.sh
start-dfs.sh
start-yarn.sh

啟動全部
start-all.sh
查看目前所執行的行程
jps
8.開啟Hadoop Resource­Manager Web介面
Hadoop Resource­Manager Web介面網址
http://localhost:8088/
9.NameNode HDFS Web介面
開啟HDFS Web UI網址
http://localhost:50070/



以上內容節錄自這本書。很適合入門初學者:
  Python+Spark 2.0+Hadoop機器學習與大數據分析實戰 http://pythonsparkhadoop.blogspot.tw/2016/10/pythonspark-20hadoop.html

《購買本書 限時特價專區》
博客來:http://www.books.com.tw/products/0010730134?loc=P_007_090  

天瓏:https://www.tenlong.com.tw/items/9864341537?item_id=1023658
露天拍賣:http://goods.ruten.com.tw/item/show?21640846068139
蝦皮拍賣:https://goo.gl/IEx13P 



Share on Google Plus

About kevin

This is a short description in the author block about the author. You edit it by entering text in the "Biographical Info" field in the user admin panel.
    Blogger Comment
    Facebook Comment

0 意見:

張貼留言