<> == Preface == We work with Apache Hadoop release 1.0.4 from http://hadoop.apache.org/, which is stable version in February 2013.<
> In our setup the ''secondarynamenode'' is running on other machine (ssu04) than ''namenode'' (ssu03). Both namenode and secondarynamenode are also ''datanodes'' and ''tasktracker''. ''Jobtracker'' is same machine than namenode. For data storage we use a RAID 6 disk array mounted under /dcache with 16 TB.<
> If all configuration is done, we start hadoop on the namenode {{{ start-all.sh }}} Next is formatting the namenode {{{ hadoop namenode -format }}} Now we can import some data to ''HDFS'' {{{ hadoop dfs -copyFromLocal /data/billing-2012 /billing-2012 }}} and start a mapreduce job {{{ hadoop jar billing-job.jar -Din=/billing-2012 -Dout=/b12.out }}} == Configuration on all machines in cluster == * We have to add user hadoop in group hadoop on all machines in Cluster: {{{ groupadd -g 790 hadoop }}} {{{ useradd --comment "Hadoop" --shell /bin/zsh -m -r -g 790 -G hadoop --home /usr/local/hadoop hadoop }}} * in zshrc we have to add some variables: {{{ export HADOOP_INSTALL=/usr/local/hadoop/hadoop-1.0.4 export HADOOP_CONF_DIR=$HADOOP_INSTALL/conf export PATH=$PATH:$HADOOP_INSTALL/bin }}} == Configuration of Hadoop framework == === conf/hadoop-env.xml === Following lines are to add in hadoop-env.xml * Setting JAVA_HOME {{{ export JAVA_HOME=/etc/alternatives/jre_oracle }}} * Setting cluster members in plain text file slaves {{{ export HADOOP_SLAVES=$HADOOP_HOME/conf/slaves }}} * Setting path where hadoop conf should be rsync'd {{{ export HADOOP_MASTER=ssu03:/usr/local/hadoop/hadoop-1.0.4 }}} === conf/core-site.xml === We manipulated following ''properties'': {{{ fs.default.name hdfs://ssu03 hadoop.tmp.dir /dcache/hadoop/tmp fs.inmemory.size.mb 200 io.sort.factor 100 io.sort.mb 200 }}} === conf/hdfs-site.xml === We manipulate following ''properties'': {{{ hadoop.tmp.dir /dcache/hadoop/tmp dfs.data.dir /dcache/hadoop/hdfs/data dfs.name.dir /dcache/hadoop/hdfs/name fs.default.name hdfs://ssu03 dfs.hosts $(HADOOP_CONF_DIR)/slaves dfs.replication 3 Default block replication dfs.secondary.http.address ssu04:50090 fs.checkpoint.dir ssu04:/dcache/hadoop/secondary dfs.http.address ssu03:50090 }}} === conf/mapred-site.xml === We manipulate following ''properties'' * we have to attach the ''mapred/system'' directory on all machines: {{{ mkdir -p /dcache/mapred/system }}} {{{ mapred.system.dir /dcache/hadoop/mapred/system mapred.job.tracker ssu03:9001 mapred.hosts ${HADOOP_CONF_DIR}/slaves dfs.hosts ${HADOOP_CONF_DIR}/slaves }}} === conf/master === We have to add host name of our namenode/jobtracker in plain text file master {{{ ssu03 }}} === conf/slaves === We have to add host names of all datanodes/tasktracker in plain text file slaves {{{ ssu01 ssu03 ssu04 ssu05 }}} == ssh settings == We need a passwordless login for user hadoop from all nodes in cluster. For that we have to generate a RSA key pair {{{ ssh-keygen -t rsa -f ~/.ssh/id_rsa }}} and add the public key to ~/.ssh/authorized_keys on all nodes {{{ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys }}} == Ports to open for datanode communication == * 22 * 50010 * 50020 * 9000 * 9001 == Ports to open for cluster administration == * 50030 * 50060 * 50070 * 50075 * 50470 * 50475 * 50090 == node commissioning == Add new node hostnames to conf/slaves == node decommissioning == * create a ''exclude file'' and add hostnames of nodes to be decommissioned <
> Add following properties to hdfs-site before starting DFS {{{ dfs.hosts.exclude $/usr/local/hadoop/hadoop-1.0.4/conf/exclude \ mapred.hosts.exclude $/usr/local/hadoop/hadoop-1.0.4/conf/exclude }}} then start DFS and {{{ hadoop dfsadmin -refreshNodes hadoop mradmin -refreshNodes }}} Go to web interface ''http://ssu03.ifh.de:50090/dfshealth.jsp''. There will be shown a message ''Decommisssioning in Progress''. If all nodes have changed their status to ''Decommisssioned'' we remove the nodes from slaves file.