Hadoop HA高可用故障转移（hdfs+yarn）

Hadoop HA高可用故障转移（hdfs+yarn） 常用集群搭建脚本

xsync.sh 文件或文件夹同步

xcall.sh   在所有集群节点执行命令

xnode.sh 在非master节点执行命令

将脚本放在/usr/bin/或创建软连接 并授予执行权限

HDFS HA 配置文件

core-site.xml

	
		fs.defaultFS
		hdfs://mycluster
		集群nameservice
	

	
		hadoop.tmp.dir
		/opt/module/tmp/ha/hadoop
		临时文件夹
	

	
		io.file.buffer.size
		131072
		4096
		流文件的缓冲区为4K
	

	
		hadoop.http.staticuser.user
		zakza
		用户
	

	
		ha.zookeeper.quorum
		master:2181,node1:2181,node2:2181
	
	

hdfs-site.xml

	
		dfs.nameservices
		mycluster
		名称服务
	

	
		dfs.ha.namenodes.mycluster
		nn1,nn2,nn3
		namenode 集群
	

	
	
		dfs.namenode.rpc-address.mycluster.nn1
		master:9820
	

	
		dfs.namenode.rpc-address.mycluster.nn2
		node1:9820
	

	
		dfs.namenode.rpc-address.mycluster.nn3
		node2:9820
	
	
	
		dfs.namenode.http-address.mycluster.nn1
		master:9870
	

	
		dfs.namenode.http-address.mycluster.nn2
		node1:9870
	

	
		dfs.namenode.http-address.mycluster.nn3
		node2:9870
	

	
	
		dfs.namenode.shared.edits.dir
		qjournal://master:8485;node1:8485;node2:8485/mycluster
	

	

	
		dfs.client.failover.proxy.provider.mycluster
		org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
	

	
		dfs.ha.fencing.methods
		sshfence
	

	
		dfs.ha.fencing.ssh.private-key-files
		~/.ssh/id_rsa
	

	
		dfs.journalnode.edits.dir
		${hadoop.tmp.dir}/journalnode/
	

	
		dfs.ha.automatic-failover.enabled
		true
	

xsync 分发同步文件

初始化启动 启动QJM集群

hdfs --workers --daemon start

任一节点格式化namenode

hdfs namenode -format

hdfs --daemon start namenode

其他节点执行命令同步信息

xnode "hdfs namenode -bootstrapStandby"

xnode "hdfs --daemon start namenode"

xcall "hdfs –-daemon start datanode"

启动jobhistory（master）

mapred --daemon start historyserver

格式化zk

stop-dfs.sh

hdfs zkfc -formatZK

start-dfs.sh

Yarn高可用配置

yarn-site.xml

	
		yarn.log-aggregation-enable
		true
		日志聚集功能
	

	
		yarn.log.server.url
		http://master:19888/jobhistory/logs
		日志web地址
	

	
	
	
		yarn.nodemanager.aux-services
		mapreduce_shuffle
	

	
	
		yarn.nodemanager.env-whitelist
		JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,HADOOP_MAPRED_HOME
		环境变量继承
	

	
		yarn.log-aggregation.retain-seconds
		604800
		日志聚集功能
	

	
	
		yarn.resourcemanager.ha.enabled
		true
	

	
	
		yarn.resourcemanager.cluster-id
		cluster-yarn1
	

	
		yarn.resourcemanager.ha.rm-ids
		rm1,rm2
	

	
		yarn.resourcemanager.hostname.rm1
		master
	

	
		yarn.resourcemanager.hostname.rm2
		node1
	

	 
	
		yarn.resourcemanager.zk-address
		master:2181,node1:2181,node2:2181
	

	 
	
		yarn.resourcemanager.recovery.enabled
		true
	

	 
	
		yarn.resourcemanager.store.class
		org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
	

xsync分发同步配置文件

start-yarn.sh

查看所有启动服务

xcall jps 

参考

Apache Hadoop 3.3.1 – HDFS High Availability Using the Quorum Journal Manager

Apache Hadoop 3.3.1 – ResourceManager High Availability

配置文件查询参考

core-default.xmlhttps://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml

hdfs-default.xmlhttps://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/hdfs-default.xmlmapred-default.xmlhttps://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xmlyarn-default.xmlhttps://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-common/yarn-default.xml