Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
来源：5-8 提交流量统计案例到YARN上运行
WuCh1k1n
2019-02-06
我在YARN上运行第5章的MapReduce任务时发生了以下的报错：
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#2
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:392)
at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:307)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:366)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198)
请问大家这个问题要怎么解决呢？
3回答
-
纪墨
2019-03-01
补充一下
hdfs-site.xml中hadoop.tmp.dir配的什么目录，在yarn-site.xml的yarn.nodemanager.local-dirs中就配这个目录加上nm-local-dir
比如hdfs-site.xml中内容：
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/export/servers/tmp</value> </property> </configuration>
yarn-site.xml中就配置
<?xml version="1.0"?> <configuration> <property> <name>yarn.nodemanager.local-dirs</name> <value>/export/servers/tmp/nm-local-dir</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
222019-11-30 -
WuCh1k1n
提问者
2019-02-07
# yarn.nodemanager.local-dirs对应的路径必须在hdfs-site.xml中的hadoop.tmp.dir路径下，否则yarn会报错找不到缓存文件
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>hadoop.tmp.dir/nm-local-dir</value>
</property>
20 -
Michael_PK
2019-02-07
给点提示：这个错误是不同版本之间配置文件中的属性值兼容性不够导致的。有些属性必须要写在指定的配置文件中。乳沟我没记错的话，应该是HDFS的属性配置错文件了
00
相似问题