在做第三章 fileSystem.open 的案例报错Could not obtain block

来源：3-20 HDFS API编程之创建文件并写入内容

我上5年级

2019-02-22

1.环境

JDK版本:JDK8
系统: CentOS Linux release 7.6.1810 腾讯云服务器
部署的hadoop版本:2.6.5

2.简要描述

   从错误堆栈信息来看,是找不到绑定块,但是我登录服务器查看,是能找到
BP-1587748039-10.163.13.96-1550778171456:blk_1073741830_1006 file=/testGetMerge.txt.
也仔细核对过,通过hadoop 命令可以查看到文件的信息.
   经过debug,底层无法建立tcp连接,但是chooseDataNode()方法执行后能拿到节点的地址,唯独建立不了连接..
   由于使用的是云服务器,有对外的公网ip 与内网ip,获得FileSystem时是通过公网ip+8020端口,debug调试时发现chooseDataNode()方法返回的是内网ip+50010端口,随后通过BlockReaderFactory的build()方法里建立连接时失败.
   我觉得是这个内网ip无法访问造成的,请问hadoop中有没有什么配置项可以配置ip.
   我通过ifconfig查看了网络信息,只有内网的ip.

3.错误的堆栈信息如下:

	org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1587748039-10.163.13.96-1550778171456:blk_1073741830_1006 file=/testGetMerge.txt
	
	at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:930)
	at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:609)
	at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:841)
	at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:889)
	at java.io.DataInputStream.read(DataInputStream.java:100)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:78)
	at com.itdoge.bigdata.hadoop.hdfs.HDFSApp.textTest(HDFSApp.java:56)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

写回答

4回答