IDEA 无法访问远程虚拟机的datanode拿到数据

来源：4-16 功能开发之完成ETL数据到HBase落地的全过程

weixin_慕妹8043461

2020-06-06

pk哥你好：
我遇到的问题是，当我想用IEDA访问云端的HDFS，得到如下的错误：

0/06/05 16:59:14 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, ANY, 7978 bytes)
20/06/05 16:59:14 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
20/06/05 16:59:14 INFO rdd.WholeTextFileRDD: Input split: Paths:/test/test-access.log:0+2715
20/06/05 17:00:14 WARN hdfs.BlockReaderFactory: I/O error constructing remote block reader.
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/172.31.69.xxx:50010]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3590)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:849)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:764)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:377)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:666)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:904)

网上搜索之后发现，是因为虚拟机和IDEA的主机之间通过外网ip传递信息，但是当想要或许HDFS中datanode的数据时，namenode传递给IDEA主机上面的是datanode的内网ip，导致无法访问，我做的尝试：
我根据一些帖子的指导做了相应的配置改变，hdfs-site.xml

<property>
        <name>dfs.client.use.datanode.hostname</name>
        <value>true</value>
</property>

<property>
        <name>dfs.datanode.use.datanode.hostname</name>
        <value>true</value>
</property>

类似于虚拟机上的HDSF用域名传递而非ip传递给主机上的IDEA，并且主机也配置好了映射：
18.207.78.xxx hadoop000 //(公共ip，idea可访问)

虚拟机配置映射：
172.31.69.xxx hadoop000 //（内网ip）

因为是伪分布，按照你的视频，datanode的 hostname 也是叫hadoop000.

运行结果还是报同样的错，IDEA还是尝试解析内网ip导致超时，希望您能帮忙看看。

写回答

2回答

weixin_慕妹8043461

提问者

2020-06-07

更新报错：

20/06/07 10:33:14 INFO datasources.FileScanRDD: Reading File path: hdfs://hadoop000:8020/test/drivers.csv, range: 0-1997, partition values: [empty row]

20/06/07 10:33:14 INFO codegen.CodeGenerator: Code generated in 7.95912 ms

20/06/07 10:34:15 WARN hdfs.BlockReaderFactory: I/O error constructing remote block reader.

org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/172.31.69.197:50010]（还是能接收到内网ip并且尝试链接的）

at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)

at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3590)

at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:849)

at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:764)

at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:377)

at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:666)

at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:904)

at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:963)

at java.io.DataInputStream.read(DataInputStream.java:149)

at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)

at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)

at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)

太长省略一些。。。。

0/06/07 10:34:15 WARN hdfs.DFSClient: Failed to connect to /172.31.69.197:50010 for block BP-536449180-172.31.69.197-1591536838863:blk_1073741825_1001, add to deadNodes and continue.

省略一些。。。。。。。。。。。。。。。

20/06/07 10:34:15 INFO hdfs.DFSClient: Could not obtain BP-536449180-172.31.69.197-1591536838863:blk_1073741825_1001 from any node: No live nodes contain current block Block locations: DatanodeInfoWithStorage[172.31.69.197:50010,DS-84a95ba2-e53a-41b2-8224-0d38ad969b24,DISK] Dead nodes: DatanodeInfoWithStorage[172.31.69.197:50010,DS-84a95ba2-e53a-41b2-8224-0d38ad969b24,DISK]. Will get new block locations from namenode and retry...

20/06/07 10:34:15 WARN hdfs.DFSClient: DFS chooseDataNode: got # 1 IOException, will wait for 892.3526112545212 msec.

Michael_PK

云主机内部配置的是内网ip，外面访问过去是公网ip

2020-06-08

共1条回复

Michael_PK

2020-06-06

你是云主机那，那你的代码需要设置dfs.client.use.datanode.hostname是true设置到Configuration中

weixin_慕妹8043461

Michael_PK

pk哥抱歉打扰你了，之前测试是读取csv文件，这次用您提供的log做测试，添加了参数icode，就还是会报之前的错误，同样在本地运行没问题，jar包什么都按照你说的替换过了： package com.imooc.bigdata.spark import org.apache.spark.sql.SparkSession object Testapp { def main(args: Array[String]): Unit = { val day = "20190131" val input = "hdfs://hadoop000:8020/test/*" val spark = SparkSession.builder().config("dfs.client.use.datanode.hostname","true").appName("Testapp").master("local[2]").getOrCreate() System.setProperty("icode", "3435DDCB1E3F98B5") val logdf = spark.read.format("com.imooc.bigdata.spark.pk") .option("path", input) .load() logdf.show() spark.stop()

2020-06-08

共8条回复

Spark进阶大数据离线与实时项目实战

大数据生态圈实用框架(Spark/Hbase/Redis/Hadoop)整合应用及调优

700 学习 · 190 问题

查看课程

相似问题

压缩的数据是不是有问题啊，程序报空指针异常

回答 1

datanode频繁出现dead状态

回答 1

spark sql无法读取hbase数据

回答 1

大型资料里怎么没有安装环境啊？

回答 4

关于虚拟机设置机器名

回答 1

打开慕课网App查看更多内容