报错

来源:7-34 项目打包并运行在YARN上

慕九州3016327

2021-02-13

首先报错是:
[hadoop@hadoop000 lib]$ ./streaming.sh
Ivy Default Cache set to: /home/hadoop/.ivy2/cache
The jars for the packages stored in: /home/hadoop/.ivy2/jars
:: loading settings :: url = jar:file:/home/hadoop/app/spark-3.0.0-bin-2.6.0-cdh5.16.2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
org.apache.spark#spark-streaming_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-77e344fc-074a-4eeb-9448-ec0e19644377;1.0
confs: [default]
:: resolution report :: resolve 157ms :: artifacts dl 1ms
:: modules in use:
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 0 | 0 | 0 | 0 || 0 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-77e344fc-074a-4eeb-9448-ec0e19644377
confs: [default]
0 artifacts copied, 0 already retrieved (0kB/10ms)
17:47:17 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Error: Failed to load class com.meng.bigdata.project.stream.StreamingApp.
17:47:20 INFO ShutdownHookManager: Shutdown hook called
17:47:20 INFO ShutdownHookManager: Deleting directory /tmp/spark-7b229ce8-9090-4810-888d-1d062ce11549
[hadoop@hadoop000 lib]$

配置如下:
vi streaming.sh
图片描述
idea代码中的
图片描述
图片描述

我clear掉了所有的缓存,重新package生成的jar包,我看配置什么都没错误,老师求解

写回答

1回答

Michael_PK

2021-02-13

Error: Failed to load class com.meng.bigdata.project.stream.StreamingApp

很明显,要么你的jar里面没有这个class,要么你的jar的路径不对

0
11
慕九州3016327
回复
Michael_PK
跑了下测试的圆周率那个yarn正常,yarn环境是ok的。还有我直接把命令粘到窗口执行,如:[hadoop@hadoop000 bin]$****命令****,就可以成功,就是没有消费出来数据。其次我按照老师你那个脚本形式去弄(命令是一样的),确实也可以跑yarn,ui确实已经显示消费数据了,可是消费了两轮就报了这个错误(如上错误)老师我换一个窗口提问吧,贴个图片还是这个问题
2021-02-16
共11条回复

Spark3实时处理-Streaming+StructuredStreaming实战

实战Spark3实时处理,掌握两套企业级处理方案

340 学习 · 238 问题

查看课程