依赖的jar包
来源:9-25 -数据清洗作业运行到YARN上
catcher_1224
2017-08-28
我在打包时候将项目代码和所依赖的第三方jar包分开打成jar文件,所有jar文件都在lib目录下,如图:
我通过如下命令提交总是报找不到第三方依赖jar中的类
./bin/spark-submit --master local[4] \ --class com.catcher92.demo.spark.moocLog.etl.LogFormatJob \ --conf /home/hadoop/spark-sql-imooc-1.0-SNAPSHOT/lib/ \ --driver-lib-path /home/hadoop/spark-sql-imooc-1.0-SNAPSHOT/lib/ \ /home/hadoop/spark-sql-imooc-1.0-SNAPSHOT/lib/spark-sql-imooc-1.0-SNAPSHOT.jar hdfs://hadoop01:9000/data/filterOutput hdfs://hadoop01:9000/data/formatOutput
报错日志如下:
Caused by: java.lang.NoClassDefFoundError: com/alibaba/druid/pool/DruidDataSourceFactory at com.catcher92.demo.spark.moocLog.util.MySqlUtil$.init(MySqlUtil.scala:31) at com.catcher92.demo.spark.moocLog.util.MySqlUtil$.getConnection(MySqlUtil.scala:37) at com.catcher92.demo.spark.moocLog.util.IpUtil$.getCityByIp(IpUtil.scala:19) at com.catcher92.demo.spark.moocLog.entity.MoocLog$.convertLog2MoocLog(MoocLog.scala:30) at com.catcher92.demo.spark.moocLog.etl.LogFormatJob$$anonfun$1.apply(LogFormatJob.scala:21) at com.catcher92.demo.spark.moocLog.etl.LogFormatJob$$anonfun$1.apply(LogFormatJob.scala:21) at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
如果一个一个通过--jars添加会有点慢,有什么办法能把那个目录下的所有jar都添加进去
写回答
1回答
-
你把需要的依赖的jar放到一个文件夹下,然后通过shell读取出来以逗号拼接设置到--jars就行
012017-08-30
相似问题