spark-2.1.0的编译问题咨询,非常感谢

来源:3-2 -Spark源码编译

慕神816625

2018-06-03

[hadoop@hadoop001 spark-2.1.0]$ ./build/mvn -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0 -DskipTests clean package

exec: curl --progress-bar -L https://downloads.typesafe.com/zinc/0.3.9/zinc-0.3.9.tgz

######################################################################## 100.0%

Using `mvn` from path: /home/hadoop/source/spark-2.1.0/build/apache-maven-3.3.9/bin/mvn

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0

[INFO] ------------------------------------------------------------------------

[INFO] Building Spark Project Launcher 2.1.0

[INFO] ------------------------------------------------------------------------

[WARNING] The POM for org.apache.hadoop:hadoop-client:jar:2.6.0-cdh5.7.0 is missing, no dependency information available

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO] 

[INFO] Spark Project Parent POM ........................... SUCCESS [  5.356 s]

[INFO] Spark Project Tags ................................. SUCCESS [ 19.086 s]

[INFO] Spark Project Sketch ............................... SUCCESS [  6.654 s]

[INFO] Spark Project Networking ........................... SUCCESS [ 13.416 s]

[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  9.109 s]

[INFO] Spark Project Unsafe ............................... SUCCESS [ 11.544 s]

[INFO] Spark Project Launcher ............................. FAILURE [  0.034 s]

[INFO] Spark Project Core ................................. SKIPPED

[INFO] Spark Project ML Local Library ..................... SKIPPED

[INFO] Spark Project GraphX ............................... SKIPPED

[INFO] Spark Project Streaming ............................ SKIPPED

[INFO] Spark Project Catalyst ............................. SKIPPED

[INFO] Spark Project SQL .................................. SKIPPED

[INFO] Spark Project ML Library ........................... SKIPPED

[INFO] Spark Project Tools ................................ SKIPPED

[INFO] Spark Project Hive ................................. SKIPPED

[INFO] Spark Project REPL ................................. SKIPPED

[INFO] Spark Project YARN Shuffle Service ................. SKIPPED

[INFO] Spark Project YARN ................................. SKIPPED

[INFO] Spark Project Hive Thrift Server ................... SKIPPED

[INFO] Spark Project Assembly ............................. SKIPPED

[INFO] Spark Project External Flume Sink .................. SKIPPED

[INFO] Spark Project External Flume ....................... SKIPPED

[INFO] Spark Project External Flume Assembly .............. SKIPPED

[INFO] Spark Integration for Kafka 0.8 .................... SKIPPED

[INFO] Spark Project Examples ............................. SKIPPED

[INFO] Spark Project External Kafka Assembly .............. SKIPPED

[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED

[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED

[INFO] Kafka 0.10 Source for Structured Streaming ......... SKIPPED

[INFO] Spark Project Java 8 Tests ......................... SKIPPED

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 01:06 min

[INFO] Finished at: 2018-06-03T17:45:23+08:00

[INFO] Final Memory: 46M/365M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal on project spark-launcher_2.11: Could not resolve dependencies for project org.apache.spark:spark-launcher_2.11:jar:2.1.0: Failure to find org.apache.hadoop:hadoop-client:jar:2.6.0-cdh5.7.0 in https://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]

[ERROR] 

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR] 

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

[ERROR] 

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :spark-launcher_2.11

以上spark编译没有成功,报错了,是什么原因呢,谢谢

写回答

2回答

Michael_PK

2018-06-03

按照视频里面来,你这需要在pom中添加cdh的仓库

0
1
慕神816625
老师,您好,非常感谢,后来找到了,谢谢老师
2018-06-03
共1条回复

慕神816625

提问者

2018-06-03

执行mvn的时候,是用自带的mvn命令,还是用apache-maven-3.5.3或者apache-maven-3.3.9再配置好环境变量后,即,$MAVEN_HOME后再执行mvn呢,非常感谢老师

0
0

以慕课网日志分析为例 进入大数据Spark SQL的世界

快速转型大数据:Hadoop,Hive,SparkSQL步步为赢

1644 学习 · 1129 问题

查看课程