两种方式编译spark源码均抛出异常

来源:3-3 补录:Spark源码编译中的坑

你猜一下

2018-02-01

使用mvn直接编译的方式,

./build/mvn -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0 -DskipTests clean package

提示的错误如下:

[... spark-2.1.0]$ ./build/mvn -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0 -DskipTests clean package

Using `mvn` from path: /home/.../spark/source/spark-2.1.0/build/apache-maven-3.3.9/bin/mvn

[INFO] Scanning for projects...

Downloading: https://repo1.maven.org/maven2/org/apache/apache/14/apache-14.pom

Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/apache/14/apache-14.pom

[ERROR] [ERROR] Some problems were encountered while processing the POMs:

[WARNING] 'parent.relativePath' of POM org.apache.spark:spark-parent_2.10:2.1.0 (/home/.../spark/source/spark-2.1.0/pom.xml) points at org.apache.spark:spark-parent_2.10 instead of org.apache:apache, please verify your project structure @ line 22, column 11

[FATAL] Non-resolvable parent POM for org.apache.spark:spark-parent_2.10:2.1.0: Could not transfer artifact org.apache:apache:pom:14 from/to central (https://repo1.maven.org/maven2): java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty and 'parent.relativePath' points at wrong local POM @ line 22, column 11

 @

[ERROR] The build could not read 1 project -> [Help 1]

[ERROR]

[ERROR]   The project org.apache.spark:spark-parent_2.10:2.1.0 (/home/.../spark/source/spark-2.1.0/pom.xml) has 1 error

[ERROR]     Non-resolvable parent POM for org.apache.spark:spark-parent_2.10:2.1.0: Could not transfer artifact org.apache:apache:pom:14 from/to central (https://repo1.maven.org/maven2): java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty and 'parent.relativePath' points at wrong local POM @ line 22, column 11 -> [Help 2]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException

[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

使用make-distribution.sh工具编译,

./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz  -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0

提示的错误如下:

[... spark-2.1.0]$ ./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz  -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0

+++ dirname ./dev/make-distribution.sh

++ cd ./dev/..

++ pwd

+ SPARK_HOME=/home/.../spark/source/spark-2.1.0

+ DISTDIR=/home/.../spark/source/spark-2.1.0/dist

+ MAKE_TGZ=false

+ MAKE_PIP=false

+ MAKE_R=false

+ NAME=none

+ MVN=/home/.../spark/source/spark-2.1.0/build/mvn

+ ((  8  ))

+ case $1 in

+ NAME=2.6.0-cdh5.7.0

+ shift

+ shift

+ ((  6  ))

+ case $1 in

+ MAKE_TGZ=true

+ shift

+ ((  5  ))

+ case $1 in

+ break

+ '[' -z /home/.../spark/app/jdk-1.7-7u60 ']'

+ '[' -z /home/.../spark/app/jdk-1.7-7u60 ']'

++ command -v git

+ '[' /home/.../.jumbo/bin/git ']'

++ git rev-parse --short HEAD

++ :

+ GITREV=

+ '[' '!' -z '' ']'

+ unset GITREV

++ command -v /home/.../spark/source/spark-2.1.0/build/mvn

+ '[' '!' /home/.../spark/source/spark-2.1.0/build/mvn ']'

++ /home/.../spark/source/spark-2.1.0/build/mvn help:evaluate -Dexpression=project.version -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0

++ grep -v INFO

++ tail -n 1

+ VERSION='[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException'

连接http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException打开找不到page

写回答

2回答

Michael_PK

2018-02-01

重新换台机器试试,看不出啥问题

0
3
Michael_PK
回复
你猜一下
哎,好几个小伙伴了,换机器就行,也不知道是什么原因
2018-02-07
共3条回复

Michael_PK

2018-02-01

改过pom没

0
3
你猜一下
回复
Michael_PK
一致的,只不过我的用的压缩包是直接从spark官网上下载的 central Maven Repository https://repo1.maven.org/maven2 true false cloudera https://repository.cloudera.com/artifactory/cloudera-repos/
2018-02-01
共3条回复

以慕课网日志分析为例 进入大数据Spark SQL的世界

快速转型大数据:Hadoop,Hive,SparkSQL步步为赢

1644 学习 · 1129 问题

查看课程