两种方式编译spark源码均抛出异常
来源:3-3 补录:Spark源码编译中的坑
你猜一下
2018-02-01
使用mvn直接编译的方式,
./build/mvn -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0 -DskipTests clean package
提示的错误如下:
[... spark-2.1.0]$ ./build/mvn -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0 -DskipTests clean package
Using `mvn` from path: /home/.../spark/source/spark-2.1.0/build/apache-maven-3.3.9/bin/mvn
[INFO] Scanning for projects...
Downloading: https://repo1.maven.org/maven2/org/apache/apache/14/apache-14.pom
Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/apache/14/apache-14.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[WARNING] 'parent.relativePath' of POM org.apache.spark:spark-parent_2.10:2.1.0 (/home/.../spark/source/spark-2.1.0/pom.xml) points at org.apache.spark:spark-parent_2.10 instead of org.apache:apache, please verify your project structure @ line 22, column 11
[FATAL] Non-resolvable parent POM for org.apache.spark:spark-parent_2.10:2.1.0: Could not transfer artifact org.apache:apache:pom:14 from/to central (https://repo1.maven.org/maven2): java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty and 'parent.relativePath' points at wrong local POM @ line 22, column 11
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.apache.spark:spark-parent_2.10:2.1.0 (/home/.../spark/source/spark-2.1.0/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM for org.apache.spark:spark-parent_2.10:2.1.0: Could not transfer artifact org.apache:apache:pom:14 from/to central (https://repo1.maven.org/maven2): java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty and 'parent.relativePath' points at wrong local POM @ line 22, column 11 -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
使用make-distribution.sh工具编译,
./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0
提示的错误如下:
[... spark-2.1.0]$ ./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0
+++ dirname ./dev/make-distribution.sh
++ cd ./dev/..
++ pwd
+ SPARK_HOME=/home/.../spark/source/spark-2.1.0
+ DISTDIR=/home/.../spark/source/spark-2.1.0/dist
+ MAKE_TGZ=false
+ MAKE_PIP=false
+ MAKE_R=false
+ NAME=none
+ MVN=/home/.../spark/source/spark-2.1.0/build/mvn
+ (( 8 ))
+ case $1 in
+ NAME=2.6.0-cdh5.7.0
+ shift
+ shift
+ (( 6 ))
+ case $1 in
+ MAKE_TGZ=true
+ shift
+ (( 5 ))
+ case $1 in
+ break
+ '[' -z /home/.../spark/app/jdk-1.7-7u60 ']'
+ '[' -z /home/.../spark/app/jdk-1.7-7u60 ']'
++ command -v git
+ '[' /home/.../.jumbo/bin/git ']'
++ git rev-parse --short HEAD
++ :
+ GITREV=
+ '[' '!' -z '' ']'
+ unset GITREV
++ command -v /home/.../spark/source/spark-2.1.0/build/mvn
+ '[' '!' /home/.../spark/source/spark-2.1.0/build/mvn ']'
++ /home/.../spark/source/spark-2.1.0/build/mvn help:evaluate -Dexpression=project.version -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0
++ grep -v INFO
++ tail -n 1
+ VERSION='[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException'
连接http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException打开找不到page
2回答
-
Michael_PK
2018-02-01
重新换台机器试试,看不出啥问题
032018-02-07 -
Michael_PK
2018-02-01
改过pom没
032018-02-01
相似问题