Cannot run program "python3.6": error=2, 没有那个文件或目录

来源:5-4 -standalone模式spark-submit运行

慕粉1926212165

2021-10-09

环境变化:
export JAVA_HOME=/home/hadoop/app/jdk1.8.0_91
export PATH=JAVAHOME/bin:JAVA_HOME/bin:JAVAHOME/bin:PATH

export SCALA_HOME=/home/hadoop/app/scala-2.11.8
export PATH=SCALAHOME/bin:SCALA_HOME/bin:SCALAHOME/bin:PATH

export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export PATH=HADOOPHOME/bin:HADOOP_HOME/bin:HADOOPHOME/bin:PATH

export MAVEN_HOME=/home/hadoop/app/apache-maven-3.3.9
export PATH=MAVENHOME/bin:MAVEN_HOME/bin:MAVENHOME/bin:PATH

export PATH=/home/hadoop/app/python3/bin:/usr/bin/python:$PATH

export PYSPARK_PYTHON=python3.6

export SPARK_HOME=/home/hadoop/app/spark-2.3.0-bin-2.6.0-cdh5.7.0
export PATH=SPARKHOME/bin:SPARK_HOME/bin:SPARKHOME/bin:PATH

手工运行python可以
但用standalone运行报错:
Caused by: java.io.IOException: Cannot run program “python3.6”: error=2, 没有那个文件或目录
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:168)
at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:94)
at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:70)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)
at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:86)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.ExecutorTaskRunner.run(Executor.scala:345)atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)atjava.util.concurrent.ThreadPoolExecutorTaskRunner.run(Executor.scala:345) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutorTaskRunner.run(Executor.scala:345)atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)atjava.util.concurrent.ThreadPoolExecutorWorker.run(ThreadPoolExecutor.java:617)
… 1 more
Caused by: java.io.IOException: error=2, 没有那个文件或目录
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:248)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
… 14 more
不知道哪里还需要配置?

写回答

1回答

Michael_PK

2021-10-09

python3.6的有加在系统环境变量中吗

0
2
慕粉1926212165
将 export PYSPARK_PYTHON=python3.6 改成 export PYSPARK_PYTHON=python
2021-10-09
共2条回复

Python3实战Spark大数据分析及调度

使用Python3对Spark应用程序进行开发调优,掌握Azkaban任务调度

1046 学习 · 434 问题

查看课程