老师,您好,请教您一下,下述问题出在哪里

来源:3-11 -使用IDE开发pyspark应用程序

喜格米

2020-05-29

老师您好,我是在使用pycharm运行这段代码的时候报错的,麻烦您帮忙看看
from pyspark import SparkConf,SparkContext
conf=SparkConf().setMaster(“local[0]”).setAppName(“My APP”)
sc=SparkContext(conf=conf)
data=[1,2,3,4,5]
distData=sc.parallelize(data)
print(distData.collect())
sc.stop()
图片描述
具体的运行后的信息为,得不出结果
20/05/29 20:25:06 WARN Utils: Your hostname, Gavin-MacBook-Pro.local resolves to a loopback address: 127.0.0.1; using 192.168.3.17 instead (on interface en0)
20/05/29 20:25:06 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
20/05/29 20:25:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
20/05/29 20:25:08 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Asked to run locally with 0 threads
at org.apache.spark.SparkContext.org.org.orgapachesparksparksparkSparkContextKaTeX parse error: Expected group after '_' at position 1305: …, line 130, in _̲_init__ con…createTaskScheduler(SparkContext.scala:2864)
at org.apache.spark.SparkContext.(SparkContext.scala:548)
at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
辛苦老师了,麻烦您帮我看看问题出在哪里

写回答

1回答

Michael_PK

2020-05-29

建议先用上课的版本学习,学完后再用你想要的比如spark3,不同版本的不一定兼容。pyspark的spark3我还真没测试过

0
0

Python3实战Spark大数据分析及调度

使用Python3对Spark应用程序进行开发调优,掌握Azkaban任务调度

1046 学习 · 434 问题

查看课程