在windows环境下,IDEA写json文件报错,麻烦帮忙看看,谢谢!

来源:7-5 -操作Parquet文件数据

慕斯8493011

2019-10-16

在windows环境下,IDEA写json文件报错,麻烦帮忙看看,谢谢!
object ParquetApp {
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().appName(“ParquetApp”).
master(“local[2]”).getOrCreate()

val path = "E:/scalaproject/users.parquet"
val userDF = spark.read.format("parquet").load(path)
userDF.printSchema()
userDF.show()

//未指定format,默认为parquet
val userDF3 = spark.read.load(path)
userDF3.show()

val path2 = "E:/scalaproject/jsonout"
val userDF2 = userDF.filter("favorite_color != 'null'").
  select("name","favorite_color")
userDF2.show()
userDF2.write.format("json").save(path2)

spark.stop()

}

19/10/16 09:44:18 ERROR Utils: Aborting task
java.lang.NullPointerException
19/10/16 09:44:18 ERROR FileFormatWriter: Job job_20191016094417_0005 aborted.
19/10/16 09:44:18 ERROR Executor: Exception in task 0.0 in stage 5.0 (TID 5)
org.apache.spark.SparkException: Task failed while writing rows
19/10/16 09:44:18 WARN TaskSetManager: Lost task 0.0 in stage 5.0 (TID 5, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows
19/10/16 09:44:18 ERROR TaskSetManager: Task 0 in stage 5.0 failed 1 times; aborting job
19/10/16 09:44:18 ERROR FileFormatWriter: Aborting job null.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 1 times, most recent failure: Lost task 0.0 in stage 5.0 (TID 5, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows

写回答

1回答

Michael_PK

2019-10-16

默认就是parquet。你把你的parquet数据读出来看看有没有你的字段吧。还有你本地盘符那么写?确认下是否能读得到

0
3
慕斯8493011
非常感谢!
2019-10-17
共3条回复

以慕课网日志分析为例 进入大数据Spark SQL的世界

快速转型大数据:Hadoop,Hive,SparkSQL步步为赢

1644 学习 · 1129 问题

查看课程