在windows环境下,IDEA写json文件报错,麻烦帮忙看看,谢谢!
来源:7-5 -操作Parquet文件数据
慕斯8493011
2019-10-16
在windows环境下,IDEA写json文件报错,麻烦帮忙看看,谢谢!
object ParquetApp {
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().appName(“ParquetApp”).
master(“local[2]”).getOrCreate()
val path = "E:/scalaproject/users.parquet"
val userDF = spark.read.format("parquet").load(path)
userDF.printSchema()
userDF.show()
//未指定format,默认为parquet
val userDF3 = spark.read.load(path)
userDF3.show()
val path2 = "E:/scalaproject/jsonout"
val userDF2 = userDF.filter("favorite_color != 'null'").
select("name","favorite_color")
userDF2.show()
userDF2.write.format("json").save(path2)
spark.stop()
}
19/10/16 09:44:18 ERROR Utils: Aborting task
java.lang.NullPointerException
19/10/16 09:44:18 ERROR FileFormatWriter: Job job_20191016094417_0005 aborted.
19/10/16 09:44:18 ERROR Executor: Exception in task 0.0 in stage 5.0 (TID 5)
org.apache.spark.SparkException: Task failed while writing rows
19/10/16 09:44:18 WARN TaskSetManager: Lost task 0.0 in stage 5.0 (TID 5, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows
19/10/16 09:44:18 ERROR TaskSetManager: Task 0 in stage 5.0 failed 1 times; aborting job
19/10/16 09:44:18 ERROR FileFormatWriter: Aborting job null.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 1 times, most recent failure: Lost task 0.0 in stage 5.0 (TID 5, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows
1回答
-
默认就是parquet。你把你的parquet数据读出来看看有没有你的字段吧。还有你本地盘符那么写?确认下是否能读得到
032019-10-17
相似问题