rdd操作过程中的数据转换
来源:6-6 -DataFrame与RDD互操作方式一
慕九州9163395
2017-09-20
scala> val jsonRDD = sc.textFile("file:///home/hadoop/data/people.json") jsonRDD: org.apache.spark.rdd.RDD[String] = file:///home/hadoop/data/people.json MapPartitionsRDD[1] at textFile at <console>:24 scala> jsonRDD.collect() res0: Array[String] = Array({"name":"Michael"}, {"name":"Andy", "age":30}, {"name":"Justin", "age":19}) scala> import scala.util.parsing.json.JSON import scala.util.parsing.json.JSON scala> val mapRDD = jsonRDD.flatMap(x=>JSON.parseFull(x)) mapRDD: org.apache.spark.rdd.RDD[Any] = MapPartitionsRDD[2] at flatMap at <console>:30 scala> mapRDD.collect() res1: Array[Any] = Array(Map(name -> Michael), Map(name -> Andy, age -> 30.0), Map(name -> Justin, age -> 19.0)) scala> mapRDD.map(x=>(x.get("name"),1)) <console>:33: error: value get is not a member of Any mapRDD.map(x=>(x.get("name"),1)) ^
报value get is not a member of Any的错误,但是怎么操作没搞出来。
写回答
1回答
-
你这是rdd,里面没有get方法的,你弄混了,应该是sparksql里面的
032018-02-09
相似问题