运行下面代码,出现了错误
import spark.implicits._
val tmpSeq = JavaConverters.asScalaIteratorConverter(list.iterator).asScala.toSeq
val ds = spark.createDataset(tmpSeq)
Error:(32, 33) Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases.
觉得因为Dataset是强类型,没有指定好类型,于是修改如下面这样,果然错误消失了:
import spark.implicits._
// List 转 Seq
val tmpSeq = JavaConverters.asScalaIteratorConverter(list.iterator).asScala.toSeq
sinaStockStream.close()
// spark.sparkContext.parallelize(tmpSeq);
val personEncoder: Encoder[KLineModel] = Encoders.bean(classOf[KLineModel])
val ds = spark.createDataset(tmpSeq)(personEncoder)