spark错误:Unable to find encoder for type stored in a Dataset

运行下面代码,出现了错误

import spark.implicits._
val tmpSeq = JavaConverters.asScalaIteratorConverter(list.iterator).asScala.toSeq
val ds = spark.createDataset(tmpSeq)

Error:(32, 33) Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases.

觉得因为Dataset是强类型,没有指定好类型,于是修改如下面这样,果然错误消失了:

 import spark.implicits._
// List 转 Seq
val tmpSeq = JavaConverters.asScalaIteratorConverter(list.iterator).asScala.toSeq
sinaStockStream.close()
// spark.sparkContext.parallelize(tmpSeq);
val personEncoder: Encoder[KLineModel] = Encoders.bean(classOf[KLineModel])
val ds = spark.createDataset(tmpSeq)(personEncoder)

关注公众号“大模型全栈程序员”回复“小程序”获取1000个小程序打包源码。更多免费资源在http://www.gitweixin.com/?p=2627

发表评论

邮箱地址不会被公开。 必填项已用*标注