2019年4月 – gitweixin

Spark 4月 25,2019

spark错误:scalac: Error: object FloatRef does not have a member create

运行spark项目，出现下面奇怪的错误：

Error:scalac: Error: object FloatRef does not have a member create
scala.reflect.internal.FatalError: object FloatRef does not have a member create

at scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1166)

at scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1183)

at scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1218)

at scala.tools.nsc.transform.LambdaLift$$anonfun$scala$tools$nsc$transform$LambdaLift$$refCreateMethod$1.apply(LambdaLift.scala:41)

at scala.tools.nsc.transform.LambdaLift$$anonfun$scala$tools$nsc$transform$LambdaLift$$refCreateMethod$1.apply(LambdaLift.scala:41)

at scala.reflect.internal.util.Collections$$anonfun$mapFrom$1.apply(Collections.scala:182)

at scala.reflect.internal.util.Collections$$anonfun$mapFrom$1.apply(Collections.scala:182)

at scala.collection.immutable.List.map(List.scala:274)

at scala.reflect.internal.util.Collections$class.mapFrom(Collections.scala:182)

at scala.reflect.internal.SymbolTable.mapFrom(SymbolTable.scala:16)

at scala.tools.nsc.transform.LambdaLift.scala$tools$nsc$transform$LambdaLift$$refCreateMethod$lzycompute(LambdaLift.scala:41)

这是scala依赖库的问题，把
最后把IDEA依赖的scala版本改成scala2.10.4 就可以。

在IDEA下载scala很慢，给相关scala2.10.4下载地址:

https://scala-lang.org/files/archive/scala-2.10.4.zip

作者 east

bug清单 4月 22,2019

Spark Uri错误：java.lang.IllegalArgumentException: Illegal character in opaque part at index 5

val destinationPath = "file:\\E:\\newcode\\MyFirstProject\\data\\stockresult.txt" ;
FileSystem fs = FileSystem.get(URI.create(path),conf);
writer = new BufferedWriter(new OutputStreamWriter(fs.create(new Path(path))));
if(null!=writer){
    logger.info("[HdfsOperate]>> initialize writer succeed!");
}

出现了下面的错误：

Exception in thread “main” java.lang.IllegalArgumentException: Illegal character in opaque part at index 5: file:\E:\newcode\MyFirstProject\data\stockresult.txt
at java.net.URI.create(URI.java:852)

问题是Uri格式没写对，写成下面这样就对了

val destinationPath = "file:/E:/newcode/MyFirstProject/data/stockresult.txt" ;

作者 east

Spark 4月 19,2019

spark错误：Unable to find encoder for type stored in a Dataset

运行下面代码，出现了错误

import spark.implicits._
val tmpSeq = JavaConverters.asScalaIteratorConverter(list.iterator).asScala.toSeq
val ds = spark.createDataset(tmpSeq)

Error:(32, 33) Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases.

觉得因为Dataset是强类型，没有指定好类型，于是修改如下面这样，果然错误消失了：

 import spark.implicits._
 // List 转 Seq
 val tmpSeq = JavaConverters.asScalaIteratorConverter(list.iterator).asScala.toSeq
 sinaStockStream.close()
// spark.sparkContext.parallelize(tmpSeq);
val personEncoder: Encoder[KLineModel] = Encoders.bean(classOf[KLineModel])
 val ds = spark.createDataset(tmpSeq)(personEncoder)

作者 east

Spark 4月 18,2019

scala获取免费的股票日k线数据

接口的的抓取使用了Scala标准库的Source


class KLineModel {
  var dateStr ="";
  var openPrice = 0f;
  var closePrice = 0f;
  var highPrice = 0f;
  var lowPrice = 0f;

  private var stockInfo :String =""

  def this(stockInfo:String)
  {
    this()
    this.stockInfo=stockInfo /** 根据腾讯的数据接口解析数据 **/
  val stockDetail=stockInfo.split(Array(' ',' ',' ',' ',' '))
    if (stockDetail.length>4){
      this.dateStr=stockDetail(0)
      this.openPrice=stockDetail(1).toFloat
      this.closePrice =stockDetail(2).toFloat
      this.highPrice=stockDetail(3).toFloat
      this.lowPrice =stockDetail(4).toFloat

    }
  }


  override def toString = s"KLineModel($dateStr, $openPrice, $closePrice, $highPrice, $lowPrice)"

import scala.io.Source
object KLineAnalyse {
  def main(args: Array[String]): Unit = {
    println("查询日k线股票 http://data.gtimg.cn/flashdata/hushen/daily/19/sh603000.js")
    val sinaStockStream = Source.fromURL("http://data.gtimg.cn/flashdata/hushen/daily/19/sh603000.js","utf-8")
    val sinaLines=sinaStockStream.getLines
    for(line <- sinaLines) { /** 将每行数据解析成SinaStock对象，并答应对应的股票信息 **/
      if(line.length > 20) {
        println(new KLineModel(line).toString)
      }
      }
      sinaStockStream.close()
      }

}

作者 east

Spark 4月 18,2019

spark中删除文件夹或文件

这个方法能删除HDFS或本地的文件夹或文件，

val spark = SparkSession.builder().appName("USQL").master("local[*]").getOrCreate(); 
deleteOutPutPath(spark.sparkContext,"E:\\newcode\\MyFirstProject\\data\\output\\")

/**
  * 删除文件夹或文件
  * @param sc
  * @param outputPath
  */
def deleteOutPutPath(sc: SparkContext,outputPath: String):Unit={
  val path = new Path(outputPath)
  val hadoopConf = sc.hadoopConfiguration
  val hdfs = org.apache.hadoop.fs.FileSystem.get(hadoopConf)
  if(hdfs.exists(path)){
    hdfs.delete(path,true)
  }
}

如果是删除文件夹的，前面要加下面的话


spark.sparkContext.hadoopConfiguration.setBoolean("mapreduce.input.fileinputformat.input.dir.recursive", true)

作者 east

bug清单 4月 15,2019

在Fragment调用UI控件出现Activity has been destroyed

在Fragment的代码中，

new Handler().postDelayed(new Runnable(){
 CustomDialogFactory fragmentFactory = new CustomDialogFactory(getChildFragmentManager());
。。。
}, 3000);

出现错误

ava.lang.IllegalStateException

Activity has been destroyed

分析到原因，可能是由于定时原因，Activity已经结束，还执行到Fragment定时任务。可以在fragment判断activity是否结束。

new Handler().postDelayed(new Runnable(){
if(getActivity()==null){
    return;
}
if(getActivity().isFinishing()){
    return;
}
 CustomDialogFactory fragmentFactory = new   CustomDialogFactory(getChildFragmentManager());
。。。
}, 3000);

作者 east

月度归档4月 2019