执行Spark SQL语句时，出现joinedRow.isNullAt的空指针异常
现象描述
在执行Spark
SQL语句时，出现“joinedRow.isNullAt”的空指针异常，异常信息如下所示。
6/09/08 11:04:11 WARN TaskSetManager: Lost task 1.0 in stage 7.0 (TID 10, vm1, 1): java.lang.NullPointerException
at org.apache.spark.sql.catalyst.expressions.JoinedRow.isNullAt(JoinedRow.scala:70)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificMutableProjection.apply(Unknown Source)
at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator$$anonfun$generateProcessRow$1.apply(TungstenAggregationIterator.scala:194)
at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator$$anonfun$generateProcessRow$1.apply(TungstenAggregationIterator.scala:192)
at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.processInputs(TungstenAggregationIterator.scala:372)
at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.start(TungstenAggregationIterator.scala:626)
at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1.org$apache$spark$sql$execution$aggregate$TungstenAggregate$$anonfun$$executePartition$1(TungstenAggregate.scala:135)
at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$3.apply(TungstenAggregate.scala:144)
at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$3.apply(TungstenAggregate.scala:144)
at org.apache.spark.rdd.MapPartitionsWithPreparationRDD.compute(MapPartitionsWithPreparationRDD.scala:64)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:334)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:267)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:334)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:267)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:75)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:42)
at org.apache.spark.scheduler.Task.run(Task.scala:90)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
可能原因
由如下日志信息可知，该错误是由于内存不足，导致buffer在申请内存时申请失败返回为null，对null进行操作就返回了空指针异常。
当集群中内存相关的关键配置项的值设置的比较小时，例如设置为如下所示的值：
spark.executor.cores
= 8
spark.executor.memory
= 512M
spark.buffer.pageSize
= 16M
此时，执行任务会出现内存申请失败返回null的异常，关键日志如下：
6/09/08 11:04:11 WARN TaskSetManager: Lost task 1.0 in stage 7.0 (TID 10, vm1, 1): java.lang.NullPointerException
at org.apache.spark.sql.catalyst.expressions.JoinedRow.isNullAt(JoinedRow.scala:70)
定位思路
在使用Spark
SQL时，需要满足如下条件：
spark.executor.memory
* spark.shuffle.memoryFraction *spark.shuffle.safetyFraction / (num *
spark.executor.cores) > spark.buffer.pageSize
“spark.shuffle.memoryFraction”默认值为“0.2”。“spark.shuffle.safetyFraction”默认值为“0.8”。“spark.buffer.pageSize”默认值为“16M”。
常数num的经验取值为8，根据不同的SQL语句取值不同，每个task最多可以去申请16次pageSize，所以num的最大值为16。将公式中的参数num设置为16时，即可满足Spark
SQL出现问题的所有场景。但通常情况下8即能满足绝大多数的场景要求。
处理步骤
根据executor日志提示信息，您可以通过调整如下两个参数解决此问题。在客户端的“spark-defaults.conf”配置文件中调整如下参数。
- spark.executor.memory：增加executor的内存，即根据实际业务量，适当增大“spark.executor.memory”的参数值。需满足公式：spark.executor.memory
> spark.buffer.pageSize * (num * spark.executor.cores) /
spark.shuffle.memoryFraction / spark.shuffle.safetyFraction
- spark.executor.cores：减小executor的核数，即减小executor-cores的参数值。需满足公式：spark.executor.cores
< spark.executor.memory / spark.buffer.pageSize / num *
spark.shuffle.memoryFraction * spark.shuffle.memoryFraction。
在调整这两个参数时，需满足spark.executor.memory
* spark.shuffle.memoryFraction *spark.shuffle.safetyFraction / (num *
spark.executor.cores) >
spark.buffer.pageSize公式，在内存充足的情况下，建议直接将常数num设置为16，可解决所有场景遇到的内存问题。