出现Address already in use: Service ‘SparkUI’ failed after 16 retries!异常

现象描述

提交任务时，出现以下错误。此现象多发生在同时有多个任务提交的情况下。

2014-09-17 15:27:04,597 INFO [main] Successfully started service 'HTTP file server' on port 23503. org.apache.spark.Logging$class.logInfo(Logging.scala:59)
2014-09-17 15:27:04,875 WARN [main] Service 'SparkUI' could not bind on port 23000. Attempting port 23001. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:04,942 WARN [main] Service 'SparkUI' could not bind on port 23001. Attempting port 23002. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,003 WARN [main] Service 'SparkUI' could not bind on port 23002. Attempting port 23003. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,086 WARN [main] Service 'SparkUI' could not bind on port 23003. Attempting port 23004. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,147 WARN [main] Service 'SparkUI' could not bind on port 23004. Attempting port 23005. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,213 WARN [main] Service 'SparkUI' could not bind on port 23005. Attempting port 23006. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,275 WARN [main] Service 'SparkUI' could not bind on port 23006. Attempting port 23007. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,336 WARN [main] Service 'SparkUI' could not bind on port 23007. Attempting port 23008. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,396 WARN [main] Service 'SparkUI' could not bind on port 23008. Attempting port 23009. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,461 WARN [main] Service 'SparkUI' could not bind on port 23009. Attempting port 23010. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,523 WARN [main] Service 'SparkUI' could not bind on port 23010. Attempting port 23011. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,586 WARN [main] Service 'SparkUI' could not bind on port 23011. Attempting port 23012. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,644 WARN [main] Service 'SparkUI' could not bind on port 23012. Attempting port 23013. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,704 WARN [main] Service 'SparkUI' could not bind on port 23013. Attempting port 23014. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,763 WARN [main] Service 'SparkUI' could not bind on port 23014. Attempting port 23015. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,825 WARN [main] Service 'SparkUI' could not bind on port 23015. Attempting port 23016. org.apache.spark.Logging$class.logWarning(Logging.scala:71)
2014-09-17 15:27:05,887 ERROR [main] Failed to bind SparkUI org.apache.spark.Logging$class.logError(Logging.scala:96)
java.net.BindException: Address already in use: Service 'SparkUI' failed after 16 retries!
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.Server.doStart(Server.java:293)
¡­¡­

可能原因

每一个Spark任务都会起一个Driver端口，即SparkUI，默认为23000，如果被占用则随机选取端口重试，默认会重试16次。16次重试都失败后，会放弃任务的运行。

定位思路

使用jps命令查看当前节点上提交的任务数量，如果当前节点的任务数超过了16个，就会造成这样的错误。

处理步骤

使用以下步骤中的任何一个可以解决。

• 初始化SparkConf时，添加conf.set(“spark.port.maxRetries”,“100”)语句
• 使用spark-submit提交任务时，在命令行中添加 –conf spark.port.maxRetries=100
• 在spark-defaults.conf中添加spark.port.maxRetries 100

可以将100替换为任何想要的数字，数字越大，允许同时运行的任务越多。