Spark local模式启动报错

来源:3-4 Spark Local模式环境搭建

慕函数2082699

2018-02-16

18/02/15 23:05:48 ERROR SparkContext: Error initializing SparkContext.

java.net.BindException: 无法指定被请求的地址: Service 'sparkDriver' failed after 16 retries (starting from 0)! Consider explicitly setting the appropriate port for the service 'sparkDriver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.

at sun.nio.ch.Net.bind0(Native Method)

at sun.nio.ch.Net.bind(Net.java:444)

at sun.nio.ch.Net.bind(Net.java:436)

at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)

at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127)

at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501)

at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218)

at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)

at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)

at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:965)

at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:210)

at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:353)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)

at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)

at java.lang.Thread.run(Thread.java:744)

java.net.BindException: 无法指定被请求的地址: Service 'sparkDriver' failed after 16 retries (starting from 0)! Consider explicitly setting the appropriate port for the service 'sparkDriver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.

  at sun.nio.ch.Net.bind0(Native Method)

  at sun.nio.ch.Net.bind(Net.java:444)

  at sun.nio.ch.Net.bind(Net.java:436)

  at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)

  at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127)

  at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501)

  at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218)

  at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)

  at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)

  at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:965)

  at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:210)

  at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:353)

  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)

  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)

  at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)

  at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)

  at java.lang.Thread.run(Thread.java:744)

<console>:14: error: not found: value spark

       import spark.implicits._

              ^

<console>:14: error: not found: value spark

       import spark.sql

              ^

Welcome to

      ____              __

     / __/__  ___ _____/ /__

    _\ \/ _ \/ _ `/ __/  '_/

   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0

      /_/

         

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51)

Type in expressions to have them evaluated.

Type :help for more information.


写回答

1回答

Michael_PK

2018-02-16

这个默认的端口被占用了,从4040开始,它会自动加1,但是这个默认一个机器最多只能启动16个。你可以jps看下,是否已经有16个spark submit进程了。你可以调大那个参数即可

0
0

以慕课网日志分析为例 进入大数据Spark SQL的世界

快速转型大数据:Hadoop,Hive,SparkSQL步步为赢

1644 学习 · 1129 问题

查看课程