シングルモードで Spark クラスターを起動しました。
start-master.sh -h 10.0.0.56
start-slave.sh spark://10.0.0.56:7077
Scala 用の spark-core lib を使用してジョブを正常に実行できます。ジョブ管理に Spark JobServer を使用したいと考えています。マスターホストノードのdockerで開始しました：
docker run -d -p 8090:8090 -e SPARK_MASTER=spark://10.0.0.56:7077 velvia/spark-jobserver:0.6.0
次に、既に追加されたジョブを実行してみます (追加されたことを UI で確認しました)。
curl-d "input.string =somedata" '10.0.0.56:8090/jobs?appName=myjob&classPath=sparkscala.Job'
応答：
{
"status": "ERROR",
"result": {
"message": "Futures timed out after [15 seconds]",
"errorClass": "java.util.concurrent.TimeoutException",
"stack": ["scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)", "scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)", "scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)", "akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoo
lBuilder.scala:169)", "scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640)", "akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:167)", "akka.dispatch.BatchingExecutor$Batch.blockOn(BatchingExecutor.scala:101)", "scala.concurrent.Await$.result(packa
ge.scala:107)", "spark.jobserver.WebApi.spark$jobserver$WebApi$$getJobManagerForContext(WebApi.scala:478)", "spark.jobserver.WebApi$$anonfun$jobRoutes$1$$anonfun$apply$33$$anonfun$apply$34.apply(WebApi.scala:419)", "spark.jobserver.WebApi$$anonfun$jobRoutes$1$$anonfun$apply$33$$anonfun$apply$34.apply(WebApi.scala:412
)", "spray.routing.ApplyConverterInstances$$anon$18$$anonfun$apply$5.apply(ApplyConverterInstances.scala:53)", "spray.routing.ApplyConverterInstances$$anon$18$$anonfun$apply$5.apply(ApplyConverterInstances.scala:52)", "spray.routing.ConjunctionMagnet$$anon$1$$anon$2$$anonfun$happly$1$$anonfun$apply$1.apply(Directive.
scala:38)", "spray.routing.ConjunctionMagnet$$anon$1$$anon$2$$anonfun$happly$1$$anonfun$apply$1.apply(Directive.scala:37)", "spray.routing.directives.BasicDirectives$$anon$2.happly(BasicDirectives.scala:79)", "spray.routing.Directive$$anon$7$$anonfun$happly$4.apply(Directive.scala:86)", "spray.routing.Directive$$anon
$7$$anonfun$happly$4.apply(Directive.scala:86)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly
$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(Bas
icDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives
.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30)", "spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$a
pply$1.apply(BasicDirectives.scala:30)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$a
nonfun$apply$1.apply(BasicDirectives.scala:30)", "spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$an
on$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)", "spray.rout
ing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$m
cVL$sp.apply(AbstractPartialFunction.scala:25)", "spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext$$anon$3.handle(RequestContext.scala:102)", "akka.spray.UnregisteredActorRefBase.$bang(U
nregisteredActorRefBase.scala:72)", "spray.routing.RequestContext.reject(RequestContext.scala:202)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)", "spray.routing.directives.BasicDirectives$$anon$3
$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)", "spray.routing.RouteConcatenation$Rout
eConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)", "spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialF
unction$mcVL$sp.apply(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)", "spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.Requ
estContext$$anon$3.handle(RequestContext.scala:102)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext.reject(RequestContext.scala:202)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)", "spray.routing.directives.RouteDire
ctives$$anon$1.apply(RouteDirectives.scala:34)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$ti
lde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)", "spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)", "scala.runtime.AbstractPartial
Function$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)", "spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)", "
akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext$$anon$3.handle(RequestContext.scala:102)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext.reject(RequestContext.scala:202)", "spray.routing.direct
ives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.a
pply(BasicDirectives.scala:92)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)", "spray.routing.RequestContext$$anon
fun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPar
tialFunction.scala:25)", "spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext.reject(RequestContext.scala:202)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scal
a:35)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.RouteCo
ncatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)", "spray.routing.R
outeConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)", "spray.rou
ting.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)", "spray.routing.directives.OnSuccessFutureMagnet$$anon$2$$anonfun$happly$2$$anonfun$apply$2.apply(FutureDirective
s.scala:81)", "spray.routing.directives.OnSuccessFutureMagnet$$anon$2$$anonfun$happly$2$$anonfun$apply$2.apply(FutureDirectives.scala:79)", "scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)", "akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)", "akka.dispatch.Ba
tchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)", "akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)", "akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)", "scala.concurrent.BlockContext$.withBlockContext(BlockContext.sc
ala:72)", "akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)", "akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)", "akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)", "scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)", "s
cala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)", "scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)", "scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)"]
}
Job Server にログオンします。
[2016-05-05 13:27:00,655] INFO ient.AppClient$ClientActor [] [] - Connecting to master akka.tcp://sparkMaster@10.0.0.56:7077/user/Master...
[2016-05-05 13:27:14,457] ERROR ocalContextSupervisorActor [] [] - Exception after sending Initialize to JobManagerActor
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://JobServer/user/context-supervisor/c7f81d9f-sparkscala.Server#-306023562]] after [15000 ms]
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333)
at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)
at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:745)
Spark バージョン: 1.6.1 Hadoop 2.6 用に
事前構築済み Spark JobServer: 0.6.0
Job Scala バージョン 2.10.6