5

log4j プロパティ用に持っている spark-submit コマンドを使用して、次のように Spark-submit を呼び出しています。

/opt/spark-1.6.2-bin-hadoop2.6/bin/spark-submit \
--driver-java-options \
"-Dlog4j.configuration=file:/home/test_api/log4j-driver.properties\ --class Test testing.jar

--driver-java-optionscurl (Apache Spark の Hidden REST API) を介してジョブを送信するにはどうすればよいですか?

私はこれを試しました：

curl -X POST http://host-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
"action" : "CreateSubmissionRequest",
"appArgs" : [ "" ],
"appResource" : "hdfs://host-ip:9000/test/testing.jar",
"clientSparkVersion" : "1.6.2",
"environmentVariables" : {
"SPARK_ENV_LOADED" : "1"
},
"mainClass" : "Test",
"spark.driver.extraJavaOptions" : "-Dlog4j.configuration=file:/home/test_api/log4j-driver.properties",
"sparkProperties" : {
"spark.jars" : "hdfs://host-ip:9000/test/testing.jar",
"spark.app.name" : "Test",
"spark.eventLog.enabled": "true",
"spark.eventLog.dir": "hdfs://host-ip:9000/test/spark-events",
"spark.submit.deployMode" : "cluster",
"spark.master" : "spark://host-ip:7077"
}
}'

ジョブが正常に送信され、応答が返されましたが、uknownField が 1 つありました:

{
  "action" : "CreateSubmissionResponse",
  "message" : "Driver successfully submitted as driver-20160810210057-0091",
  "serverSparkVersion" : "1.6.2",
  "submissionId" : "driver-20160810210057-0091",
  "success" : true,
  "unknownFields" : [ "spark.driver.extraJavaOptions" ]
}

"unknownFields" : [ "spark.driver.extraJavaOptions" ]

私も次のように試しdriverExtraJavaOptionsました：

curl -X POST http://host-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
"action" : "CreateSubmissionRequest",
"appArgs" : [ "" ],
"appResource" : "hdfs://host-ip:9000/test/testing.jar",
"clientSparkVersion" : "1.6.2",
"environmentVariables" : {
"SPARK_ENV_LOADED" : "1"
},
"mainClass" : "Test",
"driverExtraJavaOptions" : "-Dlog4j.configuration=file:/home/test_api/log4j-driver.properties",
"sparkProperties" : {
"spark.jars" : "hdfs://host-ip:9000/test/testing.jar",
"spark.app.name" : "Test",
"spark.eventLog.enabled": "true",
"spark.eventLog.dir": "hdfs://host-ip:9000/test/spark-events",
"spark.submit.deployMode" : "cluster",
"spark.master" : "spark://host-ip:7077"
}
}'

しかし、同様の応答を得ました：

{
  "action" : "CreateSubmissionResponse",
  "message" : "Driver successfully submitted as driver-20160810211432-0094",
  "serverSparkVersion" : "1.6.2",
  "submissionId" : "driver-20160810211432-0094",
  "success" : true,
  "unknownFields" : [ "driverExtraJavaOptions" ]
}

どうしてこれなの？spark-submit.scala
を見て、 Spark REST APIを参照しました

4

1 に答える 1

2

Dlog4j.configuration=file:/// (/// ローカル ファイルのパス) を配置し、sparkProperties 内に spark.driver.extraJavaOptions を配置して動作するようになりました。

curl -X POST http://host-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
"action" : "CreateSubmissionRequest",
"appArgs" : [ "" ],
"appResource" : "hdfs://host-ip:9000/test/testing.jar",
"clientSparkVersion" : "1.6.2",
"environmentVariables" : {
"SPARK_ENV_LOADED" : "1"
},
"mainClass" : "Test",
"sparkProperties" : {
"spark.jars" : "hdfs://host-ip:9000/test/testing.jar",
"spark.driver.extraJavaOptions" : "-Dlog4j.configuration=file:///home/log4j-driver.properties",
"spark.app.name" : "Test",
"spark.eventLog.enabled": "true",
"spark.eventLog.dir": "hdfs://host-ip:9000/test/spark-events",
"spark.submit.deployMode" : "client",
"spark.master" : "spark://host-ip:7077"
}
}'
于 2016-08-12T15:47:24.577 に答える