java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.io.orc.OrcStruct.<init>() の解決方法を教えてください。
pyspark の起動に使用するコマンド
pyspark --jars "hive-exec-0.13.1-cdh5.3.3.jar,hadoop-common-2.5.0-cdh5.3.3.jar,hadoop-mapreduce-client-app-2.5.0-cdh5.3.3.jar ,hadoop-mapreduce-client-common-2.5.0-cdh5.3.3.jar,hadoop-mapreduce-client-core-2.5.0-cdh5.3.3.jar,hadoop-core-2.5.0-mr1-cdh5.3.3 .jar、ハイブ-メタストア-0.13.1-cdh5.3.3.jar"
pyspark シェルで次のコマンドを実行しました
distFile = sc.newAPIHadoopFile(path="orcdatafolder/",inputFormatClass="org.apache.hadoop.hive.ql.io.orc.OrcNewInputFormat", keyClass="org.apache.hadoop.io.NullWritable" , valueClass=" org.apache.hadoop.hive.ql.io.orc.OrcStruct")
エラー:
16/07/31 19:49:53 WARN scheduler.TaskSetManager: ステージ 0.0 でタスク 0.0 が失われました (TID 0、sj1dra096.corp.adobe.com): java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache. hadoop.hive.ql.io.orc.OrcStruct.<init>() の org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) の org.apache.hadoop.io.WritableUtils.clone(WritableUtils) .java:217) org.apache.spark.api.python.WritableToJavaConverter.org$apache$spark$api$python$WritableToJavaConverter$$convertWritable(PythonHadoopUtil.scala:96) で org.apache.spark.api.python. WritableToJavaConverter.convert(PythonHadoopUtil.scala:104) at org.apache.spark.api.python.PythonHadoopUtil$$anonfun$convertRDD$1.apply(PythonHadoopUtil.scala:183) at org.apache.spark.api.python.PythonHadoopUtil$ $anonfun$convertRDD$1.apply(PythonHadoopUtil.scala:183) scala.collection.Iterator$$anon$11.next(Iterator.scala:328) で scala.collection.Iterator$$anon$10.next(Iterator.scala:312) で scala.collection.Iterator$class.foreach (Iterator.scala:727) で scala.collection.AbstractIterator.foreach(Iterator.scala:1157) で scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) で scala.collection .mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) で scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) で scala.collection.TraversableOnce$class.to (TraversableOnce.scala:273) で scala.collection.AbstractIterator.to(Iterator.scala:1157) で scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) で scala.collection.AbstractIterator.toBuffer(Iterator. scala:1157) で scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) で scala.collection.AbstractIterator.toArray(Iterator.scala:1157) org.apache.spark.rdd.RDD$$anonfun$26.apply(RDD.scala:1081) で org.apache .spark.rdd.RDD$$anonfun$26.apply(RDD.scala:1081) org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1319) org.apache.spark.SparkContext $$anonfun$runJob$4.apply(SparkContext.scala:1319) org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) org.apache.spark.scheduler.Task.run(Task.scala) :56) org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) で java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) で java.util.concurrent.ThreadPoolExecutor$ でWorker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 原因: java.lang.NoSuchMethodException: java.lang.Class.getConstructor0(Class.java:2849) の org.apache.hadoop.hive.ql.io.orc.OrcStruct.() java.lang.Class.getDeclaredConstructor(Class.java:2053) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) ... 28 詳細