RStudio で SparkR を使用しています。hiveContextからデータを抽出するために使用しようとしていますHive/HDFS。データにはフォーマットがあるように見えますが、データベース内AVROの s が適切に宣言されていないため競合しているようです。関数NULLを使用してデータをロードしようとすると、次のような警告が表示されます。SparkR::sql
このエラーを克服する解決策を提案できる人はいますか?
> library(SparkR, lib.loc = "/opt/spark-1.5.0-bin-hadoop2.4/R/lib")
Attaching package: ‘SparkR’
The following objects are masked from ‘package:stats’:
filter, na.omit
The following objects are masked from ‘package:base’:
intersect, rbind, sample, subset, summary, table, transform
Warning message:
package ‘SparkR’ was built under R version 3.2.1
> sc <- sparkR.init(sparkHome = "/opt/spark-1.5.0-bin-hadoop2.4/")
Launching java with spark-submit command /opt/spark-1.5.0-bin-hadoop2.4//bin/spark-submit sparkr-shell /tmp/RtmpgO9iNq/backend_portb16a2bf909d3
Warning message:
In Filter(function(x) { : bytecode version mismatch; using eval
> sqlContext <- sparkRSQL.init(sc)
> hiveContext <- sparkRHive.init(sc)
> statki_pvki_datetime <- sql(hiveContext, select)
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
> str(statki_pvki_datetime)
Formal class 'DataFrame' [package "SparkR"] with 2 slots
..@ env:<environment: 0x3c6cfcf8>
..@ sdf:Class 'jobj' <environment: 0x3c6cb250>
> head(statki_pvki_datetime)
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 0.0 failed 1 times, most recent failure: Lost task 2.0 in stage 0.0 (TID 2, localhost): org.apache.avro.AvroTypeException: Found pl.wp.avro.server.Server, expecting union
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:292)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:155)
at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
at org.apache.hadoop.hive.serde2.avro.AvroDeserializer$SchemaReEncoder.reencode(AvroDeserializer.java:11
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]
[WARNING] Avro: Invalid default for field all: null not a [{"type":"map","values":"string"},"null"]