3

を使用してこれらのエラーが発生してsbt assemblyいます。

この問題の根本にあると思われる Spark を使用しています。

val Spark =  Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion
)

エラー:

[error] 12 errors were encountered during merge
[trace] Stack trace suppressed: run last coreBackend/*:assembly for the full output.
[trace] Stack trace suppressed: run last core/*:assembly for the full output.
[trace] Stack trace suppressed: run last commons/*:assembly for the full output.
[error] (coreBackend/*:assembly) deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.osgi/org.osgi.core/org.osgi.core-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.osgi/org.osgi.compendium/org.osgi.compendium-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Absent.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Absent.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Function.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Function.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Optional$1$1.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1$1.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Optional$1.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Optional.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Present.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Present.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Supplier.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Supplier.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-core_2.11/spark-core_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-launcher_2.11/spark-launcher_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.spark-project.spark/unused/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-shuffle_2.11/spark-network-shuffle_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-unsafe_2.11/spark-unsafe_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-sql_2.11/spark-sql_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-catalyst_2.11/spark-catalyst_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-streaming_2.11/spark-streaming_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] (core/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.core/jars/org.osgi.core-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.compendium/jars/org.osgi.compendium-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Absent.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Absent.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Function.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Function.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Optional$1$1.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1$1.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Optional$1.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Optional.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Present.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Present.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Supplier.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Supplier.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
[error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
[error] deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-core_2.11/jars/spark-core_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-launcher_2.11/jars/spark-launcher_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.spark-project.spark/unused/jars/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-shuffle_2.11/jars/spark-network-shuffle_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-unsafe_2.11/jars/spark-unsafe_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-sql_2.11/jars/spark-sql_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-catalyst_2.11/jars/spark-catalyst_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-streaming_2.11/jars/spark-streaming_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
[error] (commons/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.core/jars/org.osgi.core-4.3.1.jar:OSGI-OPT/bnd.bnd
[error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.compendium/jars/org.osgi.compendium-4.3.1.jar:OSGI-OPT/bnd.bnd

ここで推奨されるすべてのソリューションを試してみましたが、うまくいきませんでした。

sbt-assembly: 重複排除エラーが見つかりました

commons-validator の重複排除 - sbt アセンブリ

spark + sbt-assembly: "重複排除: 以下で見つかった異なるファイルの内容"

4

4 に答える 4

1
assemblyMergeStrategy in assembly := {
case PathList("org", "apache", xs @ _*) => MergeStrategy.last
case PathList("com", "google", xs @ _*) => MergeStrategy.last
case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

これは私にとってはうまくいきます。

于 2016-07-10T11:02:21.617 に答える