1

展開モードで Nutch2.0 を使用して URL をクロールしています。

  1. 私はAntでNutchを構築します
  2. Hadoop で Nutch-2.0.job を使用しました。

次のコマンドを実行しました。

  1. sudo -u hdfs hadoop jar /home/bluesky/nutch/runtime/deploy/apache-nutch-2.0.job org.apache.nutch.crawl.InjectorJob /user/bluesky/nutch/urls

  2. sudo -u hdfs hadoop jar /home/bluesky/nutch/runtime/deploy/apache-nutch-2.0.job org.apache.nutch.crawl.GeneratorJob

  3. sudo -u hdfs hadoop jar /home/bluesky/nutch/runtime/deploy/apache-nutch-2.0.job org.apache.nutch.fetcher.FetcherJob (batch -id)

しかし、最後のコマンドを実行すると、ログにエラーが表示されます:

2012-10-09 15:35:04,292 ERROR org.apache.nutch.fetcher.FetcherJob: Unexpected error for http://www.imdb.com/title/tt0327056/
java.lang.AbstractMethodError: org.apache.gora.persistency.ListGenericArray.size()I
        at org.apache.avro.generic.GenericDatumWriter.getArraySize(GenericDatumWriter.java:142)
        at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:128)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
        at org.apache.gora.avro.PersistentDatumWriter.write(PersistentDatumWriter.java:61)
        at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:102)
        at org.apache.gora.avro.PersistentDatumWriter.writeRecord(PersistentDatumWriter.java:74)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:64)
        at org.apache.gora.avro.PersistentDatumWriter.write(PersistentDatumWriter.java:61)
        at org.apache.gora.util.IOUtils.serialize(IOUtils.java:170)
        at org.apache.gora.sql.store.SqlStore.setField(SqlStore.java:718)
        at org.apache.gora.sql.store.SqlStore.setObject(SqlStore.java:669)
        at org.apache.gora.sql.statement.MySqlInsertUpdateStatement.toStatement(MySqlInsertUpdateStatement.java:96)
        at org.apache.gora.sql.store.SqlStore.put(SqlStore.java:616)
        at org.apache.gora.mapreduce.GoraRecordWriter.write(GoraRecordWriter.java:60)
        at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:514)
        at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
        at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.output(FetcherReducer.java:668)
        at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:540)
2012-10-09 15:35:04,293 ERROR org.apache.nutch.fetcher.FetcherJob: fetcher throwable caught
java.lang.AbstractMethodError: org.apache.gora.persistency.ListGenericArray.size()I
        at org.apache.avro.generic.GenericDatumWriter.getArraySize(GenericDatumWriter.java:142)
        at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:128)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
        at org.apache.gora.avro.PersistentDatumWriter.write(PersistentDatumWriter.java:61)
        at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:102)
        at org.apache.gora.avro.PersistentDatumWriter.writeRecord(PersistentDatumWriter.java:74)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:64)
        at org.apache.gora.avro.PersistentDatumWriter.write(PersistentDatumWriter.java:61)
        at org.apache.gora.util.IOUtils.serialize(IOUtils.java:170)
        at org.apache.gora.sql.store.SqlStore.setField(SqlStore.java:718)
        at org.apache.gora.sql.store.SqlStore.setObject(SqlStore.java:669)
        at org.apache.gora.sql.statement.MySqlInsertUpdateStatement.toStatement(MySqlInsertUpdateStatement.java:96)
        at org.apache.gora.sql.store.SqlStore.put(SqlStore.java:616)
        at org.apache.gora.mapreduce.GoraRecordWriter.write(GoraRecordWriter.java:60)
4

0 に答える 0