We migrated our cluster to Cassandra 1.2 (from 1.1.7) and we tried to run a repair after the migration (we run them periodically, as recommended).
The repair failed due to the SSTable version being different, so we used nodetool upgradesstables
and nodetool scrub
and retried.
But then the repair failed with this exception :
java.io.IOException: Broken pipe
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:90)
at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
We also noticed that before dying, nodetool netstats
shows weird statistics (SSTable transfers being at 756845%...).
EDIT
It seems to be a problem with the transfer of compressend SSTables, I will try to disable compression on this ColumnFamily and retry.
RE-EDIT
Disabling compression fixed the problem so we tried to re-enable it and got a
java.lang.RuntimeException:
Last written key DecoratedKey(97115[...]3136) >= current key DecoratedKey(8...
Any idea?