Note: The behavior this question and answer describes are now available using built in API in NSTableView on OS X 10.9 Mavericks and later. See NSTableViewDraggingDestinationFeedbackStyleGap.
This answer may still be useful if this behavior is needed in an app targeting OS X 10.8 or earlier.
Original answer below:
I've implemented this now. My basic approach looks like this:
@interface ORSGapOpeningTableView : NSTableView
@property (nonatomic) NSInteger dropTargetRow;
@property (nonatomic) CGFloat heightOfDraggedRows;
@end
@implementation ORSGapOpeningTableView
#pragma mark - Dragging
- (NSDragOperation)draggingUpdated:(id<NSDraggingInfo>)sender
{
NSInteger oldDropTargetRow = self.dropTargetRow;
NSDragOperation result = [super draggingUpdated:sender];
CGFloat imageHeight = [[sender draggedImage] size].height;
self.heightOfDraggedRows = imageHeight;
NSMutableIndexSet *changedRows = [NSMutableIndexSet indexSet];
if (oldDropTargetRow > 0) [changedRows addIndex:oldDropTargetRow-1];
if (self.dropTargetRow > 0) [changedRows addIndex:self.dropTargetRow-1];
[self noteHeightOfRowsWithIndexesChanged:changedRows];
return result;
}
- (void)draggingExited:(id<NSDraggingInfo>)sender
{
self.dropTargetRow = -1;
[self noteHeightOfRowsWithIndexesChanged:[NSIndexSet indexSetWithIndexesInRange:NSMakeRange(0, [self numberOfRows])]];
[super draggingExited:sender];
}
- (void)draggingEnded:(id<NSDraggingInfo>)sender
{
self.dropTargetRow = -1;
self.heightOfDraggedRows = 0.0;
self.draggedRows = nil;
[self noteHeightOfRowsWithIndexesChanged:[NSIndexSet indexSetWithIndexesInRange:NSMakeRange(0, [self numberOfRows])]];
}
- (BOOL)performDragOperation:(id<NSDraggingInfo>)sender
{
self.dropTargetRow = -1;
self.heightOfDraggedRows = 0.0;
self.draggedRows = nil;
[self noteHeightOfRowsWithIndexesChanged:[NSIndexSet indexSetWithIndexesInRange:NSMakeRange(0, [self numberOfRows])]];
return [super performDragOperation:sender];
}
// In my delegate and data source:
- (NSDragOperation)tableView:(NSTableView *)tableView validateDrop:(id<NSDraggingInfo>)info proposedRow:(NSInteger)row proposedDropOperation:(NSTableViewDropOperation)dropOperation
{
if (dropOperation == NSTableViewDropOn)
{
dropOperation = NSTableViewDropAbove;
[self.tableView setDropRow:++row dropOperation:dropOperation];
}
NSDragOperation result = [self.realDataSource tableView:tableView validateDrop:info proposedRow:row proposedDropOperation:dropOperation];
if (result != NSDragOperationNone)
{
self.tableView.dropTargetRow = row;
}
else
{
self.tableView.dropTargetRow = -1; // Don't open a gap
}
return result;
}
- (CGFloat)tableView:(NSTableView *)tableView heightOfRow:(NSInteger)row
{
CGFloat result = [tableView rowHeight];
if (row == self.tableView.dropTargetRow - 1 && row > -1)
{
result += self.tableView.heightOfDraggedRows;
}
return result;
}
Note that this is simplified code, not a verbatim copy/paste from my program. I actually ended up making this all contained within an NSTableView subclass that uses proxy delegate and data source objects so the code in data source/delegate methods above is actually inside the proxies' intercept of the calls to the real delegate and data source. That way, the real data source and delegate don't have to do anything special to get the gap opening behavior. Also, there's sometimes a little flakiness with the table view animations, and this doesn't work for drags above the first row (no gap is opened since there's no row to make taller). All in all, despite the room for further improvement, this approach works reasonably well.
I'd still like to try a similar approach, but insert a blank row (as Caleb suggested) instead of changing the row height.