Potatoswatter is right, but I think I have a bit "clearer" explanation. I think the OP is getting confused on what happens at run-time with dynamic type lookup versus compile-time, and when up-casting happens automatically, versus when it does not.
First off, return type does NOT affect which overload is called. You probably know that, but it bears repeating. A return type mis-match will cause an error at compile-time, but not run-time, and does not affect which overload is called. Also it's worth noting that as long as it is a compatible pointer type (in a hierarchy together) returning a pointer doesn't ever "change" it. It is still the same pointer, unlike converting floats to ints, where there is an actual change.
Now to go through the calls one-by-one. This is my understanding of the process, not necessarily what the standard, or what "really" happens.
When you call b->set(b)
the compiler (not run-time) goes "looking for a method named set with an argument of pointer to B" which it finds with the one that outputs set1. It's virtual, so there's code to check if the class points to anything lower, but there isn't, so it just calls it, and returns the this
pointer into a
.
Now you're calling b->set(a)
. Again it's the compiler that goes "does b have an overload that takes pointer to A?" Yes it does, so it calls the "set2" method. It's the compiler that sees an A*
and so the call is "determined" at that point. Even though the pointer points to an object that is of type B, the compiler doesn't know that, or care. So it's the compile-time types of the arguments that determine which overloaded method get taken. From that point on, where in the hierarchy the virtual gets taken is on the underlying type of the this
pointer, but only downward.
Here's a different case though. Try this: b->set(dynamic_cast<B*>(a))
This should call the "set1" method, because the compiler is going to definitely have a pointer to B (even if it's nullptr
).
Now the third case: a->set(b)
. What's happening here is the compiler says "there is only one set
method, so can the argument be up-cast or constructed to that type?" The answer is yes, as B
is a child of A
. So that cast happens transparantly, and the compiler calls the ABSTRACT dispatcher for the set method of the type A. This occurs at compile time before the "real" type of what a
is pointer to. Then at run-time, the program "walks the virtual" and finds the lowest one, the B->set(A*)
method that emits "set2". The actual type of what the argument points to isn't used, only the type to the left of the arrow operator, and that only determines how far down the hierarchy.
And the fourth case is just the 3rd again. The type of the argument (the pointer, not whta is pointed to) is compatible, so it goes as before. If you want a dramatic demonstration of this, try this:
a->set((A*)nullptr) // prints "set2 has been called"
b->set((A*)nullptr) // prints "set2 has been called"
b->set((B*)nullptr) // prints "set1 has been called"
The underlying type of what the arguments point to doesn't affect dynamic dispatch. Only their "surface" type affects the overload called.