flatten' is tail recursive, so it shouldn't take any stack space. It will however walk down the left side of the tree, spitting out a bunch of thunks in the heap. If you invoke it on your example tree, and reduce it to WHNF, you should get something that looks like this:
:
/ \
1 flatten' Tip :
/ \
2 flatten' (Node 4) []
/ \
(Node 3) Tip
/ \
Tip Tip
The algorithm is O(N), but it has to examine the Tips as well as the Nodes.
This looks to be the most efficient way to flatten your tree in-order. The Data.Tree module has a flatten function here which does much the same thing, except it prefers a pre-order traversal.
Update:
In a graph reduction engine, the flatten in main will generate a graph like this:
@
/ \
@ []
/ \
/ \
/ \
flatten' Node 2
/ \
/ \
/ \
Node 1 Node 4
/ \ / \
Tip Tip / \
/ \
Node 3 Tip
/ \
Tip Tip
In order to reduce this to WHNF, the graph reduction engine will unroll the spine, pushing the [] and the Node 2 onto the stack. It will then invoke the flatten' function, which will rewrite the graph to this:
@
/ \
/ \
/ \
@ :
/ \ / \
/ \ 2 \
/ \ \
flatten' Node 1 \
/ \ \
Tip Tip @
/ \
@ []
/ \
/ \
/ \
flatten' Node 4
/ \
/ \
/ \
Node 3 Tip
/ \
Tip Tip
And will pop the two arguments from the stack. The root node is still not in WHNF, so the graph reduction engine will unroll the spine, pushing the 2:... and the Node 1 onto the stack. It will then invoke the flatten' function, which will rewrite the graph to this:
@
/ \
/ \
/ \
@ :
/ \ / \
/ \ 1 \
/ \ \
flatten' Tip @
/ \
/ \
/ :
@ / \
/ \ 2 \
/ Tip @
/ / \
flatten' @ []
/ \
/ \
/ \
flatten' Node 4
/ \
/ \
/ \
Node 3 Tip
/ \
Tip Tip
And will pop the two arguments from the stack. The root node is still not in WHNF, so the graph reduction engine will unroll the spine, pushing the 1:... and the Tip onto the stack. It will then invoke the flatten' function, which will rewrite the graph to this:
:
/ \
1 \
\
@
/ \
/ \
/ :
@ / \
/ \ 2 \
/ Tip @
/ / \
flatten' @ []
/ \
/ \
/ \
flatten' Node 4
/ \
/ \
/ \
Node 3 Tip
/ \
Tip Tip
And will pop the two arguments from the stack. We are now in WHNF, having consumed a maximum of two stack entries (assuming the Tree nodes were not thunks that required additional stack space to evaluate).
So, flatten' is tail-recursive. It replaces itself without having to evaluate additional nested redexes. The second flatten' remains a thunk in the heap, not the stack.