I would like to use chunks inside celery chain command.
chain = task1.s(arg1) | task2.chunks(?,CHUNK_SIZE) | task3.chunks(?, CHUNK_SIZE)
Basicly what I would like to do is run task1, chunk it's results and send chunks to task2 which should then call task3 which should also receive chunked results from task2 to finish the process. Why? Because task1 and task2 can both return fair amount of items which I would like to process in more batches.
The code above does not work since I'm not quite sure what to put instead of question marks in order to make it work.
I'm not quite sure this is even possible since searching didn't provide much results, so in the case it's not possible to construct such workflow I'd be interested into reasonable alternatives.