The slowdown in caused by a quadradic effect: as n gets larger you have to do more multiplications, but you also have to multiply larger numbers.
Finding a better algorithm won't be easy. You can try to exploit symmetries (as in FFT). It could also well pay to do multiplications in a different order, with intermediate results, such that you end up with multiplying only a few very big numbers at the end, but I haven't thought that to the end. In any case, you will have to find a law to exploit.
Look here for further inspiration.