It++ ライブラリのコンパイルに問題があります。正しくインストールし、mex を使用しない簡単な例でテストしました。
mex it_mex_fkmigration.cpp -litpp
it_mex_fkmigration.cpp:1:25: error: itpp/itcomm.h: No such file or directory
it_mex_fkmigration.cpp:2:26: error: itpp/itmex.h: No such file or directory
it_mex_fkmigration.cpp:4: error: expected namespace-name before ';' token
it_mex_fkmigration.cpp:4: error: '<type error>' is not a namespace
it_mex_fkmigration.cpp:6: error: 'mxArray' has not been declared
it_mex_fkmigration.cpp:6: error: expected ',' or '...' before '*' token
it_mex_fkmigration.cpp:6: error: ISO C++ forbids declaration of 'mxArray' with no type
it_mex_fkmigration.cpp: In function 'void mexFunction(int, int**, int, int)':
it_mex_fkmigration.cpp:9: error: 'mexErrMsgTxt' was not declared in this scope
it_mex_fkmigration.cpp:10: error: 'mexErrMsgTxt' was not declared in this scope
it_mex_fkmigration.cpp:13: error: 'bvec' was not declared in this scope
it_mex_fkmigration.cpp:13: error: expected `;' before 'input_bits'
it_mex_fkmigration.cpp:16: error: 'cvec' was not declared in this scope
it_mex_fkmigration.cpp:16: error: expected `;' before 'output_symbols'
it_mex_fkmigration.cpp:17: error: 'QPSK' was not declared in this scope
it_mex_fkmigration.cpp:17: error: expected `;' before 'qpsk'
it_mex_fkmigration.cpp:19: error: 'output_symbols' was not declared in this scope
it_mex_fkmigration.cpp:19: error: 'qpsk' was not declared in this scope
it_mex_fkmigration.cpp:19: error: 'input_bits' was not declared in this scope
it_mex_fkmigration.cpp:23: error: 'mxCOMPLEX' was not declared in this scope
it_mex_fkmigration.cpp:23: error: 'mxCreateDoubleMatrix' was not declared in this scope
it_mex_fkmigration.cpp:26: error: 'cvec2mxArray' was not declared in this scope