約 20,000 のドキュメントを含むコーパスがあり、LDA を使用してトピック モデリング用にそのデータ セットをトレーニングする必要があります。
import logging, gensim
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
id2word = gensim.corpora.Dictionary('questions.dict')
mm = gensim.corpora.MmCorpus('questions.mm')
lda = gensim.models.ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=100, update_every=0, chunksize=3000, passes=20)
lda.print_topics(20)
このプログラムを実行するたびに、次のエラーが発生します。
2013-04-28 09:57:09,750 : INFO : adding document #0 to Dictionary(0 unique tokens)
2013-04-28 09:57:09,759 : INFO : built Dictionary(11 unique tokens) from 14 documents (total 14 corpus positions)
2013-04-28 09:57:09,785 : INFO : loaded corpus index from questions.mm.index
2013-04-28 09:57:09,790 : INFO : initializing corpus reader from questions.mm
2013-04-28 09:57:09,796 : INFO : accepted corpus with 19188 documents, 15791 features, 106222 non-zero entries
2013-04-28 09:57:09,802 : INFO : using serial LDA version on this node
2013-04-28 09:57:09,808 : INFO : running batch LDA training, 100 topics, 20 passes over the supplied corpus of 19188 documents, updating model once every 19188 documents
2013-04-28 09:57:10,267 : INFO : PROGRESS: iteration 0, at document #3000/19188
Traceback (most recent call last):
File "C:/Users/Animesh/Desktop/NLP/topicmodel/lda.py", line 10, in <module>
lda = gensim.models.ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=100, update_every=0, chunksize=3000, passes=20)
File "C:\Python27\lib\site-packages\gensim-0.8.6-py2.7.egg\gensim\models\ldamodel.py", line 265, in __init__
self.update(corpus)
File "C:\Python27\lib\site-packages\gensim-0.8.6-py2.7.egg\gensim\models\ldamodel.py", line 445, in update
self.do_estep(chunk, other)
File "C:\Python27\lib\site-packages\gensim-0.8.6-py2.7.egg\gensim\models\ldamodel.py", line 365, in do_estep
gamma, sstats = self.inference(chunk, collect_sstats=True)
File "C:\Python27\lib\site-packages\gensim-0.8.6-py2.7.egg\gensim\models\ldamodel.py", line 318, in inference
expElogbetad = self.expElogbeta[:, ids]
IndexError: index (11) out of range (0<=index<10) in dimension 1
関数の値を変更しようとしましたLdaModelが、常に同じエラーが発生します!
何をすべきですか？