gensim/ Training a LDA Model: 'int' object is not subscriptable

I create a new word list in which stop words from 'text8' have been removed, in order to train a LDA Model. However, I received TypeError: 'int' object is not subscriptable, guessing problems from corpus, and cannot find the solutions.

Here is my code:

import gensim.downloader as api
corpus=api.load('text8')
dictionary=gensim.corpora.Dictionary(corpus) # generate a dictionary from the text corpus

# removing stop words
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
import nltk
nltk.download('stopwords')
nltk.download('punkt')

stop_words = set(stopwords.words('english'))
word_tokens = dictionary

filtered_sentence = []
for w in word_tokens:
    if word_tokens[w] not in stop_words:
        filtered_sentence.append(word_tokens[w])

#print(filtered_sentence)

# generate a new dictionary from "filtered_sentence"

dct=gensim.corpora.Dictionary([filtered_sentence])
corpus2=dct.doc2bow(filtered_sentence)

The following line is not working-- TypeError: 'int' object is not subscriptable

model=gensim.models.ldamodel.LdaModel(corpus2, num_topics=5, id2word=dct) #TypeError

model.print_topics(num_words=5)

Detailed error message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-64-75e1fe1a727b> in <module>()
----> 1 model=gensim.models.ldamodel.LdaModel(corpus2, num_topics=5, id2word=dct) #TypeError: 'int' object is not subscriptable
      2 model.print_topics(num_words=5)

3 frames
/usr/local/lib/python3.7/dist-packages/gensim/models/ldamodel.py in inference(self, chunk, collect_sstats)
    651         # to Blei's original LDA-C code, cool!).
    652         for d, doc in enumerate(chunk):
--> 653             if len(doc) > 0 and not isinstance(doc[0][0], six.integer_types + (np.integer,)):
    654                 # make sure the term IDs are ints, otherwise np will get upset
    655                 ids = [int(idx) for idx, _ in doc]

TypeError: 'int' object is not subscriptable

Really appreciate your help. Thank you so much!

Solution 1:

The error is likely related to filtered_sentence being used as corpus2. For the code to work corpus2 must be a list of lists of tuples. So, this trick should help:

corpus2 = [dct.doc2bow(filtered_sentence),]