How do indices work in a list with duplicates?

My question is why "and" here have the same index of 0 instead of 0, 3, 5?

Why

It's because list.index() returns the index of the first occurrence, so since "and" first appears in index 0 in the list, that's what you'll always get.

Solution

If you want to follow the index as you go try enumerate()

for i, token in enumerate(tokens):
    print(i, token)

Which gives the output you want:

0 and
1 of
2 then
3 and
4 for
5 and

Use enumerate.

In [1]: tokens = ["and", "of", "then", "and", "for", "and"]
In [2]: for word_index,word in enumerate(tokens):
   ....:     print (word_index, word)
   ....:     

Output

0 and
1 of
2 then
3 and
4 for
5 and