I work in NLP and I would really disagree that NLTK is still an essential player. Some academic courses still teach in it, but I think it's around five years since it has been the best option. I would definitely recommend gensim, but also SpaCy, which is very fast and has a better documented and more extensive API than NLTK. I have nothing to do with SpaCy personally (except being an enthusiastic user) but I recommend this argument: https://explosion.ai/blog/dead-code-should-be-buried

There is also a new higher-level library built on Spacy which looks good: https://textacy.readthedocs.io/en/latest/

Thanks, I tried NLTK many times and ended up dropping it because it became too frustrating to deal with various Parser with no explanation of why there are so many and how to pick between them.

I skimmed though Spacy briefly and it looks great.

How far can you go in spacy with the core 50MB english model ? the other parts are GB large.

The v2 models are much smaller (15mb), because neural networks. The parsing, NER and tagging are mostly okay with the 50mb model. There are only word vectors for the top 5k words though, which can be a problem.

The v2 English models are more accurate, and can assign vectors to any word, including unknown words using the context and the word shape. Overall it's much better -- but it's still in alpha. The docs are already better, though.