NLTK 词干化

NLTK 词干化在NLP中我们对一句话或一个文档分词之后一般要进行词干化处理。词干化处理就是把一些名词的复数去掉动词的不同时态去掉等等类似的处理。对于切词得到的英文单词要进行词干化处理主要包括将名词的复数变为单数和将动词的其他形态变为基本形态在nltk当中有两种方法做词干化处理“porter” “snowball”import nltk word_data It originated from the idea that there are readers who prefer learning new skills from the comforts of their drawing rooms words word_data.split( ) porterStemmer nltk.stem.PorterStemmer() snowballStemmer nltk.stem.SnowballStemmer(english) def stem_tokens(tokens, stemmer): stemmed [] for token in tokens: stemmed.append(stemmer.stem(token)) return stemmed print(word_data) print(stem_tokens(words,porterStemmer)) print(stem_tokens(words,snowballStemmer))It originated from the idea that there are readers who prefer learning new skills from the comforts of their drawing rooms [It, origin, from, the, idea, that, there, are, reader, who, prefer, learn, new, skill, from, the, comfort, of, their, draw, room] [it, origin, from, the, idea, that, there, are, reader, who, prefer, learn, new, skill, from, the, comfort, of, their, draw, room]