Python NLTK | nltk.tokenizer.word_tokenize() Last Updated : 12 Jun, 2019 Comments Improve Suggest changes Like Article Like Report With the help of nltk.tokenize.word_tokenize() method, we are able to extract the tokens from string of characters by using tokenize.word_tokenize() method. It actually returns the syllables from a single word. A single word can contain one or two syllables. Syntax : tokenize.word_tokenize() Return : Return the list of syllables of words. Example #1 : In this example we can see that by using tokenize.word_tokenize() method, we are able to extract the syllables from stream of words or sentences. Python3 1=1 # import SyllableTokenizer() method from nltk from nltk import word_tokenize # Create a reference variable for Class word_tokenize tk = SyllableTokenizer() # Create a string input gfg = "Antidisestablishmentarianism" # Use tokenize method geek = tk.tokenize(gfg) print(geek) Output : ['An', 'ti', 'dis', 'es', 'ta', 'blish', 'men', 'ta', 'ria', 'nism'] Example #2 : Python3 1=1 # import SyllableTokenizer() method from nltk from nltk.tokenize import word_tokenize # Create a reference variable for Class word_tokenize tk = SyllableTokenizer() # Create a string input gfg = "Gametophyte" # Use tokenize method geek = tk.tokenize(gfg) print(geek) Output : ['Ga', 'me', 'to', 'phy', 'te'] Comment More infoAdvertise with us Next Article Python NLTK | nltk.tokenizer.word_tokenize() J Jitender_1998 Follow Improve Article Tags : Python Python-nltk Practice Tags : python Similar Reads Python NLTK | nltk.tokenize.TabTokenizer() With the help of nltk.tokenize.TabTokenizer() method, we are able to extract the tokens from string of words on the basis of tabs between them by using tokenize.TabTokenizer() method. Syntax : tokenize.TabTokenizer() Return : Return the tokens of words. Example #1 : In this example we can see that b 1 min read Python NLTK | nltk.tokenize.SpaceTokenizer() With the help of nltk.tokenize.SpaceTokenizer() method, we are able to extract the tokens from string of words on the basis of space between them by using tokenize.SpaceTokenizer() method. Syntax : tokenize.SpaceTokenizer() Return : Return the tokens of words. Example #1 : In this example we can see 1 min read Python NLTK | nltk.tokenize.SExprTokenizer() With the help of nltk.tokenize.SExprTokenizer() method, we are able to extract the tokens from string of characters or numbers by using tokenize.SExprTokenizer() method. It actually looking for proper brackets to make tokens. Syntax : tokenize.SExprTokenizer() Return : Return the tokens from a strin 1 min read Python NLTK | nltk.tokenize.StanfordTokenizer() With the help of nltk.tokenize.StanfordTokenizer() method, we are able to extract the tokens from string of characters or numbers by using tokenize.StanfordTokenizer() method. It follows stanford standard for generating tokens. Syntax : tokenize.StanfordTokenizer() Return : Return the tokens from a 1 min read Python NLTK | nltk.TweetTokenizer() With the help of NLTK nltk.TweetTokenizer() method, we are able to convert the stream of words into small  tokens so that we can analyse the audio stream with the help of nltk.TweetTokenizer() method. Syntax : nltk.TweetTokenizer() Return : Return the stream of token Example #1 : In this example whe 1 min read Like