2 - 5 - Sentence Segmentation - Stanford NLP - Professor Dan Jurafsky & Chris Manning
85 0 29909
If you are interest on more free online course info, welcome to: http://opencourseonline.com/
Professor Dan Jurafsky & Chris Manning are offering a free online course on Natural Language Processing starting in March 19, 2012. http://www.nlp-class.org/
Offered by Coursera: https://www.coursera.org/
By anonymous 2017-09-20
Ok so sentence-tokenizers are something I looked at in a little detail, using regexes, nltk, CoreNLP. You end up writing your own and it depends on the application. This stuff is tricky and valuable and people don't just give their tokenizer code away. (Ultimately, tokenization is not a deterministic procedure, it's probabilistic, and also depends very heavily on your corpus or domain, e.g. social-media posts or Yelp reviews)
In general you can't rely on one single Great White infallible regex, you have to write a function which uses several regexes (both positive and negative); also a dictionary of abbreviations, and some basic language parsing which knows that e.g. 'I', 'USA', 'FCC', 'TARP' are capitalized in English.
To illustrate how easily this can get seriously complicated, let's try to write you that functional spec for a deterministic tokenizer just to decide whether single or multiple period ('.'/'...') indicates end-of-sentence, or something else:
function isEndOfSentence(leftContext, rightContext)
- Return False for decimals inside numbers or currency e.g. 1.23 , $1.23, "That's just my $.02" Consider also section references like 1.2.3 or European date formats like 09.07.2014
- Return False (and don't tokenize into individual letters) for known abbreviations e.g. "U.S. stocks are falling" ; this requires a dictionary of known abbreviations. Anything outside that dictionary you will get wrong.
- Ellipses '...' at ends of sentences are terminal, but in the middle of sentences are not. This is not as easy as you might think: you need to look at the left context and the right context, specifically is the RHS capitalized and again consider capitalized words like 'I' and abbreviations. Here's an example proving ambiguity which : She asked me to stay... I left an hour later. (Was that one sentence or two? Impossible to determine)
- You may also want to write a few patterns to detect and reject miscellaneous non-sentence-ending uses of punctuation: emoticons, ASCII art, spaced ellipses . . . and other stuff esp. Twitter. (Making that adaptive is even harder). How do we tell if @midnight is a Twitter user, the show on Comedy Central, or simply unwanted/junk/typo punctuation? Seriously non-trivial.
- After you handle all those negative cases, you could arbitrarily say that any isolated period followed by whitespace is likely to be an end of sentence. (Ultimately, if you really want to buy extra accuracy, you end up writing your own probabilistic sentence-tokenizer which uses weights, and training it on a specific corpus(e.g. legal texts, broadcast media, StackOverflow, Twitter, forums comments etc.)) Then you have to manually review exemplars and training errors. See Manning and Jurafsky book or Coursera course [a]. Ultimately you get as much correctness as you are prepared to pay for.
- All of the above is clearly specific to the English-language/ abbreviations, US number/time/date formats. If you want to make it country- and language-independent, that's a bigger proposition, you'll need corpora, native-speaking people to label and QA it all, etc.
- All of the above is still only ASCII. Allow the input to be Unicode, and things get harder still (and the training-set necessarily must be either much bigger or much sparser)
In the simple (deterministic) case,
function isEndOfSentence(leftContext, rightContext) returns boolean, but in the more general sense, it's probabilistic: it returns a float (confidence level that that particular '.' is a sentence end).