How to read values from numbers written as words?
As we all know numbers can be written either in numerics, or called by their names. While there are a lot of examples to be found that convert 123 into one hundred twenty three, I could not find good examples of how to convert it the other way around.
Some of the caveats:
- cardinal/nominal or ordinal: "one" and "first"
- common spelling mistakes: "forty"/"fourty"
- hundreds/thousands: 2100 -> "twenty one hundred" and also "two thousand and one hundred"
- separators: "eleven hundred fifty two", but also "elevenhundred fiftytwo" or "eleven-hundred fifty-two" and whatnot
- colloquialisms: "thirty-something"
- fractions: 'one third', 'two fifths'
- common names: 'a dozen', 'half'
And there are probably more caveats possible that are not yet listed. Suppose the algorithm needs to be very robust, and even understand spelling mistakes.
What fields/papers/studies/algorithms should I read to learn how to write all this? Where is the information?
PS: My final parser should actually understand 3 different languages, English, Russian and Hebrew. And maybe at a later stage more languages will be added. Hebrew also has male/female numbers, like "one man" and "one woman" have a different "one" — "ehad" and "ahat". Russian also has some of its own complexities.
Google does a great job at this. For example:
http://www.google.com/search?q=two+thousand+and+one+hundred+plus+five+dozen+and+four+fifths+in+decimal
(the reverse is also possible http://www.google.com/search?q=999999999999+in+english)
Solution 1:
I was playing around with a PEG parser to do what you wanted (and may post that as a separate answer later) when I noticed that there's a very simple algorithm that does a remarkably good job with common forms of numbers in English, Spanish, and German, at the very least.
Working with English for example, you need a dictionary that maps words to values in the obvious way:
"one" -> 1, "two" -> 2, ... "twenty" -> 20,
"dozen" -> 12, "score" -> 20, ...
"hundred" -> 100, "thousand" -> 1000, "million" -> 1000000
...and so forth
The algorithm is just:
total = 0
prior = null
for each word w
v <- value(w) or next if no value defined
prior <- case
when prior is null: v
when prior > v: prior+v
else prior*v
else
if w in {thousand,million,billion,trillion...}
total <- total + prior
prior <- null
total = total + prior unless prior is null
For example, this progresses as follows:
total prior v unconsumed string
0 _ four score and seven
4 score and seven
0 4
20 and seven
0 80
_ seven
0 80
7
0 87
87
total prior v unconsumed string
0 _ two million four hundred twelve thousand eight hundred seven
2 million four hundred twelve thousand eight hundred seven
0 2
1000000 four hundred twelve thousand eight hundred seven
2000000 _
4 hundred twelve thousand eight hundred seven
2000000 4
100 twelve thousand eight hundred seven
2000000 400
12 thousand eight hundred seven
2000000 412
1000 eight hundred seven
2000000 412000
1000 eight hundred seven
2412000 _
8 hundred seven
2412000 8
100 seven
2412000 800
7
2412000 807
2412807
And so on. I'm not saying it's perfect, but for a quick and dirty it does quite well.
Addressing your specific list on edit:
- cardinal/nominal or ordinal: "one" and "first" -- just put them in the dictionary
- english/british: "fourty"/"forty" -- ditto
- hundreds/thousands: 2100 -> "twenty one hundred" and also "two thousand and one hundred" -- works as is
- separators: "eleven hundred fifty two", but also "elevenhundred fiftytwo" or "eleven-hundred fifty-two" and whatnot -- just define "next word" to be the longest prefix that matches a defined word, or up to the next non-word if none do, for a start
- colloqialisms: "thirty-something" -- works
- fragments: 'one third', 'two fifths' -- uh, not yet...
- common names: 'a dozen', 'half' -- works; you can even do things like "a half dozen"
Number 6 is the only one I don't have a ready answer for, and that's because of the ambiguity between ordinals and fractions (in English at least) added to the fact that my last cup of coffee was many hours ago.
Solution 2:
It's not an easy issue, and I know of no library to do it. I might sit down and try to write something like this sometime. I'd do it in either Prolog, Java or Haskell, though. As far as I can see, there are several issues:
- Tokenization: sometimes, numbers are written eleven hundred fifty two, but I've seen elevenhundred fiftytwo or eleven-hundred-fifty-two and whatnot. One would have to conduct a survey on what forms are actually in use. This might be especially tricky for Hebrew.
- Spelling mistakes: that's not so hard. You have a limited amount of words, and a bit of Levenshtein-distance magic should do the trick.
- Alternate forms, like you already mentioned, exist. This includes ordinal/cardinal numbers, as well as forty/fourty and...
- ... common names or commonly used phrases and NEs (named entities). Would you want to extract 30 from the Thirty Years War or 2 from World War II?
- Roman numerals, too?
- Colloqialisms, such as "thirty-something" and "three Euro and shrapnel", which I wouldn't know how to treat.
If you are interested in this, I could give it a shot this weekend. My idea is probably using UIMA and tokenizing with it, then going on to further tokenize/disambiguate and finally translate. There might be more issues, let's see if I can come up with some more interesting things.
Sorry, this is not a real answer yet, just an extension to your question. I'll let you know if I find/write something.
By the way, if you are interested in the semantics of numerals, I just found an interesting paper by Friederike Moltmann, discussing some issues regarding the logic interpretation of numerals.
Solution 3:
I have some code I wrote a while ago: text2num. This does some of what you want, except it does not handle ordinal numbers. I haven't actually used this code for anything, so it's largely untested!
Solution 4:
Use the Python pattern-en library:
>>> from pattern.en import number
>>> number('two thousand fifty and a half') => 2050.5