How to parse freeform street/postal address out of text, and into components
I saw this question a lot when I worked for an address verification company. I'm posting the answer here to make it more accessible to programmers searching around with the same question. The company I was at processed billions of addresses, and we learned a lot in the process.
First, we need to understand a few things about addresses.
Addresses are not regular
This means that regular expressions are out. I've seen it all, from simple regular expressions that match addresses in a very specific format, to this:
/\s+(\d{2,5}\s+)(?![a|p]m\b)(([a-zA-Z|\s+]{1,5}){1,2})?([\s|,|.]+)?(([a-zA-Z|\s+]{1,30}){1,4})(court|ct|street|st|drive|dr|lane|ln|road|rd|blvd)([\s|,|.|;]+)?(([a-zA-Z|\s+]{1,30}){1,2})([\s|,|.]+)?\b(AK|AL|AR|AZ|CA|CO|CT|DC|DE|FL|GA|GU|HI|IA|ID|IL|IN|KS|KY|LA|MA|MD|ME|MI|MN|MO|MS|MT|NC|ND|NE|NH|NJ|NM|NV|NY|OH|OK|OR|PA|RI|SC|SD|TN|TX|UT|VA|VI|VT|WA|WI|WV|WY)([\s|,|.]+)?(\s+\d{5})?([\s|,|.]+)/i
... to this where a 900+ line-class file generates a supermassive regular expression on the fly to match even more. I don't recommend these (for example, here's a fiddle of the above regex, that makes plenty of mistakes). There isn't an easy magic formula to get this to work. In theory and by theory, it's impossible to match addresses with a regular expression.
USPS Publication 28 documents the many formats of addresses that are possible, with all their keywords and variations. Worst of all, addresses are often ambiguous. Words can mean more than one thing ("St" can be "Saint" or "Street"), and there are words that I'm pretty sure they invented. (Who knew that "Stravenue" was a street suffix?)
You'd need some code that really understands addresses, and if that code does exist, it's a trade secret. But you could probably roll your own if you're really into that.
Addresses come in unexpected shapes and sizes
Here are some contrived (but complete) addresses:
1) 102 main street
Anytown, state
2) 400n 600e #2, 52173
3) p.o. #104 60203
Even these are possibly valid:
4) 829 LKSDFJlkjsdflkjsdljf Bkpw 12345
5) 205 1105 14 90210
Obviously, these are not standardized. Punctuation and line break are not guaranteed. Here's what's going on:
-
Number 1 is complete because it contains a street address and a city and state. With that information, there's enough to identify the address, and it can be considered "deliverable" (with some standardization).
-
Number 2 is complete because it contains a street address (with secondary/unit number) and a 5-digit ZIP code, which is enough to identify an address.
-
Number 3 is a complete post office box format, as it contains a ZIP code.
-
Number 4 is also complete because the ZIP code is unique, meaning that a private entity or corporation has purchased that address space. A unique ZIP code is for high-volume or concentrated delivery spaces. Anything addressed to ZIP code 12345 goes to General Electric in Schenectady, NY. This example won't reach anyone in particular, but the USPS would still deliver it.
-
Number 5 is also complete, believe it or not. With just those numbers, the full address can be discovered when parsed against a database of all possible addresses. Filling in the missing directionals, secondary designator, and ZIP+4 code is trivial when you see each number as a component. Here's what it looks like, fully expanded and standardized:
205 N 1105 W Apt 14
Beverly Hills CA 90210-5221
Address data is not your own
In most countries that provide official address data to licensed vendors, the address data itself belongs to the governing agency. In the US, the USPS owns the addresses. The same is true for Canada Post, Royal Mail, and others, though each country enforces or defines ownership a little differently. Knowing this is important since it usually forbids reverse-engineering the address database. You have to be careful how to acquire, store, and use the data.
Google Maps is a common go-to for quick address fixes, but the TOS is rather prohibitive; for example, you can't use their data or APIs without showing a Google Map, and for non-commercial purposes only (unless you pay), and you can't store the data (except for temporary caching). Makes sense. Google's data is some of the best in the world. However, Google Maps does not verify the address. If an address does not exist, it will still show you where the address would be if it did exist (try it on your own street; use a house number that you know doesn't exist). This is useful sometimes, but be aware of that.
Nominatim's usage policy is similarly limiting, especially for high volume and commercial use, and the data is mostly drawn from free sources, so it isn't as well maintained (such as the nature of open projects). However, this may still suit your needs. A great community supports it.
The USPS itself has an API, but it goes down a lot and comes with no guarantees nor support. It might also be hard to use. Some people use it sparingly with no problems. But it's easy to miss that the USPS requires that you use their API only for confirming addresses to ship through them.
People expect addresses to be hard
Unfortunately, we've conditioned our society to expect addresses to be complicated. There are dozens of good UX articles all over the Internet about this. Still, the fact is, if you have an address form with individual fields, that's what users expect, even though it makes it harder for edge-case addresses that don't fit the format the form is expecting, or maybe the form requires a field it shouldn't. Or users don't know where to put a certain part of their address.
I could go on and on about the bad UX of checkout forms these days, but instead, I'll say that combining the addresses into a single field will be a welcome change -- people will be able to type their address how they see fit, rather than trying to figure out your lengthy form. However, this change will be unexpected and users may find it a little jarring at first. Just be aware of that.
Part of this pain can be alleviated by putting the country field out front, before the address. When they fill out the country field first, you know how to make your form appear. Maybe you have a good way to deal with single-field US addresses, so if they select the United States, you can reduce your form to a single field, otherwise show the component fields. Just things to think about!
Now we know why it's hard; what can you do about it?
The USPS licenses vendors through a process called CASS™ Certification to provide verified addresses to customers. These vendors have access to the USPS database, updated monthly. Their software must conform to rigorous standards to be certified, and they don't often require agreement to such limiting terms as discussed above.
Many CASS-Certified companies can process lists or have APIs: Melissa Data, Experian QAS, and SmartyStreets, to name a few.
(Due to getting flak for "advertising," I've truncated my answer at this point. It's up to you to find a solution that works for you.)
The Truth: Really, folks, I don't work at any of these companies. It's not an advertisement.
libpostal: an open-source library to parse addresses, training with data from OpenStreetMap, OpenAddresses and OpenCage.
https://github.com/openvenues/libpostal (more info about it)
Other tools/services:
http://www.gisgraphy.com Free, open source, and ready to use geocoder and geolocalisation webservices, integrating OpenStreetMap, GeoNames and Quattroshapes.
https://github.com/kodapan/osm-common Library for accessing OpenStreetMap services, parsing and processing data.
http://wiki.openstreetmap.org/wiki/Nominatim
http://address-parser.net/
http://geoservices.tamu.edu/Services/AddressNormalization/
There are many street address parsers. They come in two basic flavors - ones that have databases of place names and street names, and ones that don't.
A regular expression street address parser can get up to about a 95% success rate without much trouble. Then you start hitting the unusual cases. The Perl one in CPAN, "Geo::StreetAddress::US", is about that good. There are Python and Javascript ports of that, all open source. I have an improved version in Python which moves the success rate up slightly by handling more cases. To get the last 3% right, though, you need databases to help with disambiguation.
A database with 3-digit ZIP codes and US state names and abbreviations is a big help. When a parser sees a consistent postal code and state name, it can start to lock on to the format. This works very well for the US and UK.
Proper street address parsing starts from the end and works backwards. That's how the USPS systems do it. Addresses are least ambiguous at the end, where country names, city names, and postal codes are relatively easy to recognize. Street names can usually be isolated. Locations on streets are the most complex to parse; there you encounter things such as "Fifth Floor" and "Staples Pavillion". That's when a database is a big help.
UPDATE: Geocode.xyz now works worldwide. For examples see https://geocode.xyz
For USA, Mexico and Canada, see geocoder.ca.
For example:
Input: something going on near the intersection of main and arthur kill rd new york
Output:
<geodata> <latt>40.5123510000</latt> <longt>-74.2500500000</longt> <AreaCode>347,718</AreaCode> <TimeZone>America/New_York</TimeZone> <standard> <street1>main</street1> <street2>arthur kill</street2> <stnumber/> <staddress/> <city>STATEN ISLAND</city> <prov>NY</prov> <postal>11385</postal> <confidence>0.9</confidence> </standard> </geodata>
You may also check the results in the web interface or get output as Json or Jsonp. eg. I'm looking for restaurants around 123 Main Street, New York
No code? For shame!
Here is a simple JavaScript address parser. It's pretty awful for every single reason that Matt gives in his dissertation above (which I almost 100% agree with: addresses are complex types, and humans make mistakes; better to outsource and automate this - when you can afford to).
But rather than cry, I decided to try:
This code works OK for parsing most Esri results for findAddressCandidate and also with some other (reverse)geocoders that return single-line address where street/city/state are delimited by commas. You can extend if you want or write country-specific parsers. Or just use this as case study of how challenging this exercise can be or at how lousy I am at JavaScript. I admit I only spent about thirty mins on this (future iterations could add caches, zip validation, and state lookups as well as user location context), but it worked for my use case: End user sees form that parses geocode search response into 4 textboxes. If address parsing comes out wrong (which is rare unless source data was poor) it's no big deal - the user gets to verify and fix it! (But for automated solutions could either discard/ignore or flag as error so dev can either support the new format or fix source data.)
/*
address assumptions:
- US addresses only (probably want separate parser for different countries)
- No country code expected.
- if last token is a number it is probably a postal code
-- 5 digit number means more likely
- if last token is a hyphenated string it might be a postal code
-- if both sides are numeric, and in form #####-#### it is more likely
- if city is supplied, state will also be supplied (city names not unique)
- zip/postal code may be omitted even if has city & state
- state may be two-char code or may be full state name.
- commas:
-- last comma is usually city/state separator
-- second-to-last comma is possibly street/city separator
-- other commas are building-specific stuff that I don't care about right now.
- token count:
-- because units, street names, and city names may contain spaces token count highly variable.
-- simplest address has at least two tokens: 714 OAK
-- common simple address has at least four tokens: 714 S OAK ST
-- common full (mailing) address has at least 5-7:
--- 714 OAK, RUMTOWN, VA 59201
--- 714 S OAK ST, RUMTOWN, VA 59201
-- complex address may have a dozen or more:
--- MAGICICIAN SUPPLY, LLC, UNIT 213A, MAGIC TOWN MALL, 13 MAGIC CIRCLE DRIVE, LAND OF MAGIC, MA 73122-3412
*/
var rawtext = $("textarea").val();
var rawlist = rawtext.split("\n");
function ParseAddressEsri(singleLineaddressString) {
var address = {
street: "",
city: "",
state: "",
postalCode: ""
};
// tokenize by space (retain commas in tokens)
var tokens = singleLineaddressString.split(/[\s]+/);
var tokenCount = tokens.length;
var lastToken = tokens.pop();
if (
// if numeric assume postal code (ignore length, for now)
!isNaN(lastToken) ||
// if hyphenated assume long zip code, ignore whether numeric, for now
lastToken.split("-").length - 1 === 1) {
address.postalCode = lastToken;
lastToken = tokens.pop();
}
if (lastToken && isNaN(lastToken)) {
if (address.postalCode.length && lastToken.length === 2) {
// assume state/province code ONLY if had postal code
// otherwise it could be a simple address like "714 S OAK ST"
// where "ST" for "street" looks like two-letter state code
// possibly this could be resolved with registry of known state codes, but meh. (and may collide anyway)
address.state = lastToken;
lastToken = tokens.pop();
}
if (address.state.length === 0) {
// check for special case: might have State name instead of State Code.
var stateNameParts = [lastToken.endsWith(",") ? lastToken.substring(0, lastToken.length - 1) : lastToken];
// check remaining tokens from right-to-left for the first comma
while (2 + 2 != 5) {
lastToken = tokens.pop();
if (!lastToken) break;
else if (lastToken.endsWith(",")) {
// found separator, ignore stuff on left side
tokens.push(lastToken); // put it back
break;
} else {
stateNameParts.unshift(lastToken);
}
}
address.state = stateNameParts.join(' ');
lastToken = tokens.pop();
}
}
if (lastToken) {
// here is where it gets trickier:
if (address.state.length) {
// if there is a state, then assume there is also a city and street.
// PROBLEM: city may be multiple words (spaces)
// but we can pretty safely assume next-from-last token is at least PART of the city name
// most cities are single-name. It would be very helpful if we knew more context, like
// the name of the city user is in. But ignore that for now.
// ideally would have zip code service or lookup to give city name for the zip code.
var cityNameParts = [lastToken.endsWith(",") ? lastToken.substring(0, lastToken.length - 1) : lastToken];
// assumption / RULE: street and city must have comma delimiter
// addresses that do not follow this rule will be wrong only if city has space
// but don't care because Esri formats put comma before City
var streetNameParts = [];
// check remaining tokens from right-to-left for the first comma
while (2 + 2 != 5) {
lastToken = tokens.pop();
if (!lastToken) break;
else if (lastToken.endsWith(",")) {
// found end of street address (may include building, etc. - don't care right now)
// add token back to end, but remove trailing comma (it did its job)
tokens.push(lastToken.endsWith(",") ? lastToken.substring(0, lastToken.length - 1) : lastToken);
streetNameParts = tokens;
break;
} else {
cityNameParts.unshift(lastToken);
}
}
address.city = cityNameParts.join(' ');
address.street = streetNameParts.join(' ');
} else {
// if there is NO state, then assume there is NO city also, just street! (easy)
// reasoning: city names are not very original (Portland, OR and Portland, ME) so if user wants city they need to store state also (but if you are only ever in Portlan, OR, you don't care about city/state)
// put last token back in list, then rejoin on space
tokens.push(lastToken);
address.street = tokens.join(' ');
}
}
// when parsing right-to-left hard to know if street only vs street + city/state
// hack fix for now is to shift stuff around.
// assumption/requirement: will always have at least street part; you will never just get "city, state"
// could possibly tweak this with options or more intelligent parsing&sniffing
if (!address.city && address.state) {
address.city = address.state;
address.state = '';
}
if (!address.street) {
address.street = address.city;
address.city = '';
}
return address;
}
// get list of objects with discrete address properties
var addresses = rawlist
.filter(function(o) {
return o.length > 0
})
.map(ParseAddressEsri);
$("#output").text(JSON.stringify(addresses));
console.log(addresses);<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<textarea>
27488 Stanford Ave, Bowden, North Dakota
380 New York St, Redlands, CA 92373
13212 E SPRAGUE AVE, FAIR VALLEY, MD 99201
1005 N Gravenstein Highway, Sebastopol CA 95472
A. P. Croll & Son 2299 Lewes-Georgetown Hwy, Georgetown, DE 19947
11522 Shawnee Road, Greenwood, DE 19950
144 Kings Highway, S.W. Dover, DE 19901
Intergrated Const. Services 2 Penns Way Suite 405, New Castle, DE 19720
Humes Realty 33 Bridle Ridge Court, Lewes, DE 19958
Nichols Excavation 2742 Pulaski Hwy, Newark, DE 19711
2284 Bryn Zion Road, Smyrna, DE 19904
VEI Dover Crossroads, LLC 1500 Serpentine Road, Suite 100 Baltimore MD 21
580 North Dupont Highway, Dover, DE 19901
P.O. Box 778, Dover, DE 19903
714 S OAK ST
714 S OAK ST, RUM TOWN, VA, 99201
3142 E SPRAGUE AVE, WHISKEY VALLEY, WA 99281
27488 Stanford Ave, Bowden, North Dakota
380 New York St, Redlands, CA 92373
</textarea>
<div id="output">
</div>