What kind of formal languages can modern regex engines parse?

Here on SO people sometimes say something like "you cannot parse X with regular expressions, because X is not a regular language". From my understanding however, modern regular expressions engines can match more than just regular languages in Chomsky's sense. My questions:

given a regular expression engine that supports

• backreferences
• lookaround assertions of unlimited width
• recursion, like (?R)

what kind of languages can it parse? Can it parse any context-free language, and if not, what would be the counterexample?

(To be precise, by "parse" I mean "build a single regular expression that would accept all strings generated by the grammar X and reject all other strings").

Add.: I'm particularly interested to see an example of a context-free language that modern regex engines (Perl, Net, python regex module) would be unable to parse.

I recently wrote a rather long article on this topic: The true power of regular expressions.

To summarize:

• Regular expressions with support for recursive subpattern references can match all context-free languages (e.g a^n b^n).
• Regular expressions with lookaround assertions and subpattern references can match at least some context-sensitive languages (e.g. ww and a^n b^n c^n).
• If the assertions have unlimited width (as you say), then all context-sensitive grammars can be matched. I don't know any regex flavor though that does not have fixed-width restrictions on lookbehind (and at the same time supports subpattern references).
• Regular expressions with backreferences are NP-complete, so any other NP problem can be solved using regular expressions (after applying a polynomial-time transformation).

Some examples:

• Matching the context-free language {a^n b^n, n>0}:

  /^(a(?1)?b)$/
  # or
  /^ (?: a (?= a* (\1?+ b) ) )+ \1 $/x
  

• Matching the context-sensitive language {a^n b^n c^n, n>0}:

  /^
      (?=(a(?-1)?b)c)
      a+(b(?-1)?c)
  $/x
  # or
  /^ (?: a (?= a* (\1?+ b) b* (\2?+ c) ) )+ \1 \2 $/x
  

Modern regex engines can certainly parse a bigger set of languages than the regular languages set. So said, none of the four classic Chomsky sets are exactly recognized by regexes. All regular languages are clearly recognized by regexes. There are some classic context-free languages that cannot be recognized by regexes, such as the balanced parenthesis language a^n b^n, unless backreferences with counting are available. However, a regex can parse the language ww which is context-sensitive.

Actually, regular expressions in formal language theory are only lightly related to regexes. Matching regexes with unlimited backreference is NP-Complete in the most general case, so all pattern matching algorithms for powerful enough regexes are exponential, at least in the general case. However most times for most input they are quite fast. It is known that matching context-free languages is at most something faster than n^3, so there are some languages in regexes that are not context-free (like ww) but not all context-free languages can be parsed by regexes. Type 0 languages are non-decidable in general, son regexes don't get there.

So as a not very conclusive conclusion, regexes can parse a broad set of languages that include all regular languages, and some context-free and context-sensitive, but it is not exactly equal to any of those sets. There are other categories of languages, and other taxonomies, where you could find a more precise answer, but no taxonomy that includes context-free languages as a proper subset in a hierarchy of languages can provide a single language exactly recognized by regexes, because regexes only intersect in some part with context-free languages, and neither is a proper subset of the other.

You can read about regexes in An Introduction to Language And Linguistics By Ralph W. Fasold, Jeff Connor-Linton P.477

Chomsky Hierarchy:

Type0 >= Type1 >= Type2 >= Type3

Computational Linguistics mainly features Type 2 & 3 Grammars

• Type 3 grammars:

–Include regular expressions and finite state automata (aka, finite state machines)

–The focal point of the rest of this talk

• Type 2 grammars:

–Commonly used for natural language parsers

–Used to model syntactic structure in many linguistic theories (often supplemented by other mechanisms)

–We will play a key roll in the next talk on parsing.

most XMLs like Microsoft DGML (Directed Graph Markup Language) that has inter-relational links are samples that Regex are useless.

and this three answers may be useful:

1 - does-lookaround-affect-which-languages-can-be-matched-by-regular-expressions

2 - regular-expressions-arent

3 - where-do-most-regex-implementations-fall-on-the-complexity-scale