Regular Expression, or regex or regexp in short, is extremely and amazingly powerful in searching and manipulating text strings, particularly in processing text files. One line of regex can easily replace several dozen lines of programming codes. Regex is supported in all the scripting languages (such as Perl, Python, PHP, and JavaScript); as well as general purpose programming languages such as Java; and even word processors such as Word for searching texts. Getting started with regex may not be easy due to its geeky syntax, but it is certainly worth the investment of your time. Regex By ExamplesThis section is meant for those who need to refresh their memory. For novices, go to the next section to learn the syntax, before looking at these examples.
Example: Numbers [0-9]+ or \d+
Code Examples (Python, Java, JavaScript, Perl, PHP)Code Example in PythonSee "Python's re module for Regular Expression" for full coverage. Python supports Regex via module re. Python also uses backslash (\) for escape sequences (i.e., you need to write \\ for \, \\d for \d), but it supports raw string in the form of r'...', which ignore the interpretation of escape sequences - great for writing regex. $ python3 ...... >>> import re >>> re.findall(r'[0-9]+', 'abc123xyz') ['123'] >>> re.findall(r'[0-9]+', 'abcxyz') [] >>> re.findall(r'[0-9]+', 'abc00123xyz456_0') ['00123', '456', '0'] >>> re.findall(r'\d+', 'abc00123xyz456_0') ['00123', '456', '0'] >>> re.sub(r'[0-9]+', r'*', 'abc00123xyz456_0') 'abc*xyz*_*' >>> re.subn(r'[0-9]+', r'*', 'abc00123xyz456_0') ('abc*xyz*_*', 3)Code Example in JavaSee "Regular Expressions (Regex) in Java" for full coverage. Java supports Regex in package java.util.regex.
The output is: find() found substring "00123" starting at index 3 and ending at index 8 find() found substring "456" starting at index 11 and ending at index 14 find() found substring "0" starting at index 15 and ending at index 16 matches() found nothing lookingAt() found nothing abc**xyz456_0 abc++xyz++_++Code Example in PerlSee "Regular Expression (Regex) in Perl" for full coverage. Perl makes extensive use of regular expressions with many built-in syntaxes and operators. In Perl (and JavaScript), a regex is delimited by a pair of forward slashes (default), in the form of /regex/. You can use built-in operators:
In Perl, you can use single-quoted non-interpolating string '....' to write regex to disable interpretation of backslash (\) by Perl.
Code Example in JavaScriptSee "Regular Expression in JavaScript" for full coverage. In JavaScript (and Perl), a regex is delimited by a pair of forward slashes, in the form of /.../. There are two sets of methods, issue via a RegEx object or a String object.
Code Example in PHP[TODO] Example: Full Numeric Strings ^[0-9]+$ or ^\d+$
Example: Positive Integer Literals [1-9][0-9]*|0 or [1-9]\d*|0
Example: Full Integer Literals ^[+-]?[1-9][0-9]*|0$ or ^[+-]?[1-9]\d*|0$
Example: Identifiers (or Names) [a-zA-Z_][0-9a-zA-Z_]* or [a-zA-Z_]\w*
Example: Image Filenames ^\w+\.(gif|png|jpg|jpeg)$
Example: Email Addresses ^\w+([.-]?\w+)*@\w+([.-]?\w+)*(\.\w{2,3})+$
Exercise: Interpret this regex, which provide another representation of email address: ^[\w\-\.\+]+\@[a-zA-Z0-9\.\-]+\.[a-zA-z0-9]{2,4}$. Example: Swapping Words using Parenthesized Back-References ^(\S+)\s+(\S+)$ and $2 $1
Code Example in PythonPython keeps the parenthesized back references in \1, \2, .... Also, \0 keeps the entire match. $ python3 >>> re.findall(r'^(\S+)\s+(\S+)$', 'apple orange') [('apple', 'orange')] >>> re.sub(r'^(\S+)\s+(\S+)$', r'\2 \1', 'apple orange') 'orange apple' >>> re.sub(r'^(\S+)\s+(\S+)$', '\\2 \\1', 'apple orange') 'orange apple'Code Example in JavaJava keeps the parenthesized back references in $1, $2, ....
Example: HTTP Addresses ^http:\/\/\S+(\/\S+)*(\/)?$
Example: Regex Patterns in AngularJSThe following rather complex regex patterns are used by AngularJS in JavaScript syntax: var ISO_DATE_REGEXP = /^\d{4,}-[01]\d-[0-3]\dT[0-2]\d:[0-5]\d:[0-5]\d\.\d+(?:[+-][0-2]\d:[0-5]\d|Z)$/; var URL_REGEXP = /^[a-z][a-z\d.+-]*:\/*(?:[^:@]+(?::[^@]+)?@)?(?:[^\s:/?#]+|\[[a-f\d:]+])(?::\d+)?(?:\/[^?#]*)?(?:\?[^#]*)?(?:#.*)?$/i; var EMAIL_REGEXP = /^(?=.{1,254}$)(?=.{1,64}@)[-!#$%&'*+/0-9=?A-Z^_`a-z{|}~]+(\.[-!#$%&'*+/0-9=?A-Z^_`a-z{|}~]+)*@[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?(\.[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?)*$/; var NUMBER_REGEXP = /^\s*(-|\+)?(\d+|(\d*(\.\d*)))([eE][+-]?\d+)?\s*$/; var DATE_REGEXP = /^(\d{4,})-(\d{2})-(\d{2})$/; var DATETIMELOCAL_REGEXP = /^(\d{4,})-(\d\d)-(\d\d)T(\d\d):(\d\d)(?::(\d\d)(\.\d{1,3})?)?$/; var WEEK_REGEXP = /^(\d{4,})-W(\d\d)$/; var MONTH_REGEXP = /^(\d{4,})-(\d\d)$/; var TIME_REGEXP = /^(\d\d):(\d\d)(?::(\d\d)(\.\d{1,3})?)?$/;Example: Sample Regex in Perls/^\s+// s/\s+$// s/^\s+.*\s+$//Regular Expression (Regex) SyntaxA Regular Expression (or Regex) is a pattern (or filter) that describes a set of strings that matches the pattern. In other words, a regex accepts a certain set of strings and rejects the rest. A regex consists of a sequence of characters, metacharacters (such as ., \d, \D, \s, \S, \w, \W) and operators (such as +, *, ?, |, ^). They are constructed by combining many smaller sub-expressions. Matching a Single CharacterThe fundamental building blocks of a regex are patterns that match a single character. Most characters, including all letters (a-z and A-Z) and digits (0-9), match itself. For example, the regex x matches substring "x"; z matches "z"; and 9 matches "9". Non-alphanumeric characters without special meaning in regex also matches itself. For example, = matches "="; @ matches "@". Regex Special Characters and Escape SequencesRegex's Special CharactersThese characters have special meaning in regex (I will discuss in detail in the later sections):
Escape SequencesThe characters listed above have special meanings in regex. To match these characters, we need to prepend it with a backslash (\), known as escape sequence. For examples, \+ matches "+"; \[ matches "["; and \. matches ".". Regex also recognizes common escape sequences such as \n for newline, \t for tab, \r for carriage-return, \nnn for a up to 3-digit octal number, \xhh for a two-digit hex code, \uhhhh for a 4-digit Unicode, \uhhhhhhhh for a 8-digit Unicode. Code Example in Python$ python3 >>> import re >>> re.findall(r'a', 'abcabc') ['a', 'a'] >>> re.findall(r'=', 'abc=abc') ['='] >>> re.findall(r'\.', 'abc.com') ['.'] >>> re.findall('\\.', 'abc.com') ['.']Code Example in JavaScript[TODO] Code Example in Java[TODO] Matching a Sequence of Characters (String or Text)Sub-ExpressionsA regex is constructed by combining many smaller sub-expressions or atoms. For example, the regex Friday matches the string "Friday". The matching, by default, is case-sensitive, but can be set to case-insensitive via modifier. OR (|) OperatorYou can provide alternatives using the "OR" operator, denoted by a vertical bar '|'. For example, the regex four|for|floor|4 accepts strings "four", "for", "floor" or "4". Bracket List (Character Class) [...], [^...], [.-.]A bracket expression is a list of characters enclosed by [ ], also called character class. It matches ANY ONE character in the list. However, if the first character of the list is the caret (^), then it matches ANY ONE character NOT in the list. For example, the regex [02468] matches a single digit 0, 2, 4, 6, or 8; the regex [^02468] matches any single character other than 0, 2, 4, 6, or 8. Instead of listing all characters, you could use a range expression inside the bracket. A range expression consists of two characters separated by a hyphen (-). It matches any single character that sorts between the two characters, inclusive. For example, [a-d] is the same as [abcd]. You could include a caret (^) in front of the range to invert the matching. For example, [^a-d] is equivalent to [^abcd]. Most of the special regex characters lose their meaning inside bracket list, and can be used as they are; except ^, -, ] or \.
Name Character Classes in Bracket List (For Perl Only?)Named (POSIX) classes of characters are pre-defined within bracket expressions. They are:
For example, [[:alnum:]] means [0-9A-Za-z]. (Note that the square brackets in these class names are part of the symbolic names, and must be included in addition to the square brackets delimiting the bracket list.) Metacharacters ., \w, \W, \d, \D, \s, \SA metacharacter is a symbol with a special meaning inside a regex.
Examples: \s\s \S\S\s \s+ \S+\s\S+Backslash (\) and Regex Escape SequencesRegex uses backslash (\) for two purposes:
Take note that in many programming languages (C, Java, Python), backslash (\) is also used for escape sequences in string, e.g., "\n" for newline, "\t" for tab, and you also need to write "\\" for \. Consequently, to write regex pattern \\ (which matches one \) in these languages, you need to write "\\\\" (two levels of escape!!!). Similarly, you need to write "\\d" for regex metacharacter \d. This is cumbersome and error-prone!!! Occurrence Indicators (Repetition Operators): +, *, ?, {m}, {m,n}, {m,}A regex sub-expression may be followed by an occurrence indicator (aka repetition operator):
For example: The regex xy{2,4} accepts "xyy", "xyyy" and "xyyyy". ModifiersYou can apply modifiers to a regex to tailor its behavior, such as global, case-insensitive, multiline, etc. The ways to apply modifiers differ among languages. In Perl, you can attach modifiers after a regex, in the form of /.../modifiers. For examples: m/abc/i m/abc/gIn Java, you apply modifiers when compiling the regex Pattern. For example, Pattern p1 = Pattern.compile(regex, Pattern.CASE_INSENSITIVE); Pattern p2 = Pattern.compile(regex, Pattern.MULTILINE); Pattern p3 = Pattern.compile(regex, Pattern.DOTALL);The commonly-used modifer modes are:
Greediness, Laziness and Backtracking for Repetition OperatorsGreediness of Repetition Operators *, +, ?, {m,n}: The repetition operators are greedy operators, and by default grasp as many characters as possible for a match. For example, the regex xy{2,4} try to match for "xyyyy", then "xyyy", and then "xyy". Lazy Quantifiers *?, +?, ??, {m,n}?, {m,}?, : You can put an extra ? after the repetition operators to curb its greediness (i.e., stop at the shortest match). For example, input = "The <code>first</code> and <code>second</code> instances" regex = <code>.*</code> matches "<code>first</code> and <code>second</code>" But regex = <code>.*?</code> produces two matches: "<code>first</code>" and "<code>second</code>"Backtracking: If a regex reaches a state where a match cannot be completed, it backtracks by unwinding one character from the greedy match. For example, if the regex z*zzz is matched against the string "zzzz", the z* first matches "zzzz"; unwinds to match "zzz"; unwinds to match "zz"; and finally unwinds to match "z", such that the rest of the patterns can find a match. Possessive Quantifiers *+, ++, ?+, {m,n}+, {m,}+: You can put an extra + to the repetition operators to disable backtracking, even it may result in match failure. e.g, z++z will not match "zzzz". This feature might not be supported in some languages. Position Anchors ^, $, \b, \B, \<, \>, \A, \ZPositional anchors DO NOT match actual character, but matches position in a string, such as start-of-line, end-of-line, start-of-word, and end-of-word.
Capturing Matches via Parenthesized Back-References & Matched Variables $1, $2, ...Parentheses ( ) serve two purposes in regex:
These back-references (or capturing groups) are stored in special variables $1, $2, … (or \1, \2, ... in Python), where $1 contains the substring matched the first pair of parentheses, and so on. For example, (\S+)\s+(\S+) creates two back-references which matched with the first two words. The matched words are stored in $1 and $2 (or \1 and \2), respectively. Back-references are important to manipulate the string. Back-references can be used in the substitution string as well as the pattern. For examples, s/(\S+) (\S+)/$2 $1/; re.sub(r'(\S+) (\S+)', r'\2 \1', inStr) s/(\w+) $1/$1/; re.sub(r'(\w+) \1', r'\1', inStr)(Advanced) Lookahead/Lookbehind, Groupings and ConditionalThese feature might not be supported in some languages. Positive Lookahead (?=pattern)The (?=pattern) is known as positive lookahead. It performs the match, but does not capture the match, returning only the result: match or no match. It is also called assertion as it does not consume any characters in matching. For example, the following complex regex is used to match email addresses by AngularJS: ^(?=.{1,254}$)(?=.{1,64}@)[-!#$%&'*+/0-9=?A-Z^_`a-z{|}~]+(\.[-!#$%&'*+/0-9=?A-Z^_`a-z{|}~]+)*@[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?(\.[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?)*$The first positive lookahead patterns ^(?=.{1,254}$) sets the maximum length to 254 characters. The second positive lookahead ^(?=.{1,64}@) sets maximum of 64 characters before the '@' sign for the username. Negative Lookahead (?!pattern)Inverse of (?=pattern). Match if pattern is missing. For example, a(?=b) matches 'a' in 'abc' (not consuming 'b'); but not 'acc'. Whereas a(?!b) matches 'a' in 'acc', but not abc. Positive Lookbehind (?<=pattern)[TODO] Negative Lookbehind (?<!pattern)[TODO] Non-Capturing Group (?:pattern)Recall that you can use Parenthesized Back-References to capture the matches. To disable capturing, use ?: inside the parentheses in the form of (?:pattern). In other words, ?: disables the creation of a capturing group, so as not to create an unnecessary capturing group. Example: [TODO] Named Capturing Group (?<name>pattern)The capture group can be referenced later by name. Atomic Grouping (>pattern)Disable backtracking, even if this may lead to match failure. Conditional (?(Cond)then|else)[TODO] UnicodeThe metacharacters \w, \W, (word and non-word character), \b, \B (word and non-word boundary) recongize Unicode characters. [TODO] Regex in Programming LanguagesPython: See "Python re module for Regular Expression" Java: See "Regular Expressions in Java" JavaScript: See "Regular Expression in JavaScript" Perl: See "Regular Expressions in Perl" PHP: [Link] C/C++: [Link] REFERENCES & RESOURCES Last modified: November, 2018 |