Working with Unicode

Unicode is a standard for representing a wide variety of characters from different writing systems in computers. In the context of regular expressions, Unicode refers to the ability to create patterns that can match characters beyond the standard ASCII character set.

This enables regular expressions to work with symbols, special characters, and text in different languages that are represented using the Unicode standard. Various programming languages, including PHP, support Unicode in regular expressions, often through specific syntax or modifiers, allowing for more flexible and inclusive text processing.

Unicode Modifier

In PHP, you can work with Unicode characters in regular expressions by using the u modifier. This allows you to match characters from various languages and special symbols.

If you don't enable Unicode in a regular expression, it means that the pattern will only understand the standard ASCII characters, typically the English alphabet, and some punctuation. It won't correctly understand or match characters from other languages.

With Unicode Enabled (u modifier)

$pattern = '/\w+/u'; // Unicode enabled
 
$string = "こんにちは, world!";
 
preg_match_all($pattern, $string, $matches);
 
print_r($matches[0]);
// Output: Array ( [0] => こんにちは [1] => world )

Here, \w+ matches both English and Japanese characters, so it finds two words: "こんにちは" and "world".

Without Unicode Enabled

$pattern = '/\w+/'; // Unicode NOT enabled
 
$string = "こんにちは, world!";
 
preg_match_all($pattern, $string, $matches);
 
print_r($matches[0]);
// Output: Array ( [0] => world )

Here, \w+ only understands English alphabet characters, so it only finds one word: "world". The Japanese characters are treated as if they don't form a word, so they're ignored by the pattern.

In other words, without enabling Unicode, the regular expression is like a person who only understands English trying to read a book with multiple languages. They would only pick out the English words and miss the rest. By enabling Unicode, you allow the pattern to "understand" and match characters from many different languages.

Unicode in Ranges

If you're working with sets and ranges, it's especially important to add the u modifier to your regular expression. Otherwise, you might get unexpected results.

Suppose you want to find specific fancy characters, like 𝒳 or 𝒴, in a string using regular expressions. You might try something like this:

$string = "𝒳";
 
$pattern = '/[𝒳𝒴]/';
 
$result = preg_match($pattern, $string, $matches);
 
print_r($matches); // Output shows a strange character, like [�]

The result of this regular expression would produce �. Why does this happen? Well, these special characters (𝒳 and 𝒴) are made up of two "parts" called "surrogates." But the regular expression engine, by default, doesn't know that. It thinks 𝒳 and 𝒴 are actually four different things:

• The first half of 𝒳.
• The second half of 𝒳.
• The first half of 𝒴.
• The second half of 𝒴.

So, instead of looking for 𝒳 or 𝒴, it's looking for these four "half-parts." And when it finds one of these half-parts, it doesn't know how to show it properly, so you see a strange character like �.

It's like trying to find a matching pair of gloves but only looking at the left gloves or the right gloves separately. The matching doesn't work if you don't consider both parts of the pair together.

In the case of regular expressions, you would need to enable special handling for these types of characters, called "Unicode support," to correctly match them. Without that, the regular expression gets confused by these special two-part characters.

If we add u modifier, the behavior will be correct:

$string = "𝒳";
 
$pattern = '/[𝒳𝒴]/u';
 
$result = preg_match($pattern, $string, $matches);
 
print_r($matches); // 𝒳

Key Takeaways

• In PHP, Unicode support in regular expressions is not enabled by default. You'll need to explicitly indicate you want to work with Unicode.
• To enable Unicode support, you can use the u modifier in your regular expression pattern. E.g., /pattern/u.
• Some Unicode characters are made of two "halves." Without the u modifier, regular expressions might treat these as separate characters, leading to unexpected results.
• Using the Unicode modifier allows for more accurate matching of Unicode characters, including those outside the ASCII range.