Even experienced Perl hackers might need more than a second glance to fully grasp this snippet, so let's go over it a bit.

Regex operator context

  "([^"\\]*(\\.[^"\\]*)*)",?|([^,]+),?|,

Frankly, looking at the expression is not meaningful until you also consider how it is used. In this case, it is applied via the match operator, using the /g modifier, as the conditional of a while. This is discussed in detail in the meat of the chapter, but the crux of it is that the match operator behaves differently depending how and where it is used. In this case, the body of the while loop is executed once each time the regex matches in $text. Within that body are available any $&, $1, $2, etc., set by each respective match.

Details on the regular expression itself

 "([^"\\]*(\\.[^"\\]*)*)",?

This is our friend from Chapter 5 to match a doublequoted string, here with ,? appended. The marked parentheses add no meaning to the regular expression itself, so they are apparently used only to capture text to $1. This alternative obviously deals with doublequoted fields of the CSV data.

 ([^,]+),?

This matches a non-empty sequence of non-commas, optionally followed by a comma. Like the first alternative, the parentheses are used only to capture matched text -- this time everything up to a comma (or the end of text). This alternative deals with non-quoted fields of the CSV data.

 ,  

Not much to say here -- it simply matches a comma.

These are easy enough to understand as individual components, except perhaps the significance of the ,?, which I'll get to in a bit. However, they do not tell us much individually -- we need to look at how they are combined and how they work with the rest of the program.

How the expression is actually applied

The three alternatives represent the three types of fields: quoted, unquoted, and empty. You'll notice that there's nothing in the second alternative to stop it from matching a quoted field. There's no need to disallow it from matching what the first alternative can match since Perl's Traditional NFA's non-greedy alternation guarantees the first alternative will match whatever it should, never leaving a quoted field for the second alternative to match against our wishes.

You'll also notice that by whichever alternative the first field is matched, the expression always matches through to the field-separating comma. This has the benefit of leaving the current position of the /g modifier right at the start of the next field. Thus, when the while plus m/.../g combination iterates and the regular expression is applied repeatedly, we are always sure to begin the match at the start of a field. This ``keeping in synch'' concept can be quite important in many situations where the /g modifier is used. In fact, it is exactly this reason why I made sure the first two alternatives ended with ,?. (The question mark is needed because the final field on the line, of course, will not have a trailing comma.) We'll definitely be seeing this keeping-in-synch concept again.

Now that we can match each field, how do we fill @fields with the data from each match? Let's look at $1 and such. If the field is quoted, the first alternative matches, capturing the text within the quotes to $1. However, if the field is unquoted, the first alternative fails and the second one matches, leaving $1 undefined and capturing the field's text to $3. Finally, on an empty field, the third alternative is the one that matches, leaving both $1 and $3 undefined. This all crystallizes to:

push(@fields, defined($1) ? $1 : $3);

The marked section says ``Use $1 if it is defined, or $3 otherwise.'' If neither is defined, the result is $3's undef, which is just what we want from an empty field. Thus, in all cases, the value that gets added to @fields is exactly what we desire, and the combination of the while loop, the /g modifier, and keeping-in-synch allows us to process all the fields.

Well, almost all the fields. If the last field is empty (as demonstrated by a line-ending comma), our program accounts for it not with the main regex, but after the fact with a separate line to add an undef to the list. In these cases, you might think that we can just let the main regular expression match the nothingness at the end of the line. This would work for lines that do end with an empty field, but would tack on a phantom empty field to lines that don't, since all lines have nothingness at the end, so to speak.

Regular Expressions and The Perl Way

• Judging from traffic in the Perl newsgroups and my private mail, parsing CSV data is a fairly common task. It is a problem that would likely be tackled by using brute force character-by-character analysis with languages such as C and Pascal, yet in Perl is best approached quite differently.

• It is not a problem that is solved simply with a single regular-expression match or regex search-and-replace, but one that brings to bear various intertwined features of the language. The problem first appears that it might call for split, but upon closer examination, that avenue turns out to be a dead end.

• It shows some potential pitfalls when the use of regular expressions is so closely knit with the rest of the processing. For example, although we would like to consider each of the three alternatives almost independently, the knowledge that the first alternative has two sets of parentheses must be reflected by the use of $3 to access the parentheses of the second alternative. A change in the first alternative that adds or subtracts parentheses must be reflected by changing all related occurrences of $3 -- occurrences which could be elsewhere in the code some distance away from the actual regular expression.

• It's also an example of reinventing the wheel. The standard Perl (Version 5) library module Text::ParseWords provides the quotewords routine,⁸ so:
  use Text::ParseWords;

:

@fields = quotewords(',', 0, $text);
  is all you really need.

  8

  Not to dilute my point, but I should point out that at the time of this writing, there are bugs in the quotewords routine. Among them, it strips the final field if it is the number zero, does not recognize trailing empty lines, and does not allow escaped items (except escaped quotes) within quoted fields. Also, it invokes an efficiency penalty on the whole script (=>277). I have contacted the author, and these concerns will hopefully be fixed in a future release.

  
  

  Of course, being able to solve a problem by hand is a good skill to have, but when efficiency isn't at an absolute premium, the readability and maintainability of using a standard library function is most appealing. Perl's standard library is extensive, so getting to know it puts a huge variety of both high- and low-level functionality at your fingertips.

Perl Unleashed

Version 5, Perl5 for short, was officially released in October 1994 and represented a major upgrade. Much of the language was redesigned, and many regular expression features were added or modified. One problem Larry Wall faced when creating the new features was the desire for backward compatibility. There was little room for Perl's regular expression language to grow, so he had to fit in the new notations where he could. The results are not always pretty, with many new constructs looking ungainly to novice and expert alike. Ugly yes, but as we will see, extremely powerful.

Perl4 vs. Perl5

Those comments are actually part of the regular expression... much more readable, don't you think? As we'll see, various other features of Perl5 make this solution more appealing -- we'll re-examine this example again in ``Putting It All Together'' (=>290) once we've gone over enough details to make sense of it all.

I'd like to concentrate on Perl5, but ignoring Perl4 ignores a lingering reality. I'll mention important Perl4 regex-related differences using markings that look like this: . This indicates that the first Perl4 note, related to whatever comment was just made, can be found on page 305. Since Perl4 is so old, I won't feel the need to recap everything in Perl4's manpage -- for the most part, the Perl4 notes are short and to the point, targeting those unfortunate enough to have to maintain code for both versions, such as a site's Perlmaster. Those just starting out with Perl should definitely not be concerned with an old version like Perl4.

Perl5 vs. Perl5

Around the same time, Larry added the (?#...) construct to allow comments to be embedded within a regex. A few months later, though, after discussions in the newsgroup, a raw `#' was also made to start a comment if the /x modifier were used. This appeared in version 5.002.¹⁰ There were other bug fixes and changes as well -- if you are using an early release, you might run into incompatibilities as you follow along in this book. I recommend version 5.002 or later.

As this second printing goes to press, version 5.004 is about to hit beta, with a final release targeted for Spring 1997. Regex-related changes in the works include enhanced locale support, modified pos support, and a variety of updates and optimizations. (For example, Table 7-10 will likely become almost empty.) Once 5.004 is officially released, this book's home page (see Appendix A) will note the changes.

Regex-Related Perlisms

• context An important Perl concept is that many functions and operators respond to the context they're used in. For instance, Perl expects a scalar value as the conditional of the while loop -- the main match operator in the CSV example would have behaved quite differently had it been used where Perl was expecting a list.

• dynamic scope Most programming languages support the concept of local and global variables, but Perl provides an additional twist with something known as dynamic scoping. Dynamic scoping temporarily ``protects'' a global variable by saving a copy and automatically restoring it later. It's an intriguing concept that's important for us because it affects $1 and other match-induced side effects.

• string processing Perhaps surprising to those familiar only with traditional programming languages like C or Pascal, string ``constants'' in Perl are really highly functional and dynamic operators. You can even call a function from within a string! Perl regular expressions usually receive very similar treatment, and this creates some interesting effects that we'll investigate.

Expression Context

Consider the two assignments:

$s = expression one;
@a = expression two;

Because $s is a simple scalar variable (holds a single value, not a list), it expects a simple scalar value, so the first expression, whatever it may be, finds itself in a scalar context. Similarly, because @a is an array variable and expects a list of values, the second expression finds itself in a list context. Even though the two expressions might be exactly the same, they might (depending on the case) return completely different values, and cause completely different side effects while they're at it.

Sometimes, the type of an expression doesn't exactly match the type of value expected of it, so Perl does one of two things to make the square peg fit into a round hole: 1) it allows the expression to respond to its context, returning a type appropriate to the expectation, or 2) it contorts the value to make it fit.

Allowing an expression to respond to its context

Many Perl constructs respond to their context, and the regex operators are no different. The match operator m/.../, for example, sometimes returns a simple true/false value, and sometimes a list of certain match results. All the details are found later in this chapter.

Contorting an expression

$var = ($this, &is, 0xA, 'list');

Some words used to describe how other languages deal with this issue are cast, promote, coerce, and convert, but I feel they are a bit too consistent (boring?) to describe Perl's attitude in this respect.

Dynamic Scope and Regex Match Effects

Global and private variables

Dynamically scoped values

• Originally, Perl did not provide true local variables, only global variables. Even if you just needed a temporary variable, you had no choice but to use a global, and so risked stepping on someone else's toes if your choice for a temporary variable was unlucky enough to conflict with what was being used elsewhere. Having Perl save and restore a copy of your intended variable isolated your changes of the global variable, in time, to your temporary use.

• If you use global variables to represent ``the current state'', such as the name of a file being processed, you might want to have a routine modify that state only temporarily. For example, if you process a file-include directive, you want the ``current filename'' to reflect the new file only until it's done, reverting back to the original when the directive has been processed. Using dynamic scoping, you can have the save-and-restore done automatically.

The first use listed is less important these days because Perl now offers truly local variables via the my directive. Using my creates a new variable completely distinct from and utterly unrelated to any other variable anywhere else in the program. Only the code that lies between the my to the end of the enclosing block can have direct access to the variable.

The extremely ill-named function local creates a new dynamic scope. Let me say up front that the call to local does not create a new variable. Given a global variable, local does three things:

1. saves an internal copy of the variable's value; and

2. copies a new value into the variable (either undef, or a value assigned to the local); and

3. slates the variable to have the original value restored when execution runs off the end of the block enclosing the local.

This means that ``local'' refers only to how long any changes to the variable will last. The global variable whose value you've copied is still visible from anywhere within the program -- if you make a subroutine call after creating a new dynamic scope, that subroutine (wherever it might be located in the script) will see any changes you've made. This is just like any normal global variable. The difference here is that when execution of the enclosing block finally ends, the previous value is automatically restored.

An automatic save and restore of a global variable's value -- that's pretty much all there is to local. For all the misunderstanding that has accompanied local, it's no more complex than the snippet on the right of Table 7-3 illustrates.

(As a matter of convenience, you can assign a value to local($SomeVar), which is exactly the same as assigning to $SomeVar in place of the undef assignment. Also, the parentheses can be omitted to force a scalar context.)

References to $SomeVar while within the block, or within a subroutine called from the block, or within a signal handler invoked while within the block -- any reference from the time the local is called to the time the block is exited -- references `My·Value'. If the code in the block (or anyone else, for that matter) modifies $SomeVar, everyone (including the code in the block) sees the modification, but it is lost when the block exits and the original copy is automatically restored.

As a practical example, consider having to call a function in a poorly written library that generates a lot of Use of uninitialized value warnings. You use Perl's -w option, as all good Perl programmers should, but the library author apparently didn't. You are exceedingly annoyed by the warnings, but if you can't change the library, what can you do short of stop using -w altogether? Well, you could set a local value of $^W, the in-code debugging flag (the variable name ^W can be either the two characters, caret and `W', or an actual control-W character):

The call to local saves an internal copy of the previous value of the global variable $^W, whatever it might have been. Then that same $^W receives the new value of zero that we immediately scribble in. When unruly_function is executing, Perl checks $^W and sees the zero we wrote, so doesn't issue warnings. When the function returns, our value of zero is still in effect.

So far, everything appears to work just as if you didn't use local. However, when the block is exited right after the subroutine returns, the saved value of $^W is restored. Your change of the value was local, in time, to the lifetime of the block. You'd get the same effect by making and restoring a copy yourself, as in Table 7-3, but local conveniently takes care of it for you.

For completeness, let's consider what happens if I use my instead of local.¹³ Using my creates a new variable with an initially undefined value. It is visible only within the lexical block it is declared in (that is, visible only by the code written between the my and the end of the enclosing block). It does not change, modify, or in any other way refer to or affect other variables, including any global variable of the same name that might exist. The newly created variable is not visible elsewhere in the program, including from within unruly_function. In our example snippet, the new $^W is immediately set to zero but is never again used or referenced, so it's pretty much a waste of effort. (While executing unruly_function and deciding whether to issue warnings, Perl checks the unrelated global variable $^W.)

A better analogy: clear transparencies

This analogy is actually much closer to reality than the original ``an internal copy is made'' description. Using local doesn't have Perl actually make a copy, but instead puts your new value earlier in the list of those checked whenever a variable's value is accessed (that is, it shadows the original). Exiting a block removes any shadowing values added since the block started. Values are added manually, with local, but some variables have their values automatically dynamically scoped. Before getting into that important regex-related concern, I'd like to present an extended example illustrating manual dynamic scoping.

An extended dynamic-scope example

Dynamic Scope Example

When a do-this command is found (at [7]), the DoThis function is called to process it. You can see at [1] that the function refers to the global variables $filename, $., and $command. The DoThis function doesn't know (nor care), but the values of these variables that it sees were written in ProcessFile.

The #include command's processing begins with the filename being plucked from the line at [5]. After making sure the file hasn't been processed already, we call ProcessFile recursively, at [6]. With the new call, the global variables $filename, $command, and $., as well as the filehandle FILE, are again overlaid with a transparency that is soon updated to reflect the status and commands of the second file. When commands of the new file are processed within ProcessFile and the two subroutines, $filename and friends are visible, just as before.

Nothing at this point appears to be different from straight global variables.

The benefits of dynamic scoping are apparent when the second file has been processed and the related call of ProcessFile exits. When execution falls off the block at [9], the related local transparencies laid down at [3] are removed, restoring the original file's values of $filename and such. This includes the filehandle FILE now referring to the first file, and no longer to the second.

Finally, let's look at %HaveRead, used to keep track of files we've seen ([4] and [5]). It is specifically not dynamically scoped because we really do need it to be global across the entire time the script runs. Otherwise, included files would be forgotten each time ProcessFile exits.

Regex side-effects and dynamic-scoping

To see the benefit of this, realize that each call to a subroutine involves starting a new block. For these variables, that means a new dynamic scope is created. Because the values before the block are restored when the block exits (that is, when the subroutine returns), the subroutine can't change the values that the caller sees.

As an example, consider:

if (m/(...)/)
{
    &do_some_other_stuff();
    print "the matched text was $1.\n";
}

Because the value of $1 is dynamically scoped automatically upon entering each block, this code snippet neither cares, nor needs to care, whether the function do_some_other_stuff changes the value of $1 or not. Any changes to $1 by the function are contained within the block that the function defines, or perhaps within a sub-block of the function. Therefore, they can't affect the value this snippet sees with the print after the function returns.

The automatic dynamic scoping can be helpful even when not so apparent:

if ($result =~ m/ERROR=(.*)/) {
   warn "Hey, tell $Config{perladmin} about $1!\n";
}

(The standard library module Config defines an associative array %Config, of which the member $Config{perladmin} holds the email address of the local Perlmaster.) This code could be very surprising if $1 were not automatically dynamically scoped. You see, %Config is actually a tied variable, which means that any reference to it involves a behind-the-scenes subroutine call. Config's subroutine to fetch the appropriate value when $Config{...} is used invokes a regex match. It lies between your match and your use of $1, so it not being dynamically scoped would trash the $1 you were about to use. As it is, any changes in the $Config{...} subroutine are safely hidden by dynamic scoping.

Dynamic scoping vs. lexical scoping

Special Variables Modified by a Match

14	As described on page 209, this  notation means that Perl4 note #1 is found on page 305.

$&

A copy of the text successfully matched by the regex. This variable (along with $` and $' below) is best avoided. (See ``Unsociable $& and Friends'' on page 273.) $& is never undefined after a successful match.

$`

A copy of the target text in front of (to the left of) the match's start. When used in conjunction with the /g modifier, you sometimes wish $` to be the text from start of the match attempt, not the whole string. Unfortunately, it doesn't work that way. If you need to mimic such behavior, you can try using \G([\x00-\xff]*?) at the front of the regex and then refer to $1. $` is never undefined after a successful match.

$'

A copy of the target text after (to the right of) the successfully matched text. After a successful match, the string "$`$&$'" is always a copy of the original target text.¹⁵ $' is never undefined after a successful match.

15	Actually, if the original target is undefined, but the match successful (unlikely, but possible), "$`$&$'" would be an empty string, not undefined. This is the only situation where the two differ.

$1, $2, $3, etc.

The text matched by the 1^st, 2^nd, 3^rd, etc., set of capturing parentheses. (Note that $0 is not included here -- it is a copy of the script name and not related to regexes). These are guaranteed to be undefined if they refer to a set of parentheses that doesn't exist in the regex, or to a set that wasn't actually involved in the match.

These variables are available after a match, including in the replacement operand of s/.../.../. But it makes no sense to use them within the regex itself. (That's what \1 and friends are for.) See ``Using $1 Within a Regex?'' on page 219.

The difference between (\w+) and (\w)+ can be seen in how these variables are set. Both regexes match exactly the same text, but they differ in what is matched within the parentheses. Matching against the string tubby, the first results in $1 having tubby, while the latter in it having y: the plus is outside the parentheses, so each iteration causes them to start capturing anew.

Also, realize the difference between (x)? and (x?). With the former, the parentheses and what they enclose are optional, so $1 would be either x or undefined. But with (x?), the parentheses enclose a match -- what is optional are the contents. If the overall regex matches, the contents matches something, although that something might be the nothingness x? allows. Thus, with (x?) the possible values of $1 are x and an empty string.

Perl4 and Perl5 treat unusual cases involving parentheses and iteration via star and friends slightly differently. It shouldn't matter to most of you, but I should at least mention it. Basically, the difference has to do with what $2 will receive when something like (main(OPT)?)+ matches only main and not OPT during the last successful iteration of the plus. With Perl5, because (OPT) did not match during the last successful match of its enclosing subexpression, $2 becomes (rightly, I think) undefined. In such a case, Perl4 leaves $2 as what it had been set to the last time (OPT) actually matched. Thus, with Perl4, $2 is OPT if it had matched any time during the overall match.

$+

A copy of the highest numbered $1, $2, etc. explicitly set during the match. If there are no capturing parentheses in the regex (or none used during the match), it becomes undefined. However, Perl does not issue a warning when an undefined $+ is used.

When a regex is applied repeatedly with the /g modifier, each iteration sets these variables afresh. This is why, for instance, you can use $1 within the replacement operand of s/.../.../g and have it represent a new slice of text each time. (Unlike the regex operand, the replacement operand is re-evaluated for each iteration; =>255.)

Using $1 within a regex?

A related question is whether $1 is available within a regex operand. The answer is ``Yes, but not the way you might think.'' A $1 appearing in a regex operand is treated exactly like any other variable: its value is interpolated (the subject of the next section) before the match or substitution operation even begins. Thus, as far as the regex is concerned, the value of $1 has nothing to do with the current match, but remains left over from some previous match somewhere else.

In particular, with something like s/.../.../g, the regex operand is evaluated, compiled once (also discussed in the next section), and then used by all iterations via /g. This is exactly opposite of the replacement operand, which is re-evaluated after each match. Thus, a $1 within the replacement operand makes complete sense, but in a regex operand it makes virtually none.

``Doublequotish Processing'' and Variable Interpolation

Strings as operators

$month = "January";

$message = "Report for $month:";

'Report for ' . $month . ':'

(In a general expression, a lone period is Perl's string-concatenation operator; concatenation is implicit within a doublequoted string.)

The doublequotes are really operators that enclose operands. A string such as

"the month is $MonthName[&GetMonthNum]!"

'the month is ' . $MonthName[&GetMonthNum] . '!'

One of Perl's unique features is that a doublequoted string doesn't have to actually be delimited by doublequotes. The qq/.../ notation provides the same functionality as "...", so qq/Report for $month:/ is a doublequoted string. You can also choose your own delimiters. The following example uses qq{...} to delimit the doublequoted string:

warn qq{"$ARGV" line $.: $ErrorMessage\n};

Singlequoted strings use q/.../, rather than qq/.../. Regular expressions use m/.../ and s/.../.../ for match and substitution, respectively. The ability to pick your own delimiters for regular expressions, however, is not unique to Perl: ed and its descendants have supported it for over 25 years.

Regular expressions as strings, strings as regular expressions

$field = "From";
 :
if ($headerline =~ m/^$field:/) {
 :
}

The marked section is taken as a variable reference and is replaced by the variable's value, resulting in ^From: being the regular expression actually used. ^$field: is the regex operand; ^From: is the actual regex after cooking. This seems similar to what happens with doublequoted strings, but there are a few differences (details follow shortly), so it is often said that it receives ``doublequotish'' processing.

One might view the mathematical expression ($F - 32) * 5/9 as
  Divide( Multiply( Subtract($F, 32), 5), 9 )
to show how the evaluation logically progresses. It might be instructive to see $headerline =~ m/^$field:/ presented in a similar way:
  RegexMatch( $headerline, DoubleQuotishProcessing(``^$field:'') )

Thus, you might consider ^$field: to be a match operand only indirectly, as it must pass through doublequotish processing first. Let's look at a more involved example:

This method of building up the variable $string and then using it in the regular expression is much more readable than writing the whole regex directly:

if (m/^name=(?:'[^'\\]*(?:\\.[^'\\]*)*'|"[^"\\]*(?:\\.[^"\\]*)*")$/o) {

Several important issues surround this method of building regular expressions within strings, such as all those extra backslashes, the /o modifier, and the use of (?:...) non-capturing parentheses. See ``Matching an Email Address'' (=>294) for a heady example. For the moment, let's concentrate on just how a regex operand finds its way to the regex search engine. To get to the bottom of this, let's follow along as Perl parses the program snippet:

$header =~ m/^\Q$dir\E # base directory
             \/        # separating slash
             (.*)      # grab the rest of the filename
            /xgm;

For the example, we'll assume that $dir contains `~/.bin'.

The \Q...\E that wraps the reference to $dir is a feature of doublequotish string processing that is particularly convenient for regex operands. It puts a backslash in front of most symbol characters. When the result is used as a regex, it matches the literal text the \Q...\E encloses, even if that literal text contains what would otherwise be considered regex metacharacters (with our example of `~/.bin', the first three characters are escaped, although only the dot requires it).

Also pertinent to this example are the free whitespace and comments within the regex. Starting with Perl version 5.002, a regex operand subject to the /x modifier can use whitespace liberally, and may have raw comments. Raw comments start with # and continue to the next newline (or to the end of the regex). This is but one way that doublequotish processing of regex operands differs from real doublequoted strings: the latter has no such /x modifier.

Figure 7-1 on page 223 illustrates the path from unparsed script, to regex operand, to real regex, and finally to the use in a search. Not all the phases are necessarily done at the same time. The lexical analysis (examining the script and deciding what's a statement, what's a string, what's a regex operand, and so on) is done just once when the script is first loaded (or in the case of eval with a string operand, each time the eval is re-evaluated). That's the first phase of Figure 7-1. The other phases can be done at different times, and perhaps even multiple times. Let's look at the details.

Phase A -- identifying the match operand

Figure 7-1: Perl parsing, from program text to regular expression engine

Phase B -- doublequotish processing

Although similar, differences between regex operands and doublequoted strings become apparent in this phase. Phase B realizes it is working with a regex operand, so processes a few things differently. For example, \b and \3 in a doublequoted string always represent a backspace and an octal escape, respectively. But in a regex, they could also be the word-boundary metacharacter or a backreference, depending on where in the regex they're located -- Phase B therefore leaves them undisturbed for the regex engine to later interpret as it sees fit. Another difference involves what is and isn't considered a variable reference. Something like $| will always be a variable reference within a string, but it is left for the regex engine to interpret as the metacharacters $ and |. Similarly, a string always interprets $var[2-7] as a reference to element -5 of the array @var (which means the fifth element from the end), but as a regex operand, it is interpreted as a reference to $var followed by a character class. You can use the ${...} notation for variable interpolation to force an array reference if you wish:

  ${var[2-7]}.

Due to variable interpolation, the result of this phase can depend on the value of variables (which can change as the program runs). In such a case, Phase B doesn't take place until the match code is reached during runtime. ``Perl Efficiency Issues'' (=>265) expands on this important point.

As discussed in the match and substitution operator sections later in this chapter, using a singlequote as the regex-operand delimiter invokes singlequotish processing. In such a case, this Phase B is skipped.

Phase C -- /x processing

$single = qq{'(...regex here...)'  # for singlequoted strings};

This value makes its way to $string and then to the regex operand. After Phase B, the operand is (with the intended comment bold, but the real comment underlined):  ^name=(?:'(...)'·#·for·singlequoted·strings|"(...)")$

Surprise! The comment intended only for $single ends up wiping out what follows because we forgot to end the comment with a newline.

If, instead, we use:

$single = qq{'(...regex here...)'  # for singlequoted strings\n};

Phase D -- regex compilation

Perl's Regex Flavor

Quantifiers -- Greedy and Lazy

The non-greedy versions are examples of the ``ungainly looking'' additions to the regex flavor that appeared with Perl5. Traditionally, something like *? makes no sense in a regex. In fact, in Perl4 it is a syntax error. So, Larry was free to give it new meaning. There was some thought that the non-greedy versions might be **, ++, and such, which has certain appeal, but the problematic notation for {min,max} led Larry to choose the appended question mark. This also leaves ** and the like for future expansion.

Non-greedy efficiency

It's not common that you have a free choice between using greedy and non-greedy quantifiers, since they have such different meanings, but if you do, the choice is highly dependent upon the situation. Thinking about the backtracking that either must do with the data you expect should lead you to a choice. For an example, I recommend my article in the Autumn 1996 (Volume 1, Issue 3) issue of The Perl Journal in which I take one simple problem and investigate a variety of solutions, including those with greedy and non-greedy quantifiers.

A non-greedy construct vs. a negated character class

First of all, of course, is that if the /s modifier (=>234) is not used, the dot of .+? doesn't match a newline, while the negated class in [^>]+ does.

A bigger problem that doesn't always manifest itself clearly can be shown with a simple text markup system that uses <...> to indicate emphasis, as with:

Fred was very, <very> angry. <Angry!> I tell you.

The point to remember is that the negated class in [^>]*> never matches `>', while the non-greedy construct in .*?> does if that is what it takes to achieve a match. If nothing after the lazy construct could force the regex engine to backtrack, it's not an issue. However, as the exclamation point of this example illustrates, the desire for a match allows the lazy construct past a point you really don't want it to (and that a negated character can't) exceed.

The non-greedy constructs are without a doubt the most powerful Perl5 additions to the regex flavor, but you must use them with care. A non-greedy .*? is almost never a reasonable substitute for [^...]* -- one might be proper for a particular situation, but due to their vastly different meaning, the other is likely incorrect.

Grouping

Uniquely, Perl provides two kinds of parentheses: the traditional (...) for both grouping and capturing, and new with Perl5, the admittedly unsightly (?:...) for grouping only. With (?:...), the ``opening parenthesis'' is really the three-character sequence `(?:', while the ``closing parenthesis'' is the usual `)'.

Like the notation for the non-greedy quantifiers, the sequence `(?' was previously a syntax error. Starting with version 5, it is used for a number of regex language extensions, of which (?:...) is but one. We'll meet the others soon.

Capturing vs. non-capturing parentheses

• Efficient matching from bypassing the (perhaps non-trivial) overhead involved with capturing text you don't intend to use.

• The possibility of greater internal optimizations because subexpressions that don't need to be isolated for capturing purposes might be converted to something more efficient by the regex engine. (?:return-to|reply-to):· and re(?:turn-to:·|ply-to:·), for example, are logically equivalent, but the latter is faster, with both matches and failures. (If you're not convinced, work through an attempt to match against forward-·and·reply-to·fields...) Perl's regex engine does not currently take advantage of this, but it could in the future.

• Ease in building a usable regular expression in a string.

This last item is probably the one that provides the most benefit from a user's point of view: Recall in the CSV example how because the first alternative used two sets of parentheses, those in the second alternative captured to $3 (=>204; =>207). Any time the first alternative changes the number of parentheses it uses, all subsequent references must be changed accordingly. It's a real maintenance nightmare that non-capturing parentheses can greatly alleviate.²⁰

For related reasons, the number of capturing parentheses becomes important when m/.../ is used in a list context, or at any time with split. We will see specific examples in each of the relevant sections later (=>252, 264).

Lookahead

Similarly, the negative lookahead (?!subexpression) is true when the subexpression does not match. Superficially, negative lookahead is the logical counterpart to a negated character class, but there are two major differences:

• A negated character class must match something to be successful, and consumes that text during the match. Negative lookahead is successful when it can't match anything. The second and third examples illustrate this.

• A character class (negated or not) matches a single character in the target text. Lookahead (positive or negative) can test an arbitrarily complex regular expression.

A few examples of lookahead:

Bill(?=·The·Cat|·Clinton)

Matches Bill, but only if followed by `·The·Cat' or `·Clinton'.

\d+(?!\.)

Matches a number if not followed by a period.

\d+(?=[^.])

Matches a number if followed by something other than a period. Make sure you understand the difference between this and the previous item -- consider a number at the very end of a string. Actually, I'll just put that question to you now. Which of these two will match the string `OH·44272', and where? ¤ Think carefully, then turn the page to check your answer.

Positive lookahead vs. negative lookahead

¤ Answer to the question on page 228 Both \d+(?!\.) and \d+(?=[^.]) can match `OH·44272'. The first matches OH·44272, while the second OH·44272. Remember, greediness always defers in favor of an overall match. Since \d+(?=[^.]) requires a non-period after the matched number, it will give up part of the number to become the non-period if need be. We don't know what these might be used for, but they should probably be written as \d+(?![\d.]) and \d+(?=[^.\d]).

^(?![A-Z]*$)[a-zA-Z]*$

Matches if the item contains only letters, but not all uppercase.

^(?=.*?this)(?=.*?that)

A rather ingenious (if not somewhat silly) way of checking whether this and that can match on the line. (A more logical solution that is mostly comparable is the dual-regex /this/ && /that/.)²¹

21

There are quite a variety of ways to implement ``this and that'', and as many ways to compare them. Major issues of such a comparison include understanding exactly what is matched, match efficiency, and understandability. I have an article in the Autumn 1996 (Volume 1, Issue 3) issue of The Perl Journal that examines this in great detail, providing, I hope, additional insight about what the regex engine's actions mean in the real world. By the way, the ingenious solution shown here comes from a post by Randal Schwartz, and it fared rather well in the benchmarks.

Other illustrative examples can be found at ``Keeping the Match in Synch with Expectations'' (=>237). For your entertainment, here's a particularly heady example copied from ``Adding Commas to a Number'' (=>292):

Lookahead parentheses do not capture text, so they do not count as a set of parentheses for numbering purposes. However, they may contain raw parentheses to capture the phantomly matched text. Although I don't recommend this often, it can perhaps be useful. For example, (.*?)(?=<(strong|em)\s*>), matches everything up to but not including a <strong> or <em> HTML tag on the line. The consumed text is put into $1 (and $& as well, of course), while the `strong' or `em' that allows the match to stop is placed into $2. If you don't need to know which tag allows the match to stop successfully, the ...(strong|em)... is better written as ...(?:strong|em)... to eliminate the needless capturing. As another example, I use an appended (?=(.*)) on page =>277 as part of mimicking $'. (Due to the $&-penalty, we want to avoid $' if at all possible; =>273.)

Using capturing parentheses within a negative lookahead construct makes absolutely no sense, since the negative lookahead construct matches only when its enclosed subexpression does not.

The perlre manpage rightly cautions that lookahead is very different from lookbehind. Lookahead ensures that the condition (matching or not matching the given subexpression) is true starting at the current location and looking, as normal, toward the right. Lookbehind, were it supported, would somehow look back toward the left.

For example, (?!000)\d\d\d means ``so long as they're not 000, match three digits,'' which is a reasonable thing to want to do. However, it is important to realize that it specifically does not mean ``match three digits that are not preceded by 000.'' This would be lookbehind, which is not supported in Perl or any other regex flavor that I know of. Well, actually, I suppose that a leading anchor (either type: string or word) can be considered a limited form of lookbehind.

You often use lookahead at the end of an expression to disallow it from matching when followed (or not followed) by certain things. Although its use at the beginning of an expression might well indicate a mistaken attempt at lookbehind, leading lookahead can sometimes be used to make a general expression more specific. The 000 is one example, and the use of (?!0+\.0+\.0+\.0+\b) to disallow null IP addresses in Chapter 4 (=>125) is another. And as we saw in Chapter 5's ``A Global View of Backtracking'' (=>149), lookahead at the beginning of an expression can be an effective way to speed up a match.

Otherwise, be careful with leading negative lookahead. The expression \w+ happily matches the first word in the string, but prepending (?!cat) is not enough to ensure ``the first word not beginning with cat.'' (?!cat)\w+ can't match at the start of cattle, but can still match cattle. To get the desired effect, you need additional precautions, such as \b(?!cat)\w+.

Comments within a regular expression

(?#...) appeared in version 5.000, but as of 5.002, the /x modifier enables the unadorned # comment that runs to the next newline (or the end of the regex), pretty much just like in regular code. (We saw an example of this earlier in the commafication snippet on page 229.) The /x modifier also causes most whitespace to be ignored, so you can write the

$text =~ m/"([^"\\]*(\\.[^"\\]*)*)",?|([^,]+),?|,/g

The underlying regular expression is exactly the same. As with (?#...) comments, the regex's closing delimiter still counts, so these comments are not entirely free-form either. As with most other regex metacharacters, the # and whitespace metacharacters that become active with /x are not available within a character class, so there can be no comments or ignored whitespace within classes. Like other regex metacharacters, you can escape # and whitespace within an /x

Other (?...) constructs

String Anchors

Logical lines vs. raw text

However, if a string contains embedded newlines, it's reasonable to consider the one string to be a collection of multiple logical lines. There are many ways to create strings with embedded newlines (such as "this\nway") -- when applying a regex to such a string, however it might have acquired its data, should ^Subject: find `Subject:' at the start of any of the logical lines, or only at the start of the entire multi-line string?

Perl allows you to do either. Actually, there are four distinct modes, summarized in Table 7-5

When there are no embedded newlines in the target string, all modes are equal.²⁴ You'll noticed that Table 7-5 doesn't mention...

24	The line anchor optimization does become a bit less effective in the two multi-line modes because the transmission must still search for embedded newlines at which to reapply the regex (=>158).

• how the target text might acquire multiple logical lines, that is, embedded newlines (it's irrelevant);

• when you may apply a match to such a string (anytime you like); or

• whether \n, an appropriate octal escape, or even [^x] can match a newline (they always can).

In fact, meticulous study of Table 7-5 reveals that these modes are concerned only with how the three metacharacters, caret, dollar, and dot, consider newlines.

The default behavior of caret, dollar, and dot

/m and the ill-named ``multi-line mode''

Let me say again:

The /m modifier influences only how ^ and $ treat newlines.

The /m modifier affects only regular-expression matching, and in particular, only the caret and dollar metacharacters. It has nothing whatsoever to do with anything else. Perhaps ``multi-line mode'' is better named ``line-anchors-notice-embedded-newlines mode.'' The /m and multi-line mode are debatably the most misunderstood simple features of Perl, so please allow me to get on the soapbox for a moment to make it clear that the /m modifier has nothing to do with...

• Whether you can work with data that has newlines in it. You can always work with any type of data. \n always matches a newline without regard to multi-line mode, or the lack thereof.

• Whether dot (or anything else) does or does not match a newline. The /s modifier (discussed on page 234) influences dot, but the /m modifier does not. The /m modifier influences only whether the anchors match at a newline. Conceptually, this is not entirely unrelated to whether dot matches a newline, so I often bring the description of dot's default behavior under the umbrella term ``multi-line mode,'' but the relationship is circumstantial.

• How data, multi-line or not, finds its way to the target string. Multi-line mode is often used in conjunction with other useful features of Perl, but this relationship, too, is circumstantial. A prime example is $/, the input-record separator variable. If $/ is set to the empty string, it puts <> and the like into paragraph mode in which it will return, as a single string, all the lines until (and including) the next blank line. If you set $/ to undef, Perl goes into file slurp mode where the entire (rest of the) file is returned, all in a single string.²⁵

  25

  A non-regex warning: It's a common mistake to think that undefining $/ causes the next <> read to return ``the rest of the input'' (or if used at the start of the script, ``all the input''). ``File slurp'' mode is still a file slurp mode. If there are multiple files for <> to read, you still need one call per file. If you really want to get all the input, use join('',<>).

  It's convenient to use multi-line mode in conjunction with a special setting of $/, but neither has any relationship to the other.

The /m modifier was added in Perl5 -- Perl4 used the now-obsolete special variable $* to indicate multi-line mode for all matches. When $* is true, caret and dollar behave as if /m were specified. This is less powerful than explicitly indicating that you want multi-line mode on a per-use basis, so modern programs should not use $*. However, modern programmers should worry if some old or unruly library does, so the complement of /m is the /s modifier.

Single-line mode

Clean multi-line mode

Explicit start- and end-of-string

Newline is always special in one respect

Eliminating warnings about $*

{ local($^W) = 0;  eval '$* = 1' }

This turns off warnings while $* is modified, yet when done leaves warnings on if they were on -- I explained this technique in detail in ``Dynamic Scope'' (=>213).

/m vs. (?m), /s vs. /m

Although you may be tempted to want something like (?m)...(?s)...(?m)... to change the mode mid-stream, the line mode is an all-or-nothing characteristic for the entire match. It makes no difference how the mode is specified.

Combining both /s and /m has /m taking precedence with respect to caret and dollar. Still, the use or non-use of /m has no bearing on whether a dot matches a newline or not -- only the explicit use of /s changes dot's default behavior. Thus, combining both modes creates the clean multi-line mode.

All these modes and permutations might seem confusing, but Table 7-6 should keep things straight. Basically, they can be summarized with ``/m means multi-line, /s means dot matches newline.''

All other constructs are unaffected. \n always matches a newline. A character class can be used to match or exclude the newline characters at any time. An inverted character class such as [^x] always matches a newline (unless \n is included, of course). Keep this in mind if you want to change something like .* to the seemingly more restrictive [^...]*.

Multi-Match Anchor

An example with \G

03824531449411615213441829503544272752010217443235

So, it should be apparent that changing \d\d\d\d\d to 44\d\d\d in an attempt to find only ZIP codes starting with 44 is silly -- once a match attempt fails, the transmission bumps along one character, thus putting the match for the 44 out of synch with the start of each ZIP code. Using 44\d\d\d incorrectly finds ...5314494116... as the first match.

You could, of course, put a caret or \A at the head of the regex, but they allow a target ZIP code to match only if it's the first in the string. We need to keep the regex engine in synch manually by writing our regex to pass over undesired ZIP codes as needed. The key here is that it must pass over full ZIP codes, not single characters as with the automatic bump-along.

Keeping the match in synch with expectations

(?:[^4]\d\d\d\d|\d[^4]\d\d\d)*...

This brute-force method actively skips ZIP codes that start with something other than 44. (Well, it's probably better to use [1235-9] instead of [^4], but as I said earlier, I am assuming properly formatted data.) By the way, we can't use (?:[^4][^4]\d\d\d)*, as it does not pass over undesired ZIP codes like 43210.

(?:(?!44)\d\d\d\d\d)*...

This similar method actively skips ZIP codes that do not start with 44. This English description sounds virtually identical to the one above, but when rendered into a regular expression looks quite different. Compare the two descriptions and related expressions. In this case, a desired ZIP code (beginning with 44) causes (?!44) to fail, thus causing the skipping to stop.

(?:\d\d\d\d\d)*?...

This method skips ZIP codes only when needed (that is, when a later subexpression describing what we do want fails). Because of the minimal-matching, (?:\d\d\d\d\d) is not even attempted until whatever follows has failed (and is repeatedly attempted until whatever follows finally does match, thus effectively skipping only what is absolutely needed).

Combining this last method with (44\d\d\d) gives us

@zips = m/(?:\d\d\d\d\d)*?(44\d\d\d)/g;

This regex can work with /g because we know each match always leaves the ``current match position'' at the start of the next ZIP code, thereby priming the next match (via /g) to start at the beginning of a ZIP code as the regex expects. You might remember that we used this same keeping-in-synch technique with the CSV example (=>206).

Hopefully, these techniques are enlightening, but we still haven't seen the \G we're supposed to be looking at. Continuing the analysis of the problem, we find a use for \G quickly.

Maintaining synch after a non-match as well

Let's look at our sample data again:

03824531449411615213441829503544272|7|5|2|010217443235

Our regex works smoothly so long it's applied at the start of a ZIP code, but the transmission's bump-along defeats it. This is where \G comes in:

@zips = m/\G(?:\d\d\d\d\d)*?(44\d\d\d)/g;

\G matches the point where the previous /g match ended (or the start of the string during the first attempt). Because we crafted the regex to explicitly end on a ZIP code boundary, we're assured that any subsequent match beginning with \G will start on that same ZIP code boundary. If the subsequent match fails, we're done for good because a bump-along match is impossible -- \G requires that we start from where we had last left off. In other words, this use of \G effectively disables the bump along.

In fact, the transmission is optimized to actually disable the bump-along in common situations. If a match must start with \G, a bump-along can never yield a match, so it is done away with altogether. The optimizer can be tricked, so you need to be careful. For example, this optimization isn't activated with something like \Gthis|\Gthat, even though it is effectively the same as \G(?:this|that) (which it does optimize).

\G in perspective

@zips = m/(?:\d\d\d\d\d)*?(44\d\d\d)(?:(?!44)\d\d\d\d\d)*/g;

After the last desired ZIP code has been matched, the added subexpression consumes the rest of the string, if any, and the m/.../g is finished.

These methods work, but frankly, it is often prudent to take some of the work out of the regular expression using other regular expressions or other language features. The following two examples are easier to understand and maintain:

@zips = grep {defined} m/(44\d\d\d)|\d\d\d\d\d/g;
@zips = grep {m/^44/} m/\d\d\d\d\d/g;

In Perl4, you can't do it all in one regex because it doesn't have many of the constructs we've used, so you need to use a different approach regardless.

Priming a /g match

@nums = $data =~ m/\d+/g;

The match of <xx> is in a scalar context, so /g doesn't perform multiple matches (=>253). Rather, it sets the pos of $data, the ``end of the last match'' position where the next /g-governed match of the same data will start. I call this technique priming the /g. Once done, m/\d+/g picks up the match at the primed point. If <xx> can't match in the first place, the subsequent m/\d+/g starts at the beginning of the string, as usual.

Two important points allow this example to work. First, the first match is in a scalar context. In a list context, the <xx> is applied repeatedly until it fails, and failure of a /g-governed match resets pos to the start of the string. Secondly, the first match must use /g. Matches that don't use /g never access pos.

And here is something interesting: Because you can assign to pos, you can prime the /g start manually:

pos($data) = $i if $i = index($data,"<xx>"), $i > 0;
@nums = $data =~ m/\d+/g;

If index finds <xx> in the string, it sets the start of the next /g-governed match of $data to begin there. This is slightly different from the previous example -- here we prime it to start at the <xx>, not after it as we did before. It turns out, though, that it doesn't matter in this case.

This example is simple, but you can imagine how these techniques could be quite useful if used carefully in limited cases. You can also imagine them creating a maintenance nightmare if used carelessly.

Word Anchors

An important warning about \b

However, if the $item were `$3.75', the regex would become \b\$3\.75\b, which requires a word boundary before the dollar sign. (The only way a word boundary can fall before a \W character like a dollar sign is if a word ends there.) I'd prepended \b with the thought that it would force the match to start where $item started its own word. But now that the start of $item can't start its own word, (`$' isn't matched by \w, so cannot possibly start a word) the \b is an impediment. The regex doesn't even match `... is $3.75 plus ...'.

We don't even want to begin the match if the character before can match \w, but this is lookbehind, something Perl doesn't have. One way to address this issue is to add \b only when the $item starts or ends with something matching \w:

This ensures that \b doesn't go where it will cause problems, but it still doesn't address the situations where it does (that is, where the $item begins or ends with a \W character). For example, an $item of -998 still matches in `800-998-9938'. If we don't mind matching more text than we really want (something that's not an option when embedded within a larger subexpression, and often not an option when applied using the /g modifier), we can use the simple, but effective, (?:\W|^)\Q$item\E(?!\w).

Dealing with a lack of separate start and end anchors

s/\b(?!\w).*\b(?=\w)//

Convenient Shorthands and Other Notations

The \n and other shorthands probably seem familiar -- these machine dependent (=>72) notations for common control characters are also available in doublequoted strings. I feel it is important to maintain the mental distinction that these regular expression metacharacters themselves are not available within strings, but that strings just happen to have their own metacharacters which parallel the regex ones in this area (=>41).

If you compare m/(?:\r\n)+$/ with

$regex = "(\r\n)+";
m/$regex$/;

On a different front, both a regex and a doublequoted string convert \055 to a dash (055 is the ASCII code for a dash), but if the regex does the conversion, it doesn't see the result as a metacharacter. The string doesn't either, but the resulting [+-/*] that the regex eventually receives has a dash that will be interpreted as part of a class range. Surprise!

Finally, \d is not a known metasequence to a doublequoted string, so it simply removes the backslash. The regex sees d. Surprise!

I want to emphasize all this because you must know what are and aren't metacharacters (and whose metacharacters they are, and when, and in what order they're processed) when building a string that you later intend to use as a regex. It can certainly be confusing at first. An extended example is presented in ``Matching an Email Address'' (=>294).

Perl and the POSIX locale

If compiled with appropriate libraries, though, Perl uses the ``is this a letter?'' routines (isalpha, isupper, and so on), as well as the mappings between uppercase and lowercase. This affects /i and the items mentioned in Table 7-8 (=>245). It also allows \w, \W, \s, \S, (but not \d), and word boundaries to respond to the locale. (Perl's regex flavor, however, explicitly does not support [:digit:] and the other POSIX bracket expression character classes listed on page 80.)

Related standard modules

Byte notations

Perl strings also allow one-digit octal escapes, but Perl regexes generally don't because something like \1 is usually taken as a backreference. In fact, multiple-digit backreferences are possible if there are enough capturing parentheses. Thus, \12 is a backreference if the expression has at least 12 sets of capturing parentheses, an octal escape (for decimal 10) otherwise. Upon reading a draft of this chapter, Wayne Berke offered a suggestion that I wholeheartedly agree with: never use a two-digit octal escape such as \12, but rather the full three-digit \012. Why? Perl will never interpret \012 as a backreference, while \12 is in danger of suddenly becoming a backreference if the number of capturing parentheses warrants.

There are two special cases: A backreference within a character class makes no sense so single-digit octal escapes are just peachy within character classes (which is why I wrote generally don't in the previous paragraph). Secondly, \0 is an octal escape everywhere, since it makes no sense as a backreference.

Bytes vs. characters, newline vs. linefeed

Character Classes

As I've cautioned throughout this book, metacharacters recognized within a character class are distinct from those recognized outside. Perl is no exception to this rule, although many metacharacters have the same meaning in both situations. In fact, the dual-personality of \b aside, everything in Table 7-7 is also supported within character classes, and I find this extremely convenient.

Many normal regex metacharacters, however, are either unspecial or utterly different within character classes. Things like star, plus, parentheses, dot, alternation, anchors and the like are all meaningless within a character class. We've seen that \b, \3, and ^ have special meanings within a class, but they are unrelated to their meaning outside of a class. Both - and ] are unspecial outside of a class, but special inside (usually).

Character classes and non-ASCII data

Lacking real POSIX locale support, octal and hexadecimal escapes are quite useful for working with non-ASCII text. For instance, when working with the Latin-1 (ISO-8859-1) encoding popular on the Web, you need to consider that u might also appear as ù, ú, û, or ü (using the character encodings \xf9 through \xfc). Thus, to match any of these u's, you can use [u\xf9-\xfc]. The uppercase versions are encoded from \xd9 through \xdc, so a case-insensitive match is [uU\xf9-\xfc\xd9-\xdc] (Using the /i modifier applies only to the ASCII u, so it is just as easy to include U directly and save ourselves the woe of the /i penalty; =>278.)

Sorting is a practical use of this technique. Normally, to sort @items, you simply use sort @items, but this sorts based on raw byte values and puts u (with ASCII value \x75) far away from û and the like. If we make a copy of each item to use as a sort key (conveniently associating them with an associated array), we can modify the key so that it will work directly with sort and yield the results we want. We can then map back to the unmodified key while keeping the sorted order.

Here's a simple implementation:

(lc is a convenient Perl5 feature, but the example is easily rewritten in Perl4.) In reality, this is only the beginning of a solution since each language has its own particular sorting requirements, but it's a step in the right direction. Another step is to use the I18N::Collate module mentioned

Modification with \Q and Friends: True Lies

For most practical purposes, they appear to be normal regex metacharacters. When used in a regex operand, the doublequotish processing of Figure 7-1's Phase B handles them, so they normally never reach the regex engine (=>222). But because of cases where it does matter, I call these Table 7-8 items second-class metacharacters.

Second-class metacharacters

• are effective only during the ``top-level'' scan of the operand

• affect only characters in the compiled regular expression

This is because they are recognized only during Phase B of Figure 7-1, not later after the interpolation has taken place. If you don't understand this, you would be confused when the following didn't work:

$ForceCase = $WantUpper ? '\U' : '\L';
if (m/$ForceCase$RestOfRegex/) {
 :

Because the \U or \L of $ForceCase are in the interpolated text (which is not processed further until Phase C), nothing recognizes them as special. Well, the regex engine recognizes the backslash, as always, so \U is treated as the general case of an unknown escape: the escape is simply ignored. If $RestOfRegex contains Path and $WantUpper is true, the search would be for the literal text UPath, not PATH as had been desired.

Another effect of the ``one-level'' rule is that something like m/([a-z])...\U\1 doesn't work. Ostensibly, the goal is to match a lowercase letter, eventually followed by an uppercase version of the same letter. But \U works only on the text in the regular expression itself, and \1 represents text matched by (some other part of the) expression, and that's not known until the match attempt is carried out. (I suppose a match attempt can be considered a ``Phase E'' of Figure 7-1.)

The Match Operator

• Specifying the match operator itself (the regex operand)

• Specifying the target string to match against (the target operand)

• The match's side effects

• The value returned by a match

• Outside influences that affect the match

The Perl regular-expression match is an operator that takes two operands (a target string operand and a regex operand) and returns a value, although exactly what kind of value depends on context. Also, there are optional modifiers which change how the match is done. (Actually, I suppose these can be considered operands as well.)

Match-Operand Delimiters

For example, to apply the expression ^/(?:[^/]+/)+Perl$ using the match operator with standard delimiters, m/^\/(?:[^\/]+\/)+Perl$/ is required. As presented in Phase A of Figure 7-1, a closing delimiter that appears within the regular expression must be escaped to hide the character's delimiter status. (These escapes are bold in the example.) Characters escaped for this reason are passed through to the regex as if no escape were present. Rather than suffer from backslashitis, it's more readable to use a different delimiter -- two examples of the same regex are m!^/(?:[^/]+/)+Perl$! and m,^/(?:[^/]+/)+Perl$,. Other common delimiters are m|...|, m#...#, and m%...%.

There are several special-case delimiters:

• The four special cases m(...), m{...}, m[...], and m<...> have different opening and closing delimiters, and may be nested. Because parentheses and square brackets are so prevalent in regular expressions, m(...) and m[...] are probably not so appealing, but the other two can be. In particular, with the /x modifier, something such as the following becomes possible:

  m{
      regex  # comments
      here   # here
  }x;

  When these special-case delimiters are nested, matching pairs can appear unescaped. For example, m(^/((?:[^/]+/)+)Perl$) is valid, although visually confusing.

• If a singlequote is used as the delimiter, the regex operand undergoes singlequotish interpolation, as opposed to the normal doublequotish interpolation discussed in detail throughout this chapter. This means that Figure 7-1's Phase B is skipped, turning off variable interpolation and rendering inactive the items listed the Table 7-8. This could be useful if an expression has many $ and @ whose escaping becomes too untidy.

• If question marks are used as the delimiter, a very special kind of match operation is done. Normally, the match operator returns success whenever its regex operand can match its target string operand, but with this special ?-delimited match, it returns success only with the first successful match. Subsequent uses fail, even if they could otherwise match, until reset in the same package is called.  With a list-context /g match, ``returning only once'' still means ``all the matches'' as normal.

  This can be useful if you want a particular match to be successful only once during a loop. For example, when processing an email header, you could use:

  $subject = $1 if m?^Subject: (.*)?;

  Once you've matched the subject once, there's no reason to bother checking for it on every line that follows. Still, realize that if, for whatever reason, the header has more than one subject line, m?...? matches only the first, while m/.../ matches them all -- once the header has been processed, m?...? will have left $subject with the data from the first subject line, while m/.../ will have left it with data from the last.

The substitution operator s/.../.../ has other special delimiters that we'll talk about later (=>255); the above are the special cases for the match operator.

As yet another special case, if the delimiter is either the commonly used slash, or the special match-only-once question mark, the m itself becomes optional. It's common to use /.../ for matches.

Finally, the generic pattern target =~ expression (with no delimiters and no m) is supported. The expression is evaluated as a generic Perl expression, taken as a string, and finally fed to the regex engine. This allows you to use something like $text =~ &GetRegex() instead of the longer:

my $temp_regex = &GetRegex();
... $text =~ m/$temp_regex/ ...

Similarly, using $text =~ "...string..." could be useful if you wanted real doublequote processing, rather than the doublequoteish processing discussed earlier. But frankly, I would leave this to an Obfuscated Perl Contest.

The default regex

The default regex is never recompiled (even if the original had been built via variable interpolation without the /o modifier). This can be used to your advantage for creating efficient tests. There's an example in ``The /o Modifier'' (=>270).

Match Modifiers

• how the regex operand is to be interpreted (/o =>268;  /x =>223)

• how the regex engine considers the target text (/i =>42;  /m, /s =>233)

• how the engine applies the regex (/g =>253)

You can group several modifier letters together and place them in any order after the closing delimiter,³² whatever it might be. For example, m/<code>/i applies the regular expression <code> with the /i modifier, resulting in a case-insensitive match. Do keep in mind that the slash is not part of the modifier -- you could write this example as m|<code>|i or perhaps m{<code>}i or even m<<code>>i.

As discussed earlier (=>231), the modifiers /x, /i, /m, and /s can also appear within a regular expression itself using the (?...) construct. Allowing these options to be indicated in the regular expression directly is extremely convenient when one operator is applying different expressions at different times (usually due to variable interpolation). When you use /i, for example, every application of an expression via the match operator in question is case-insensitive. By allowing each regular expression to choose its own options, you get more general-purpose code.

A great example is a search engine on a Web page that offers a full Perl regular-expression lookup. Most search engines offer very simple search specifications that leaves power users frustrated, so offering a full regex option is appealing. In such a case, the user could use (?i) and the like, as needed, without the CGI having to provide special options to activate these modifiers.

m/.../g with a regex that can match nothingness

Perl version 5.000 is broken in this respect, and indeed repeats until you run out of electrons. Perl4 and later versions of Perl5 work, although differently. They both begin the match where the previous one left off unless the previous match was of no text, in which case a special bump-along happens and the match is re-applied one character further down. Thus, each match after the first is guaranteed to progress down the string at least one character, and the infinite loop is avoided.

Except when used with the substitution operator, Perl5 takes the additional step of disallowing any match that ends at the same position as the previous match -- in such a case the automatic one-character heave-ho is done (if not already at the end of the string). This difference from Perl4 can be important -- Table 7-9 shows a few simple examples. (This table is not light reading by any means -- it might take a while to absorb.) Things are quite different with the substitution operator, but that's saved for ``The Substitution Operator'' (=>255).

Specifying the Match Target Operand

Since the whole ``expr =~ m/.../'' is an expression itself, you can use it wherever an expression is allowed. Some examples (each separated by a wavy line):

If the target operand is the variable $_, you can omit the ``$_ =~'' altogether. In other words, the default target operand is $_.

Something like $line =~ m/regex/ means ``Apply regex to the text in $line, ignoring the return value but doing the side effects.'' If you forget the `~', the resulting $line = m/regex/ becomes ``Apply regex to the text in $_, returning a true or false value that is then assigned to $line.'' In other words, the following are the same:

$line =        m/regex/
$line = ($_ =~ m/regex/)

You can also use !~ instead of =~ to logically negate the return value. (Return values and side effects are discussed soon.) $var !~ m/.../ is effectively the same as not ($var =~ m/.../). All the normal side effects, such as the setting of $1 and the like, still happen. It is merely a convenience in an ``If this doesn't match'' situation. Although you can use !~ in a list context, it doesn't make much sense.

Other Side Effects of the Match Operator

Two involve ``invisible status.'' First, if the match is specified using m?...?, a successful match dooms future matches to fail, at least until the next reset (=>247). Of course, if you use m?...? explicitly, this particular side effect is probably the main effect desired. Second, the regex in question becomes the default regex until the dynamic scope ends, or another regex matches (=>248).

Finally, for matches with the /g modifier, the pos of the target string is updated to reflect the index into the string of the match's end. (A failed attempt always resets pos.) The next /g-governed match attempt of the same string starts at that position, unless:

• the string is modified. (Modifying a string resets its pos.)

• pos is assigned to. (The match will start at that position.)

• the previous match did not match at least one character.

As discussed earlier (=>249), in order to avoid an infinite loop, a successful match that doesn't actually match characters causes the next match to begin one character further into the string. During the begun-further attempt, pos properly reflects the end of the match because it's at the start of the subsequent attempt that the anti-loop movement is done. During such a next match, \G still refers to the true end of the previous match, so cannot be successful. (This is the only situation where \G doesn't mean ``the start of the attempt.'')

Match Operator Return Value

Scalar context, without the /g modifier

On failure, it returns an empty string (which is considered a Boolean false).

List context, without the /g modifier

($year, $month, $day) = $date =~ m{^ (\d+) / (\d+) / (\d+) $}x;

The three matched numbers are then available in the three variables (and $1 and such as well). There is one element in the return-value list for each set of capturing parentheses, or an empty list upon failure. Of course, it is possible for a set or sets to have not been part of a match, as is certainly guaranteed with one in m/(this)|(that)/. List elements for such sets exist, but are undefined. If there are no sets of capturing parentheses to begin with, a successful list-context non-/g match returns the list (1).

Expanding a bit on the date example, using a match expression as the conditional of an if (...) can be useful. Because of the assignment to ($year, ...), the match operator finds itself in a list context and returns the values for the variables. But since that whole assignment expression is used in the scalar context of the if's conditional, it is then contorted into the count of items in the list. Conveniently, this is interpreted as a Boolean false if there were no matches, true if there were.

List context, with the /g modifier

alias  jeff      jfriedl@ora.com
alias  perlbug   perl5-porters@perl.org
alias  prez      president@whitehouse

( 'jeff', 'jfriedl@ora.com', 'perlbug',
  'perl5-porters@perl.org', 'prez', 'president@whitehouse' )

%alias = $text =~ m/^alias\s+(\S+)\s+(.+)/mg;

Scalar context, with the /g modifier

This is quite convenient as the conditional of a while loop. Consider:

while ($ConfigData =~ m/^(\w+)=(.*)/mg) {
    my($key, $value) = ($1, $2);
      :
}

All matches are eventually found, but the body of the while loop is executed between the matches (well, after each match). Once an attempt fails, the result is false and the while loop finishes. Also, upon failure, the /g state (given by pos) is reset. Finally, be careful not to modify the target data within the loop unless you really know what you're doing: it resets pos. 

Outside Influences on the Match Operator

• context -- Has a large influence on how the match is performed, as well as on its return value and side effects.

• pos(...) -- Set explicitly, or implicitly by the /g modifier, it indicates where in the string the next /g-governed match should begin. Also, see \G in ``Multi-Match Anchor'' (=>236).

• $* -- A holdover from Perl4, can influence the caret and dollar anchors. See ``String Anchors'' (=>232).

• default regex -- Is used if the provided regex is empty (=>248).

• study -- Has no effect on what is matched or returned, but if the target string has been study'd, the match might be faster (or slower). See ``The Study Function'' (=>287).

• m?...? and reset -- affects the invisible ``has/hasn't matched'' status of m?...? operators (=>247).

Keeping your mind in context (and context in mind)

while ("Larry Curly Moe" =~ m/\w+/g) {
   print "WHILE stooge is $&.\n";
}
print "\n";

if ("Larry Curly Moe" =~ m/\w+/g) {
   print "IF stooge is $&.\n";
}
print "\n";

foreach ("Larry Curly Moe" =~ m/\w+/g) {
   print "FOREACH stooge is $&.\n";
}

It's a bit tricky. ¤ Turn the page to check your answer.

while vs. foreach vs. if

¤ Answer to the question on page 254. The results differ depending on your version of Perl: Perl4 Perl5 WHILE stooge is Larry. WHILE stooge is Curly. WHILE stooge is Moe. IF stooge is Larry. FOREACH stooge is . FOREACH stooge is . FOREACH stooge is . WHILE stooge is Larry. WHILE stooge is Curly. WHILE stooge is Moe. IF stooge is Larry. FOREACH stooge is Moe. FOREACH stooge is Moe. FOREACH stooge is Moe. Note that if the print within the foreach loop had referred to $_ rather than $&, its results would have been the same as the while's. In this foreach case, however, the result returned by the m/.../g, ('Larry', 'Curly', 'Moe'), goes unused. Rather, the side effect $& is used, which almost certainly indicates a programming mistake, as the side effects of a list-context m/.../g are not often useful.

The Substitution Operator

• The replacement operand and additional operand delimiters

• The /e modifier

• Context and return value

• Using /g with a regex that can match nothingness

The Replacement Operand

Perl normally provides true doublequoted processing of the replacement operand, although there are a few special-case delimiters. The processing happens after the match (with /g, after each match), so $1 and the like are available to refer to the proper match slice.

Special delimiters of the replacement operand are:

• As with the match operator, you can use singlequotes for the regex operand as well, but that's almost a separate issue. (If you use them for the regex operand, you must use them for the replacement operand, but not vice-versa.)

• The special match-operator once-only delimiter ?...? is not special with the substitution operator.

• Despite previous Perl5 documentation to the contrary, the use of backquotes as the delimiter is not special as it was in Perl4. In Perl4, the replacement operand first underwent doublequote interpolation, the results of which were executed as a shell command. The resulting output was then used as the replacement text. When needed (rarely), this behavior can be closely mimicked in Perl5. Compare the following for matching command names and fetching their -version strings:
  

  Perl4 :	s`version of (\w+)`$1 --version 2>&1`g;
  Both Perl4 and Perl5 :	s/version of (\w+)/`$1 --version 2>&1`/e;

  
  The marked portion is the replacement operand. In the first version, it is executed as a system command due to the special delimiter. In the second version, the backquotes are not special until the whole replacement is evaluated due to the /e modifier. The /e modifier, discussed in detail in just a moment, indicates that the replacement operand is a mini Perl snippet that should be executed, and whose final value should be used as the replacement text.

Remember that this replacement-operand processing is all quite distinct from regex-operand processing, which usually gets doublequotish processing and has its own set of special-case delimiters.

The /e Modifier

As an example, you can encode special characters of a World Wide Web URL using % followed by their two-digit hexadecimal representation. To encode all non-alphanumerics this way, you can use

$url =~ s/([^a-zA-Z0-9])/sprintf('%%%02x', ord($1))/ge;

$url =~ s/%([0-9a-f][0-9a-f])/pack("C",hex($1))/ige;

In short, pack("C", value) converts from a numeric value to the character with that value, while sprintf('%%%02x', ord(character)) does the opposite; see your favorite Perl documentation for more information. (Also, see the footnote on page 66 for more on this example.)

The moving target of interpretation

To add to the fray, if using the /g modifier as well, should `echo $$` be evaluated just once (with the result being used for all replacements), or should it be done after each match? When $1 and such appear in the replacement operand, it obviously must be evaluated on a per-match basis so that the $1 properly reflects its after-match status. Other situations are less clear. With this echo example, Perl version 5.000 does only one evaluation, while other versions before and after evaluate on a per-match basis.

/eieio

$Old_MacDonald = q#print #; $had_a_farm = (q-q:Just another Perl hacker,:-);
s/^/q[Sing it, boys and girls...],$Old_MacDonald.$had_a_farm/eieio;

The eval due to the first /e sees

q[Sing it, boys and girls...],$Old_MacDonald.$had_a_farm

Actually, this kind of construct is sometimes useful. Consider wanting to interpolate variables into a string manually (such as if the string is read from a configuration file). A simple approach uses $data =~ s/(\$[a-zA-Z_]\w*)/$1/eeg;. Applying this to `option=$var', the regex matches option=$var. The first eval simply sees the snippet $1 as provided in the replacement-operand, which in this case expands to $var. Due to the second /e, this result is evaluated again, resulting in whatever value the variable $var has at the time. This then replaces the matched `$var', in effect interpolating the variable.

I actually use something like this with my personal Web pages -- most of them are written in a pseudo Perl/HTML code that gets run through a CGI when pulled by a remote client. It allows me to calculate things on the fly, such as to remind readers how few shopping days are left until my birthday.³⁶

Context and Return Value

The return value is either the number of substitutions performed or, if none were done, an empty string. When interpreted as a Boolean (such as for the conditional of an if), the return value conveniently interprets as true if any substitutions were done, false if not.

Using /g with a Regex That Can Match Nothingness

The Split Operator

In its most simple form with simple data like this, split is as easy to understand as it is useful. However, when the use of split or the data are complicated, understanding is less clear-cut. First, I'll quickly cover some of the basics.

Basic Split

split(match operand, target string, chunk-limit operand)

(The parentheses are optional with Perl5.) Default values, discussed below, are provided for operands left off the end.

Basic match operand

There is a default match operand if one is not provided, but it is one of the complex special cases discussed later.

Target string operand

Basic chunk-limit operand

( 'IO.SYS', '225558', '95-10-03:-a-sh:optional' )

This shows that split stopped after /:/ matched twice, resulting in the requested three-chunk partition. It could have matched additional times, but that's irrelevant here because of the chunk-limit. The limit is an upper bound, so no more than that many elements will ever be returned, but note that it doesn't guarantee that many elements -- no extra are produced to ``fill the count'' if the data can't be partitioned enough to begin with. split(/:/, $text, 1234) still returns only a five-element list. Still, there is an important difference between split(/:/, $text) and split(/:/, $text, 1234) which does not manifest itself with this example -- keep this in mind for when the details are discussed later.

Remember that the chunk-limit operand is not a match-limit operand. Had it been for the example above, the three matches would have partitioned to

('IO.SYS', '225558', '95-10-03', '-a-sh:optional')

One comment on efficiency: Let's say you intended to fetch only the first few fields, such as with

($filename, $size, $date) = split(/:/, $text)

Advanced Split

Although useful, split is not straightforward to master. Some important points to consider are:

• split's match operand differs from the normal m/.../ match operator. In addition, there are several special-case match operands.

• When the match operand matches at the beginning or end of the target string, or twice in a row, it usually returns empty strings on one side or the other. Usually, but not always.

• What happens when the regex can match nothingness?

• A scalar-context split, now deprecated, stuffs the list into @_ instead of returning it.

• split behaves differently when its regex has capturing parentheses.

Split can return empty elements

:IO.SYS:225558:::95-10-03:-a-sh:

('', 'IO.SYS', '225558', '', '', '95-10-03', '-a-sh', '')

However, this is not what split(/:/, $text) returns. Surprised?

Trailing empty elements are not (normally) returned

The chunk-limit operand's second job

If you don't want to limit the number of chunks returned, but instead only want to leave trailing empty items intact, simply choose a very large limit. Also, a negative chunk-limit is taken as an arbitrarily large limit: split(/:/, $text, -1) returns all elements, including any trailing empty ones.

At the other extreme, if you want to remove all empty items, you could put grep {length} before the split. The grep lets pass only list elements with non-zero lengths (in other words, elements that aren't empty).

Advanced Split's Match Operand

• a match(-like) operator

• the special scalar '·' (a single space)

• any general scalar expression

• the default for when no operand is provided

Split's match-operator match operand

• You should not (and in some versions of Perl, cannot) specify the match operator's target string operand. Never use =~ with split, as strange things can happen. split provides the match operand with the target string operand behind the scenes.

• We've discussed the complexity of a regex that can match nothingness, such as with m/x*/ (=>249) and s/x*/.../ (=>259). Perhaps because split provides the repetition instead of the match operator, things are much simpler here.

  The one exception is that a match of nothingness at the start of the string does not produce a leading empty element. Contrast this to a match of something at the start of the string: it does produce a leading empty element. (A match of nothingness at the end of the string does leave a trailing one, though, but such trailing empty items are removed unless a large enough, or negative, chunk-limit is used.)

  For example, m/\W*/ matches `|T|h|i|s, "T|h|a|t", O|t|h|e|r!' at the places marked (either underlined, or with | for a match of nothingness). Because of this exception, the string-leading match is ignored, leaving 13 matches, resulting in the 14 element list:

  ('T', 'h', 'i', 's', 'T', 'h', 'a', 't', 'O', 't', 'h', 'e', 'r', '')

  Of course, if no chunk-limit is specified, the final empty element is removed.

• An empty regex for split does not mean ``Use the current default regex,'' but to split the target string at each character. For example, you could use

  $text = join "\b_", split(//, $text, -1);

  to put a backspace-underline sequence after each character in $text. (However, $text =~ s/(.)/$1\b_/g might be better for a variety of reasons, one of which is that it's probably easier to understand at first glance).

• The use of a regex with split doesn't affect the default regex for later match and substitution operators. Also, the variables $&, $', $1, and so on are not available from a split. A split is completely isolated from the rest of the program with respect to side-effects.

• The /g modifier is meaningless (but harmless) when used with split.

• The special regex delimiter ?...? is not special with split.

Special match operand: a string with a single space

For instance, the call split('·', "···this···is·a·····test") returns the four-element list ('this', 'is', 'a', 'test'). As a contrast to '·', consider using m/\s+/ directly. This bypasses the leading-whitespace removal and returns ('', 'this', 'is', 'a', 'test')

Finally, both of these are quite different from using m/·/, which matches each individual space and returns:

('','','','this','','','is','a','','','','','test')

Any general scalar expression as the match operand

The default match operand

Scalar-Context Split

Split's Match Operand with Capturing Parentheses

In Perl4, this was more of a pain than a useful feature because it meant you could never (easily) use a regex that happened to require parentheses merely for grouping. If grouping is the only intent, the littering of extra elements in the return list is definitely not a feature. Now that you can select the style of parentheses -- capturing or not -- it's a great feature. For example, as part of HTML processing, split(/(<[^>]*>)/) turns

...·and·<B>very·<FONT·color=red>very</FONT>·much</B>·effort...

( '...·and ', '<B>', 'very·', '<FONT·color=red>',
  'very', '</FONT>', '·much', '</B>', '·effort...' )

  (<[^>"]*("[^"]*"[^>"]*)*>)

The added set of capturing parentheses means an added element returned for each match during the split, yielding two items per match plus the normal items due to the split. Applying this to
  Please <A HREF="test">press me</A> today
returns (with descriptive comments added):

The extra elements clutter the list. Using (?:...) for the added parentheses, however, returns the regex to split usefulness, with the results being:

Perl Efficiency Issues

There are, of course, Perl-specific efficiency issues, such as the use of non-capturing parentheses unless you specifically need capturing ones. There are some much larger issues as well, and even the issue of capturing vs. non-capturing is larger than the micro-optimization explained in Chapter 5 (=>152). In this section, we'll look at this (=>276), as well as the following topics:

• There's More Than One Way To Do It Perl is a toolbox offering many approaches to a solution. Knowing which problems are nails comes with understanding The Perl Way, and knowing which hammer to use for any particular nail goes a long way toward making more efficient and more understandable programs. Sometimes efficiency and understandability seem to be mutually exclusive, but a better understanding allows you to make better choices.

• Interpolation The interpolation and compilation of regex operands is fertile ground for saving time. The /o modifier, which I haven't discussed much yet, gives you some control over when the costly re-compilation takes place.

• $& Penalty The three match side effect variables, $`, $&, and $', can be convenient, but there's a hidden efficiency gotcha waiting in store for any script that uses them, even once, anywhere. Heck, you don't even have to use them -- the entire script is penalized if one of these variables even appears in the script.

• /i Penalty You pay another penalty when you use the /i modifier. Particularly with very long target strings, it can pay to rewrite the regex to avoid using /i.

• Substitution You can get the most out of the various ways that Perl optimizes substitutions by knowing when the optimizations kick in, what they buy you, and what defeats them.

• Benchmarking When it comes down to it, the fastest program is the one that finishes first. (You can quote me on that.) Whether a small routine, a major function, or a whole program working with live data, benchmarking is the final word on speed. Benchmarking is easy and painless with Perl, although there are various ways to go about it. I'll show you the way I do it, a simple method that has served me well for the hundreds of benchmarks I've done while preparing this chapter.

• Study Since ages past, Perl has provided the study(...) function. Using it supposedly makes regexes faster, but no one really understands it. We'll see whether we can figure it out.

• -Dr Option Perl's regex-debug flag can tell you about some of the optimizations the regex engine and transmission do, or don't do, with your regexes. We'll look at how to do this and see what secrets Perl gives up.

``There's More Than One Way to Do It''

$ip = sprintf "%03d.%03d.%03d.%03d", split(/\./, $ip);

This is a fine solution, but there are certainly other ways to do the job. In the same style as the The Perl Journal article I mentioned in the footnote on page 229, let's examine various ways of achieving the same goal. This example's goal is simple and not very ``interesting'' in and of itself, yet it represents a common text-handling task. Its simplicity will let us concentrate on the differing approaches to using Perl. Here are a few other solutions:

1. $ip =~ s/(\d+)/sprintf("%03d", $1)/eg;

2. $ip =~ s/\b(\d{1,2}\b)/sprintf("%03d", $1)/eg;

3. $ip = sprintf("%03d.%03d.%03d.%03d", $ip =~ m/(\d+)/g);

4. $ip =~ s/\b(\d\d?\b)/'0' x (3-length($1)) . $1/eg;

5. $ip = sprintf("%03d.%03d.%03d.%03d",
$ip =~ m/^(\d+)\.(\d+)\.(\d+)\.(\d+)$/);

6. $ip =~ s/\b(\d(\d?)\b)/$2 eq '' ? "00$1" : "0$1"/eg;

7. $ip =~ s/\b(\d\b)/00$1/g;
$ip =~ s/\b(\d\d\b)/0$1/g;

Like the original solution, each produces the same results when given a correct IP address, but fail in different ways if given something else. If there is any chance that the data will be malformed, more care than any of these solutions provide is needed. That aside, the practical differences lie in efficiency and readability. As for readability, about the only thing that's easy to see about most of these is that they are cryptic at best.

So, what about efficiency? I benchmarked these solutions on my system with Perl version 5.003, and have listed them in order from least to most efficient. The original solution belongs somewhere between positions four and five, the best taking only 80 percent of its time, the worst about 160 percent.³⁹ But if efficiency is really important, faster methods are still available:

substr($ip,  0, 0) = '0' if substr($ip,  1, 1) eq '.';
substr($ip,  0, 0) = '0' if substr($ip,  2, 1) eq '.';
substr($ip,  4, 0) = '0' if substr($ip,  5, 1) eq '.';
substr($ip,  4, 0) = '0' if substr($ip,  6, 1) eq '.';
substr($ip,  8, 0) = '0' if substr($ip,  9, 1) eq '.';
substr($ip,  8, 0) = '0' if substr($ip, 10, 1) eq '.';
substr($ip, 12, 0) = '0' while length($ip) < 15;

This takes only half the time as the original, but at a fairly expensive toll in understandability. Which solution you choose, if any, is up to you. There are probably other ways still. Remember, ``There's more than one way to do it.''

Regex Compilation, the /o Modifier, and Efficiency

On the other hand, if the regex changes each time, it certainly makes sense for Perl to reprocess it for us. Of course, it takes longer to redo the processing, but it's a very convenient feature that adds remarkable flexibility to the language, allowing a regex to vary with each use. Still, as useful as it may be, the extra work is sometimes needless. Consider a situation where a variable that doesn't change from use to use is used to provide a regex:

The variable $regex is set just once, before the loop. The match operator that uses it, however, is inside a loop, so it is applied over and over again, once per line of <LOGFILE>. We can look at this script and know for sure that the regex doesn't change during the course of the loop, but Perl doesn't know that. It knows that the regex operand involves interpolation, so it must re-evaluate that operand each time it is encountered.

This doesn't mean the regex must be fully recompiled each time. As an intermediate optimization, Perl uses the compiled form still available from the previous use (of the same match operand) if the re-evaluation produces the same final regex. This saves a full recompilation, but in cases where the regex never does change, the processing of Figure 7-1's Phase B and C, and the check to see if the result is the same as before, are all wasted effort.

This is where the /o modifier comes in. It instructs Perl to process and compile a regex operand the first time it is used, as normal, but to then blindly use the same internal form for all subsequent tests by the same operator. The /o ``locks in'' a regex the first time a match operator is used. Subsequent uses apply the same regex even if variables making up the operand were to change. Perl won't even bother looking. Normally, you use /o as a measure of efficiency when you don't intend to change the regex, but you must realize that even if the variables do change, by design or by accident, Perl won't reprocess or recompile if /o is used.

Now, let's consider the following situation:

The first time through the inner foreach loop (of the first time through the outer while loop), the regular expression is processed and compiled, the result of which is then used for the actual match attempt. Because the /o modifier is used, the match operator in question uses the same compiled form for all its subsequent attempts. Later, in the second iteration of the outer loop, a new $regex is read from the user with the intention of using it for a new search. It won't work -- the /o modifier means to compile a regex operator's regex just once, and since it had already been done, the original regex continues to be used -- the new value of $regex is completely ignored.

The easiest way to solve this problem is to remove the /o modifier. This allows the program to work, but it is not necessarily the best solution. Even though the intermediate optimization stops the full recompile (except when the regex really has changed, the first time through each inner loop), the pre-compile processing and the check to see if it's the same as the previous regex must still be done each and every time. The resulting inefficiency is a major drawback that we'd like to avoid if at all possible.

Using the default regex to avoid work

Unfortunately, it's usually quite difficult to find something appropriate for the sample string if you don't know the regex beforehand. Remember, a successful match is required to install the regex as the default. Additionally (in modern versions of Perl), that match must not be within a dynamic scope that has already exited.

The /o modifier and eval

Notice that the entire foreach loop is within a singlequoted string, which itself is the argument to eval. Each time the string is evaluated, it is taken as a new Perl snippet, so Perl parses it from scratch, then executes it. It is executed ``in place,'' so it has access to all the program's variables just as if it had been part of the regular code. The whole idea of using eval in this way is to delay the parsing until we know what each regex is to be.

What is interesting for us is that, because the snippet is parsed afresh with each eval, any regex operands are parsed from scratch, starting with Phase A of Figure 7-1 (=>223). As a consequence, the regular expression will be compiled when first encountered in the snippet (during the first time through the foreach loop), but not recompiled further due to the /o modifier. Once the eval has finished, that incarnation of the snippet is gone forever. The next time through the outer while loop, the string handed to eval is the same, but because the eval interprets it afresh, the snippet is considered new all over again. Thus, the regular expression is again new, and so it is compiled (with the new value of $regex) the first time it is encountered within the new eval.

Of course, it takes extra effort for eval to compile the snippet with each iteration of the outer loop. Does the /o savings justify the extra time? If the array of @items is short, probably not. If long, probably so. A few benchmarks (addressed later) can often help you decide.

This example takes advantage of the fact that, when we build a program snippet in a string to feed to eval, Perl doesn't consider it to be Perl code until eval is actually executed. You can, however, have eval work with a normally compiled block of code, instead.

Evaluating a string vs. evaluating a block

eval {foreach $item (@items) {
             if ($item =~ m/$regex/o) {
 :
             }
     }};

Included among the variety of reasons to use eval are the recompile-effects of the non-block style and the ability to trap errors. Run-time errors can be trapped with either the string or the block style, while only the string style can trap compile-time errors as well. (We saw an example of this with $* on page 235.) Trapping run-time errors (such as to test if a feature is supported in your version of Perl), and to trap warn, die, exit, and the like, are about the only reasons I can think of to use the eval {...} block version.

A third reason to use eval is to execute code that you build on the fly. The following snippet shows a common trick:

Before explaining the details, let me show an example of how it's used. Given an array of regular expressions, @regexes2check, you might use

Given a list of regular expressions (or, more specifically, a list of strings intended to be taken as regular expressions), Build_MatchMany_Function builds and returns a function that, when called, indicates whether any of the regexes match the contents of $_.

The reason to use something like this is efficiency. If you knew what the regexes were when writing the script, all this would be unnecessary. Not knowing, you might be able to get away with

while (<>) {
    foreach $regex (@regexes2check) {
        if (m/$regex/) {
            ...have a line which matches one of the regexes...
            last;
        }
    }
}

This, too, is inefficient because each regex must be reprocessed and recompiled each time. Extremely inefficient. So, spending time to build an efficient match approach in the beginning can, in the long run, save a lot.

If the strings passed to Build_MatchMany_Function are this, that, and other, the snippet that it builds and evaluates is effectively:

sub {
  return 1 if m/this/;
  return 1 if m/that/;
  return 1 if m/other/;
  return 0
}

Each time this anonymous function is called, it checks the $_ at the time for the three regexes, returning true the moment one is found.

It's a nice idea, but there are problems with how it's commonly implemented (including the one on the previous page). Pass Build_MatchMany_Function a string which contains a $ or @ that can be interpreted, within the eval, by variable interpolation, and you'll get a big surprise. A partial solution is to use a singlequote delimiter:

But there's a bigger problem. What if one of the regexes contains a singlequote (or one of whatever the regex delimiter is)? Wanting the regex don't adds
  return 1 if m'don't';
to the snippet, which results in a syntax error when evaluated. You can use \xff or some other unlikely character as the delimiter, but why take a chance? Here's my solution to take care of these problems:

I'll leave the analysis as an exercise. However, one question: What happens if this function uses local instead of my for the @R array? ¤ Turn the page to check your answer.

local vs. my

¤ Answer to the question on page 273. Before answering the question, first a short summary of binding: Whenever Perl compiles a snippet, whether during program load or eval, references to variables in the snippet are ``linked'' (bound) to the code. The values of the variables are not accessed -- during program load time, the variables don't have values yet. Values are accessed when the code is actually executed. my @R creates a new variable, private, distinct, and unrelated to all other variables in the program. When the snippet to create the anonymous subroutine (the a-sub) is evaluated, its code is linked to our private @R. ``@R'' refer to this private variable. They don't access @R yet, since that happens only when the a-sub is executed. When Build_MatchMany_Function exits, the private @R would normally disappear, but since the a-sub still links to it, @R and its strings are kept around, although they become inaccessible to all but the a-sub (references to ``@R'' elsewhere in the code refer to the unrelated global variable of the same name). When the a-sub is later executed, that private @R is referenced and our strings are used as desired. local @R, on the other hand, simply saves a copy of the global variable @R before we overwrite it with the @_ array. It's likely that there was no previous value, but if some other part of the program happens to use it (the global variable @R, that is) for whatever reason, we've saved a copy so we don't interfere, just in case. When the snippet is evaluated, the ``@R'' links to the same global variable @R. (This linking is unrelated to our use of local -- since there is no private my version of @R, references to ``@R'' are to the global variable, and any use or non-use of local to copy data is irrelevant.) When Build_MatchMany_Function exits, the @R copy is restored. The global variable @R is the same variable as it was during the eval, and is still the same variable that the a-sub links to, but the content of the variable is now different. (We'd copied new values into @R, but never referenced them! A completely wasted effort.) The a-sub expects the global variable @R to hold strings to be used as regular expressions, but we've already lost the values we'd copied into it. When the subroutine is first used, it sees what @R happens to have at the time -- whatever it is, it's not our regexes, so the whole approach breaks down. Mmm. If before we leave Build_MatchMany_Function, we actually use the a-sub (and make sure that the sample text cannot match any of the regexes), the /o would lock in our regexes while @R still holds them, getting around the problem. (If the sample text matches a regex, only the regexes to that point are locked in). This might actually be an appealing solution -- we need @R only until all the regexes are locked in. After that, the strings used to create them are unneeded, so keeping them around in the separate (but undeleteable until the a-sub is deleted) my @R wastes memory.

Unsociable $& and Friends

It makes a copy. All the variables described above actually refer to this internal-use-only copy, rather than to the original string. Having a copy means, obviously, that there are two copies of the string in memory at once. If the target string is huge, so is the duplication. But then, since you need the copy to support these variables, there is really no other choice, right?

Internal optimizations

Warning: The situations and tricks I describe exploit internal workings of Perl. It's nice if they make your programs faster, but they're not part of the Perl specification⁴⁰ and may be changed in future releases. (I am writing as of version 5.003.) If these optimizations suddenly disappear, the only effect will be on efficiency -- programs will still produce the same results, so you don't need to worry that much.

Three situations trigger the copy for a successful match or substitution :
 ·  the use of $`, $&, or $' anywhere in the entire script
 ·  the use of capturing parentheses in the regex
 ·  the use of the /i modifier with a non-/g match operator

Also, you might need additional internal copies to support:
 ·  the use of the /i modifier (with any match or substitute)
 ·  the use of many, but not all, substitution operators

I'll go over the first three here, and the other two in the following section.

$`, $&, or $' require the copy

However, it does notice whether there are no references whatsoever to $`, $&, or $' in the entire program (including all libraries referenced by the script!). Since the variables never appear in the program, Perl can be quite sure that a copy merely to support them can safely be omitted. Thus, if you can be sure that your code and any libraries it might reference never use $`, $&, or $', you are not penalized by the copy except when explicitly required by the two other cases.

Capturing parentheses require the copy

m/.../i requires the copy

An example benchmarked

The run with the extra copying consistently took over 35 percent longer than the one without. This represents an ``average worst case,'' so to speak. The more real work a program does, the less of an effect, percentage-wise, the copying has. The benchmark didn't do any real work, so the effect is highlighted.

On the other hand, in true worst-case scenarios, the extra copy might truly be an overwhelming portion of the work. I ran the same test on the same data, but this time as one huge line incorporating the more than megabyte of data rather than the 50,000 or so reasonably sized lines. Thus, the relative performance of a single match can be checked. The match without the copy returned almost immediately, since it was sure to find a `c' somewhere near the start of the string. Once it did, it was done. The test with the copy is the same except, well, it had to make a copy of the megabyte-plus-sized string first. Relatively speaking, it took over 700 times longer! Knowing the ramifications, therefore, of certain constructs allows you to tweak your code for better efficiently.

Conclusions and recommendations about $& and friends

A solution in Perl can be approached in many ways, and I've said numerous times that if you write in Perl as you write in another language (such as C), your Perl will be lacking and almost certainly inefficient. For the most part, crafting programs The Perl Way should go a long way toward putting you on the right track, but still, as with any discipline, special care can produce better results. So yes, while the copies aren't ``wrong,'' we still want to avoid unnecessary copying whenever possible. Towards that end, there are steps we can take.

Foremost, of course, is to never use $`, $&, or $' anywhere in your code. This also means to never use English.pm nor any library modules that use it, or in any other way references these variables. Table 7-10 shows a list of standard libraries (in Perl version 5.003) which reference one of the naughty variables, or uses another library that does. You'll notice that most are tainted only because they use Carp.pm. If you look into that file, you'll find only one naughty variable:

$eval =~ s/[\\\']/\\$&/g;

Changing this to

$eval =~ s/([\\\'])/\\$1/g;

If you can be sure these variables never appear, you'll know you will do the copy only when you explicitly request it with capturing parentheses, or via the rogue m/.../i. Some expressions in current code might need to be rewritten. On a case-by-case basis, $` can often be mimicked by (.*?) at the head of the regex, $& by (...) around the regex, and $' by (?=(.*)) at the end of the regex.

If your needs allow, there are other, non-regex methods that might be attempted in place of some regexes. You can use index(...) to find a fixed string, for example. In the benchmarks I described earlier, it was almost 20 percent faster than m/.../, even without the copy overhead.

How to check whether your code is tainted by $&

Especially with the use of libraries, it's not always easy to notice whether your program ever references $&, $`, or $'. I went to the trouble of modifying my version of Perl to issue a warning the first time one was encountered. (If you'd like to do the same, search for the three appearances of sawampersand in Perl's gv.c, and add an appropriate call to warn.) An easier approach is to test for the performance penalty, although it doesn't tell you where the offending variable is. Here's a subroutine that I've come up with: sub CheckNaughtiness { local($_) = 'x' x 10000; # some non-small amount of data # calculate the overhead of a do-nothing loop local($start) = (times)[0]; for ($i = 0; $i < 5000; $i++) { } local($overhead) = (times)[0] - $start; # now calculate the time for the same number $start = (times)[0]; for ($i = 0; $i < 5000; $i++) { m/^/; } local($delta) = (times)[0] - $start; # a differential of 10 is just a heuristic printf "It seems your code is %s (overhead=%.2f, delta=%.2f)\n", ($delta > $overhead*10) ? "naughty":"clean", $overhead, $delta; } This is not a function you would keep in production code, but one you might insert temporarily and call once at the beginning of the program (perhaps immediately following it with an exit, then removing it altogether once you have your answer). Once you know you program is $&-clean, there is still a chance that a rogue eval could introduce it during the run of the program, so it's also a good idea to test at the end of execution, just to be sure.

The Efficiency Penalty of the /i Modifier

Before a match or substitution operator applies an /i-governed regex, Perl first makes a temporary copy of the entire target string. This copy is in addition to any copy in support of $& and friends. The latter is done only after a successful match, while the one to support a case-insensitive match is done before the attempt. After the copy is made, the engine then makes a second pass over the entire string, converting any uppercase characters to lowercase. The result might happen to be the same as the original, but in any case, all letters are lowercase.

This goes hand in hand with a bit of extra work done during the compilation of the regex to an internal form. At that time, uppercase letters in the regex are converted to lowercase as well.

The result of these two steps is a string and a regex that then matches normally -- nothing special or extra needs to be done within the actual matching portion of the regex engine. It all appears to be a very tidy arrangement, but this has got to be one of the most gratuitous inefficiencies in all of Perl.

Methods to implement case-insensitive matching

Many subexpressions (and full regular expressions, for that matter) do not require special handling: the CSV program at the start of the chapter (=>205), the regex to add commas to numbers (=>229), and even the huge, 4,724-byte regex we construct in ``Matching an Email Address'' (=>294) are all free of the need for special case-insensitive handling. A case-insensitive match with such expressions should not have any efficiency penalty at all.

Even a character class with letters shouldn't entail an efficiency penalty. At compile time, the appropriate other-case version of any letter can easily be included. (A character class's efficiency is not related to the number of characters in the class; =>115.) So, the only real extra work would be when letters are included in literal text, and with backreferences. Although they must be dealt with, they can certainly be handled more efficiently than making a copy of the entire target string.

By the way, I forgot to mention that when the /g modifier is used, the copy is done with each match. At least the copy is only from the start of the match to the end of the string -- with a m/.../ig on a long string, the copies are successively shorter as the matches near the end.

A few /i benchmarks

As an unfair and cruel first test, I loaded the entire file into a single string, and benchmarked 1 while m/./g and 1 while m/./gi. Dot certainly doesn't care one way or the other about capitalization, so it's not reasonable to penalize this match for case-insensitive handling. On my machine, the first snippet benchmarked at a shade under 12 seconds. Simply adding the /i modifier (which, you'll note, is meaningless in this case) slowed the program by four orders of magnitude, to over a day and a half!⁴² I calculate that the needless copying caused Perl to shuffle around more than 647,585 megabytes inside my CPU. This is particularly unfortunate, since it's so trivial for the compilation part of the engine to tell the matching part that case-insensitiveness is irrelevant for ., the regex at hand.

This unrealistic benchmark is definitely a worst-case scenario. Searching a huge string for something that matches less often than . is more realistic, so I benchmarked m/\bwhile\b/gi and m/\b[wW][hH][iI][lL][eE]\b/g on the same string. Here, I try to mimic the regex-oriented approach myself. It's incredibly naïve for a regex-oriented implementation to actually turn literal text into character classes,⁴³ so we can consider the /i-equivalent to be a worst-case situation in this respect. In fact, manually turning while into [wW][hH][iI][lL][eE] also kills Perl's fixed string check (=>155), and renders study (=>287) useless for the regex. With all this against it, we should expect it to be very slow indeed. But it's still over 50 times faster than the /i version!

Perhaps this test is still unfair -- the /i-induced copy made at the start of the match, and after each of the 412 matches of \bwhile\b in my test data, is large. (Remember, the single string is over a megabyte long.) Let's try testing m/^int/i and m/^[iI][nN][tT]/ on each of the 50,000 lines of the test file. In this case, /i has each line copied before the match attempt, but since they're so short, the copy is not so crushing a penalty as before: the /i version is now just 77 percent slower. Actually, this includes the extra copies inexplicably made for each of the 148 matches -- remember, a non-/g m/.../i induces the $&-support copy.

Final words about the /i penalty

Foremost: don't use /i unless you really have to. Blindly adding it to a regex that doesn't require it invites many wasted CPU cycles. In particular, when working with long strings, it can be a huge benefit to rewrite a regex to mimic the regex-oriented approach to case insensitivity, as I did with the last two benchmarks.

Substitution Efficiency Concerns

Still, I have managed to understand a bit about how it works,⁴⁴ and have developed a few rules of thumb that I'd like to share with you. Let me warn you, up front, that there's no simple one-sentence summary to all this. Perl often takes internal optimizations in bits and pieces where they can be found, and numerous special cases and opportunities surrounding the substitution operator provide fertile ground for optimizations. It turns out that the $&-support copy disables all of the substitution-related optimizations, so that's all the more reason to banish $& and friends from your code.

Let me start by stepping back to look at the substitution operator efficiency's worst-case scenario.

The normal ``slow-mode'' of the substitution operator

s[(\d+(\.\d*)?)F\b]{sprintf "%.0fC", ($1-32) * 5/9}eg

Water boils at 212F, freezes at 32F.

At the next match (because /g is used), the text between the two matches is added to the temporary string, followed by the newly computed substitution text, `0C'. Finally, after it can't find any more matches, the remainder of the string (just the final period in this case) is copied to close out the temporary string. This leaves us with:
  Water boils at 100C, freezes at 0C.
The original target string, $_, is then discarded and replaced by the temporary string. (I'd think the original target could be used to support $1, $&, and friends, but it does not -- a separate copy is made for that, if required.)

At first, this method of building up the result might seem reasonable because for the general case, it is reasonable. But imagine something simple like s/\s+$// to remove trailing whitespace. Do you really need to copy the whole (potentially huge) string, just to lop off its end? In theory, you don't. In practice, Perl doesn't either. Well, at least not always.

$& and friends disable all substitute-operator optimizations

The $&-support copy is also done when there are capturing parentheses in the regex, although in that case, you'll likely be enjoying the fruits of that copy (since $1 and the like are supported by it). Capturing parentheses also disable the substitution optimizations, but at least only for the regexes they're used in and not for all regexes as a single stray $& does.

Replacement text larger than the match not optimized

Figure 7-2: Applying s/<FIRST>/Tom/ to `Dear·<FIRST>,[NL]'

In the case of s/\s+$//, there's no replacement text to be filled in and no match-following text to be moved down -- once the match is found, the size of the string is adjusted to lop off the match, and that's that. Very zippy. The same kinds of optimizations also apply to matches at the beginning of the string.

When the replacement text is exactly the same size as the matched text, you'd think that as a further optimization, the ``move to fill the gap'' could be omitted, since there is no gap when the sizes are an exact match. For some reason, the needless ``move'' is still done. At least the algorithm, as it stands, never copies more than half the string (since it is smart enough to decide whether it should copy the part before the match, or the part after).

A substitution with /g is a bit better still. It doesn't do the gap-filling move until it knows just how much to move (by delaying the move until it knows where the next match is). Also, it seems in this case that the worthless move to fill a non-existent gap is kindly omitted.

Only fixed-string replacement text substitutions are optimized

$given = 'Tom';
$letter =~ s/<FIRST>/$given/g;

The replacement operand has variables interpolated before any matching begins, so the size of the result is known.

Any substitution using the /e modifier, of course, doesn't know the size of the substitution text until after a match, and the substitution operand is evaluated, so there are no substitution optimizations with /e either.

Final comments on all these internal optimizations

Benchmarking

$start = (times)[0];
  :
$delta = (times)[0] - $start;
printf "took %.1f seconds\n", $delta;

An important consideration about benchmarking is that due to clock granularity (¹/60 or ¹/100 of a second on many systems), it's best to have code that runs for at least a few seconds. If the code executes too quickly, do it over and over again in a loop. Also, try to remove unrelated processing from the timed portion. For example, rather than

The biggest change is that the file I/O has been moved out from the timed portion. Of course, if you don't have enough free memory and start swapping to disk, the advantage is gone, so make sure that doesn't happen. You can simulate more data by using a smaller amount of real data and processing it several times:

for ($i = 0; $i < 10; $i++) {
   foreach (@lines) {
        $count++ while m/\b(?:char\b|return\b|void\b)/g;
   }
}

It might take some time to get used to benchmarking in a reasonable way, but the results can be quite enlightening and downright rewarding.

Regex Debugging Information

Much of what -Dr provides is beyond the scope of this book, but you can readily understand some of its information. Let's look at a simple example (I'm using Perl version 5.003):

 [1] jfriedl@tubby> perl -cwDr -e '/^Subject: (.*)/'
 [2] rarest char j at 3
 [3] first 14 next 83 offset 4
 [4]  1:BRANCH(47)
 [5]  5:BOL(9)
 [6]  9:EXACTLY(23) <Subject: >
 [7] 23:OPEN1(29)
    :
 [8] 47:END(0)
 [9] start `Subject: ' anchored minlen 9 

At [1], I invoke perl at my shell prompt, using the command-line arguments -c (which means check script, don't actually execute it), -w (issue warnings about things Perl thinks are dubious -- always used as a matter of principle), -Dr (regex debugging), and -e (the next argument is the Perl snippet itself). This combination is convenient for checking regexes right from the command line. The regex here is ^Subject:·(.*) which we've seen several times in this book.

Lines [4] through [8] represents Perl's compiled form of the regex. For the most part, we won't be concerned much about it here. However, in even a casual look, line [6] sticks out as understandable.

Literal text cognizance

Examining int|void|while, you see that `i' is required in any match. Some NFA engines can deduce exactly that (any DFA knows it implicitly), but Perl's engine is unfortunately not one of them. In the debugging output, int, void, and while appear on lines similar to [6] above, but those are local (subexpression) requirements. For literal text cognizance, Perl needs global regex-wide confidence, and as a general rule, it can't deduce fixed text from anything that's part of alternation.

Many expressions, such as <CODE>(.*?)</CODE>, have more than one clump of literal text. In these cases, Perl (somewhat magically) selects one or two of the clumps and makes them available to the rest of the optimization subroutines. The selected clump(s) are shown on a line similar to [9].

Main optimizations reported by -Dr

start `clump'

Indicates that a match must begin with clump, one of those found by literal text cognizance. This allows Perl to do optimizations such as the fixed string check and first character discrimination discussed in Chapter 5.

must have "clump" back num

Indicates a required clump of text like the start item above, but the clump is not at the start of the regex. If num is not -1, Perl knows any match must begin that many characters earlier. For example, with [Tt]ubby..., the report is `must have "ubby" back 1', meaning that if a fixed string check reveals ubby starting at such and such a location, the whole regex should be applied starting at the previous position.

Conversely, with something like .*tubby, it's not helpful to know exactly where tubby might be in a string, since a match including it could start at any previous position, so num is -1.

stclass `:kind'

Indicates that Perl realizes the match must start with some particular kind of character. With \s+, kind is SPACE, while with \d+ it is DIGIT. With a character class like the [Tt]ubby example, kind is reported as ANYOF.

plus

Indicates that the stclass or a single-character start item is governed by +, so not only can the first character discrimination find the start of potential matches, but it can also quickly zip past a leading \s+ and the like before letting the full (but slower) regex engine attempt the complete match.

anchored

Indicates that the regex begins with a caret anchor. This allows the ``String/ Line Anchor'' optimization (=>158).

implicit

Indicates that Perl has added an implicit caret to the start of the regex because the regex begins with .* (=>158).

Other optimizations arising from literal text cognizance

For the trivia-minded, Perl's idea of the rarest character is \000, followed by \001, \013, \177, and \200. Some of the rarest printable characters are ~, Q, Z, ?, and @. The least-rare characters are e, space, and t. (The manpage says that this was derived by examining a combination of C programs and English text.)

The Study Function

When you study a string, Perl takes some time and memory to build a list of places in the string each character is found. On most systems, the memory required is four times the size of the string (but is reused with subsequent calls of study). study's benefit can be realized with each subsequent regex match against the string, but only until the string is modified. Any modification of the string renders the study list invalid, as does studying a different string.

The regex engine itself never looks at the study list; only the transmission references it. The transmission looks at the start and must have debugging information mentioned on page 286 to pick what it considers a rare character (discussed It picks a rare (yet required) character because it's not likely to be found in the string, and a quick check of the study list that turns up nothing means a match can be discounted immediately without having to rescan the entire string.

If the rare character is found in the string, and if that character must occur at a known position in any possible match (such as with ..this, but not .?this), the transmission can use the study list to start matching from near the location. This saves time by bypassing perhaps large portions of the string.⁴⁷

When not to use study

• Don't use study when the target string is short. In such cases, the normal fixed-string cognizance optimization should suffice.

• Don't use study when you plan only a few matches against the target string (or, at least, few before it is modified, or before you study a different string). An overall speedup is more likely if the time spent to study a string is amortized over many matches. With just a few matches, the time spent scanning the string (to build the study list) can overshadow any savings.

  Note that with the current implementation, m/.../g is considered one match: the study list is consulted only with the first attempt. Actually, in the case of a scalar context m/.../g, it is consulted with each match, but it reports the same location each time -- for all but the first match, that location is before the beginning of the match, so the check is just a waste of time.

• Don't use study when Perl has no literal text cognizance (=>286) for the regular expressions that you intend to benefit from the study. Without a known character that must appear in any match, study is useless.

When study can help

Study in the real world

You can get around this bug with an explicit undef, or other modification, of the study'd string (when you're done with it, of course). The automatic assignment to $_ in while (<>) is not sufficient.

When study can work, it often doesn't live up to its full potential, either due to simple bugs or an implementation that hasn't matured as fast as the rest of Perl. At this juncture, I recommend against the use of study unless you have a very specific situation you know benefits. If you do use it, and the target string is in $_, be sure to undefine it when you are done.

Putting It All Together

Let's look again at the initial CSV problem. Here's my Perl5 solution, which, as you'll note, is fairly different from the original on page 205:

Like the first version, it uses a scalar-context m/.../g with a while loop to iterate over the string. We want to stay in synch, so we make sure that at least one of the alternatives matches at any location a match could be started. We allow three types of fields, which is reflected in the three alternatives of the main match.

Because Perl5 allows you to choose exactly which parentheses are capturing and which aren't, we can ensure that after any match, $+ holds the desired text of the field. For empty fields where the third alternative matches and no capturing parentheses are used, $+ is guaranteed to be undefined, which is exactly what we want. (Remember undef is different from an empty string -- returning these different values for empty and "" fields retains the most information.)

The final push covers cases in which the string ends with a comma, signifying a trailing empty field. You'll note that I don't use m/,$/ as I did earlier. I did so earlier because I was using it as an example to show regular expressions, but there's really no need to use a regex when a simpler, faster method exists.

Along with the CSV question, many other common tasks come up time and again in the Perl newsgroups, so I'd like to finish out this chapter by looking at a few of them.

Stripping Leading and Trailing Whitespace

s/^\s+//;
s/\s+$//;

For some reason, it seems to be The Thing to try to find a way to do it all in one shot, so I'll offer a few methods. I don't recommend them, but it's educational to understand why they work, and why they're not desirable.

s/\s*(.*?)\s*$/$1/

Commonly given as a great example of the non-greediness that is new in Perl5, but it's not a great example because it's so much slower (by about three times in my tests) than most of the other solutions. The reason is that with each character, before allowing dot to match, the *? must try to see whether what follows can match. That's a lot of backtracking, particularly since it's the kind that goes in and out of the parentheses (=>151).

s/^\s*(.*\S)?\s*$/$1/

Much more straightforward. The leading ^\s* takes care of leading whitespace before the parentheses start capturing. Then the .* matches to the end of the line, with the \S causing backtracking past trailing whitespace to the final non-whitespace. If there's nothing but whitespace in the first place, the (.*\S)? fails (which is fine), and the final \s* zips to the end.

$_ = $1 if m/^\s*(.*\S)?/

More or less the same as the previous method, this time using a match and assignment instead of a substitution. With my tests, it's about 10 percent faster.

s/^\s*|\s*$//g

A commonly thought-up solution that, while not incorrect, has top-level alternation that removes many of the optimizations that might otherwise be possible. The /g modifier allows each alternative to match, but it seems a waste to use /g when we know we intend at most two matches, and each with a different subexpression. Fairly slow.

Adding Commas to a Number

1 while s/^(-?\d+)(\d{3})/$1,$2/;

You can enhance this solution by using a common optimization from Chapter 5 (=>156), replacing \d{3} with \d\d\d. Why bother making the regex engine count the occurrences when you can just as easily say exactly what you want? This one change saved a whopping three percent in my tests. (A penny saved...)

Another enhancement is to remove the start-of-string anchor. This allows you to comma-ify a number (or numbers) within a larger string. As a byproduct, you can then safely remove the -?, since it exists only to tie the first digit to the anchor. This change could be dangerous if you don't know the target data, since 3.14159265 becomes 3.14,159,265. In any case, if you know the number is the string by itself, the anchored version is better.

A completely different, but almost-the-same approach I've come up with uses a single /g-governed substitution:

Removing C Comments

Chapter 5 dealt with generic NFA engines, so our comment-matching regex works fine in Perl. For extra efficiency, I'd use non-capturing parentheses, but that's about the only direct change I'd make. It's not unreasonable to use the FAQ's simpler /\*.*?\*/ -- Chapter 5's solution leads the engine to a match more efficiently, but /\*.*?\*/ is fine for applications that aren't time critical. It's certainly easier to understand at first glance, so I'll use it to simplify the first draft of our comment-stripping regex.

Here it is:

After applying the changes discussed during the Tcl treatment, and combining the two comment regexes into one top-level alternative (which is easy since we're writing the regex directly and not building up from separate $COMMENT and $COMMENT1 components), our Perl version becomes:

To make a full program out of this, just insert it into:

Yup, that's the whole program.

Matching an Email Address

Still, it's not for the faint at heart. In fact, the regex we'll come up with is 4,724 bytes long! At first thought, you might think something as simple as \w+\@[.\w]+ could work, but it is much more complex. Something like

Jeffy <"That Tall Guy"@ora.com (this address no longer active)>

Levels of interpretation

$username = "\w+";
$hostname = "\w+(\.\w+)+";
$email    = "^$username\@$hostname$";
   :
... m/$email/o ...

$username = '\w+';
$hostname = '\w+(\.\w+)+';
$email    = "^$username\@$hostname\$";

Let's start building the real regex by looking at item 16 in Table 7-11. The simple $space = "·" isn't good because if we use the /x modifier when we apply the regex (something we plan to do), spaces outside of character classes, such as this one, will disappear. We can also represent a space in the regex with \040 (octal 40 is the ASCII code for the space character), so we might be tempted to assign "\040" to $space. This would be a silent mistake because, when the doublequoted string is evaluated, \040 is turned into a space. This is what the regex will see, so we're right back where we started. We want the regex to see \040 and turn it into a space itself, so again, we must use "\\040" or '\040'.

Getting a match for a literal backslash into the regex is particularly hairy because it's also the regex escape metacharacter. The regex requires \\ to match a single literal backslash. To assign it to, say, $esc, we'd like to use '\\', but because \\ is special even within singlequoted strings,⁵¹ we need $esc = '\\\\' just to have the final regex match a single backslash. This backslashitis is why I make $esc once and then use it wherever I need a literal backslash in the regex. We'll use it a few times as we construct our address regex. Here are the preparatory variables I'll use this way: