GNU bug report logs - #64743
Speed up GNU make's internal processing

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 64743 in the body.
You can then email your comments to 64743 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: bug-automake <at> gnu.org
Cc: Paul Smith <psmith <at> gnu.org>
Subject: Speed up GNU make's internal processing
Date: Thu, 20 Jul 2023 16:07:53 +0200

[Message part 1 (text/plain, inline)]

Hi,

Paul Smith, the GNU make maintainer, gives this recommendation how
to disable built-in rules and thus speed up make's processing [1]:

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
My recommendation has been to disable the built-in rules directly, if
you don't need them.  For example you can use:

  .SUFFIXES:

to disable most of the built in rules (this is a POSIX standard
facility so it's helpful even for other versions of make).

Unfortunately that doesn't fix all problems because GNU Make also has a
few built-in rules that are defined using pattern rules (because suffix
rules are not powerful enough).  So a full list of "turn it all off"
would be this:

  .SUFFIXES:
  %:: %,v
  %:: RCS/%,v
  %:: RCS/%
  %:: s.%
  %:: SCCS/s.%
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

GNU Automake already emits the '.SUFFIXES:' line. To optimize things
for GNU make, it should also emit the remaining part.

It has no effect with non-GNU make [2].

The effect, for example in gettext's gettext-runtime/src directory, is
that
  - The output of "make -n -d" shrinks from 11028 lines to 4928 lines.
  - The number of 'stat()' calls made by "make -n" shrinks from 188 to 178.
    (make no longer tests whether various directories have an 'RCS' or 'SCCS'
    subdirectory.)

Patch is attached.

Bruno

[1] https://lists.gnu.org/archive/html/bug-make/2023-07/msg00063.html
[2] https://lists.gnu.org/archive/html/bug-make/2023-07/msg00067.html

[0001-Speed-up-GNU-make-s-internal-processing.patch (text/x-patch, attachment)]

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: bug-automake <at> gnu.org
Cc: Paul Smith <psmith <at> gnu.org>
Subject: Re: Speed up GNU make's internal processing
Date: Thu, 20 Jul 2023 23:57:07 +0200

[Message part 1 (text/plain, inline)]

> Patch is attached.

Here's a corrected patch. (Fixed another test failure.)

[0001-Speed-up-GNU-make-s-internal-processing.patch (text/x-patch, attachment)]

Message #11 received at 64743 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: bruno <at> clisp.org
Cc: 64743 <at> debbugs.gnu.org, psmith <at> gnu.org
Subject: Re: bug#64743: Speed up GNU make's internal processing
Date: Thu, 20 Jul 2023 19:03:52 -0600

Hi Bruno,

    GNU Automake already emits the '.SUFFIXES:' line. To optimize things
    for GNU make, it should also emit the remaining part.

Ok. I just hope those weird-looking %:: rules do not cause trouble with
other makes. I guess we'll find out.

The fnoc test also failed due to the new rules. I fixed that, added a
few words of documentation and NEWS, and pushed it.

Thanks Bruno. -k


2023-07-20  Bruno Haible  <bruno <at> clisp.org>

automake: disable GNU make's internal pattern rules, for speed.

From https://bugs.gnu.org/64743.

* lib/am/footer.am: Disable GNU make's internal pattern rules.
* lib/Automake/Rule.pm (_conditionals_for_rule): Add special handling
for these pattern rules from footer.am.
(define): Likewise.
* t/nodep.sh: Update test to avoid matching the new %:: rules.
* t/fnoc.sh: Update test to avoid matching the new %:: SCCS rule.

* NEWS: Mention this.
* doc/automake.texi (Suffixes): Likewise.
(Doc changes written by Karl.)
diff --git a/NEWS b/NEWS
index c64d4ef15..ced0fd640 100644
--- a/NEWS
+++ b/NEWS
@@ -21,6 +21,9 @@ New in 1.17:
   - Variables using escaped \# will trigger portability warnings, but be
     retained when appended.  GNU Make & BSD Makes are known to support it.

+  - GNU Make's default pattern rules are disabled, for speed and debugging.
+    (.SUFFIXES was already cleared.) (bug#64743)
+
   - For Texinfo documents, if a .texi.in file exists, but no .texi, the
     .texi.in will be read. Texinfo source files need not be present at
     all, and if present, need not contain @setfilename. Then the file name
diff --git a/doc/automake.texi b/doc/automake.texi
index 4561d1948..3b4f06e73 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -10977,6 +10977,14 @@ Automake generate the suffix list for @code{.SUFFIXES}.  Any given
 @code{SUFFIXES} go at the start of the generated suffixes list, followed
 by Automake generated suffixes not already in the list.

+@c https://bugs.gnu.org/64743.
+@cindex built-in suffix and pattern rules, disabled
+@cindex suffix rules of Make, disabled
+@cindex pattern rules of GNU Make, disabled
+Automake disables the Make program's built-in rules with a
+@code{.SUFFIXES:} rule, and then adds whatever suffixes are
+necessary.  Automake also disables GNU Make's built-in pattern rules.
+
 @node Include
 @chapter Include

diff --git a/lib/Automake/Rule.pm b/lib/Automake/Rule.pm
index 9f72d2728..52ee3be4a 100644
--- a/lib/Automake/Rule.pm
+++ b/lib/Automake/Rule.pm
@@ -695,6 +695,9 @@ sub _conditionals_for_rule ($$$$)

   return $cond if !$message; # No ambiguity.

+  # Don't coalesce the several pattern rules from footer.am into a single one.
+  return $cond if $target eq "%:" && $where->get =~ /\/am\/footer\.am$/;
+
   if ($owner == RULE_USER)
     {
       # For user rules, just diagnose the ambiguity.
@@ -764,23 +767,27 @@ sub define ($$$$$)

   my $tdef = _rule_defn_with_exeext_awareness ($target, $cond, $where);

-  # A GNU make-style pattern rule has a single "%" in the target name.
-  msg ('portability', $where,
-       "'%'-style pattern rules are a GNU make extension")
-    if $target =~ /^[^%]*%[^%]*$/;
-
-  # See whether this is a duplicated target declaration.
-  if ($tdef)
+  # The pattern rules in footer.am look like duplicates, but really aren't.
+  if ($source !~ /\/am\/footer\.am$/)
     {
-      # Diagnose invalid target redefinitions, if any.  Note that some
-      # target redefinitions are valid (e.g., for multiple-targets
-      # pattern rules).
-      _maybe_warn_about_duplicated_target ($target, $tdef, $source,
-                                           $owner, $cond, $where);
-      # Return so we don't redefine the rule in our tables, don't check
-      # for ambiguous condition, etc.  The rule will be output anyway
-      # because '&read_am_file' ignores the return code.
-      return ();
+      # A GNU make-style pattern rule has a single "%" in the target name.
+      msg ('portability', $where,
+           "'%'-style pattern rules are a GNU make extension")
+        if $target =~ /^[^%]*%[^%]*$/;
+
+      # See whether this is a duplicated target declaration.
+      if ($tdef)
+        {
+          # Diagnose invalid target redefinitions, if any.  Note that some
+          # target redefinitions are valid (e.g., for multiple-targets
+          # pattern rules).
+          _maybe_warn_about_duplicated_target ($target, $tdef, $source,
+                                               $owner, $cond, $where);
+          # Return so we don't redefine the rule in our tables, don't check
+          # for ambiguous condition, etc.  The rule will be output anyway
+          # because '&read_am_file' ignores the return code.
+          return ();
+        }
     }

   my $rule = _crule $target;
diff --git a/lib/am/footer.am b/lib/am/footer.am
index 9715c826c..388defb14 100644
--- a/lib/am/footer.am
+++ b/lib/am/footer.am
@@ -17,3 +17,14 @@
 # Tell versions [3.59,3.63) of GNU make to not export all variables.
 # Otherwise a system limit (for SysV at least) may be exceeded.
 .NOEXPORT:
+
+# Tell GNU make to disable its built-in pattern rules.
+## This reduces make's internal processing.
+## Recommended by Paul Smith in
+## <https://lists.gnu.org/archive/html/bug-make/2023-07/msg00063.html>.
+## These rules have no effect on non-GNU make implementations.
+%:: %,v
+%:: RCS/%,v
+%:: RCS/%
+%:: s.%
+%:: SCCS/s.%
diff --git a/t/fnoc.sh b/t/fnoc.sh
index 97b694c63..7d6b49096 100644
--- a/t/fnoc.sh
+++ b/t/fnoc.sh
@@ -35,5 +35,10 @@ END
 $ACLOCAL
 $AUTOMAKE

-grep '[^F]CC' Makefile.in | grep -v MKDEP && exit 1
+# In bug#64743, explicit rules were added to disable make's default
+# built-in rules, including one for SCCS:
+# %:: SCCS/s.%
+# So don't match that. Meanwhile, MKDEP does not occur in the output
+# any more, but leaving it in in case it comes back.
+grep '[^F]CC' Makefile.in | grep -v MKDEP | grep -v SCCS && exit 1
 exit 0
diff --git a/t/nodep.sh b/t/nodep.sh
index 4201c8ead..0174b838e 100644
--- a/t/nodep.sh
+++ b/t/nodep.sh
@@ -37,6 +37,6 @@ $ACLOCAL
 $AUTOMAKE

 sed 's/printf .*%s//' Makefile.in > Makefile.tmp
-grep '%' Makefile.tmp && exit 1
+grep -v '^%::' Makefile.tmp | grep '%' && exit 1

 :

compile finished at Thu Jul 20 18:00:35 2023

Message #19 received at 64743 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: Karl Berry <karl <at> freefriends.org>
Cc: 64743 <at> debbugs.gnu.org, psmith <at> gnu.org
Subject: Re: bug#64743: Speed up GNU make's internal processing
Date: Fri, 21 Jul 2023 03:18:11 +0200

Karl Berry wrote:
> I just hope those weird-looking %:: rules do not cause trouble with
> other makes. I guess we'll find out.

I tested the default 'make' of various OSes, before submitting the patch.
Whether some other, rarely-used 'make' implementation has problems with it,
we'll find out.

> The fnoc test also failed due to the new rules. I fixed that

I fixed it too, in the second version of the patch. Had a hard time
distinguishing spurious and persistent test failures (see bug #64756).

> added a few words of documentation and NEWS, and pushed it.

Thanks!

Bruno

Message #22 received at 64743 <at> debbugs.gnu.org (full text, mbox):

From: Nick Bowler <nbowler <at> draconx.ca>
To: Bruno Haible <bruno <at> clisp.org>
Cc: psmith <at> gnu.org, 64743 <at> debbugs.gnu.org, Karl Berry <karl <at> freefriends.org>
Subject: Re: bug#64743: Speed up GNU make's internal processing
Date: Thu, 20 Jul 2023 22:36:20 -0400

On 20/07/2023, Bruno Haible <bruno <at> clisp.org> wrote:
> Karl Berry wrote:
>> I just hope those weird-looking %:: rules do not cause trouble with
>> other makes. I guess we'll find out.
>
> I tested the default 'make' of various OSes, before submitting the patch.
> Whether some other, rarely-used 'make' implementation has problems with it,
> we'll find out.

FWIW I tried the makes on FreeBSD 9, HP-UX 11 and ULTRIX 4.5 and there was
no obvious negative effect with the snippet shown in the original post.

I expect all the non-GNU implementations are interpreting these lines
as ordinary target rules for a funnily-named files with funnily-named
prerequisites.  So it's not quite right to say it has "no effect", but
I think it shouldn't cause any particular problem for packages using
portable file names.

Cheers,
  Nick

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.