GNU bug report logs - #10852
VPATH builds cannot recover from missing parser header

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 10852 in the body.
You can then email your comments to 10852 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Akim Demaille <akim <at> lrde.epita.fr>
To: Automake Bugs <bug-automake <at> gnu.org>
Subject: VPATH builds cannot recover from missing parser header
Date: Mon, 20 Feb 2012 14:24:15 +0100

[Message part 1 (text/plain, inline)]

I am having problems in Bison (current master) to recover
from a lost parse-gram.h, generated from parse-gram.y
with regular Automake (1.11.3) handling:

> AM_YFLAGS = -d -v --warnings=all,error --report=all
> 
> src_bison_SOURCES =                             \
>   ...
>   src/output.h                                  \
>   src/parse-gram.y                              \
>   src/print-xml.c                               \
>   ...


Makefile.in:

> src/parse-gram.h: src/parse-gram.c
> 	@if test ! -f $@; then rm -f src/parse-gram.c; else :; fi
> 	@if test ! -f $@; then $(MAKE) $(AM_MAKEFLAGS) src/parse-gram.c; else :; fi

The problem is that src/parse-gram.c is in $(srcdir), so
its occurrences in the commands should be prefixed by
$(srcdir)/.  Actually, I don't understand why we don't
simply use $< (lemme guess: it's not portable for non
generic recipes, ISTR some horrors in this area that my
brains decided to quickly forget :).  Also, why two "if"?
In case some concurrent execution of Make would already have
provided $@ in the meanwhile?

The following patch extends a test which is aimed at checking
this, but does it in a non-vpath build :)

I have a question though.  I don't understand how come
Make realizes it must provide $@ in srcdir too.

The Makefile.in features:

> .y.c:
> 	$(AM_V_YACC)$(am__skipyacc) $(SHELL) $(YLWRAP) $< y.tab.c $@ y.tab.h $*.h y.output $*.output -- $(YACCCOMPILE)

and at runtime I have:

> /bin/sh ../../build-aux/ylwrap ../../src/parse-gram.y y.tab.c ../../src/parse-gram.c y.tab.h ../../src/parse-gram.h y.output ../../src/parse-gram.output -- ./tests/bison -y -d -v --warnings=all,error --report=all 
> updating ../../src/parse-gram.h
> ../../src/parse-gram.output is unchanged

which is what is expected, but I don't understand why
it works: that $< is prefixed with $(srcdir), this I
understand, but why does it appear on $@?  I
have to specify $(srcdir)/ by hand on similar patterns,
why does it work here?

Thanks!

[0001-tests-check-yacc-parser-header-recovery-in-vpath-bui.patch (application/octet-stream, attachment)]

Message #8 received at 10852 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Akim Demaille <akim <at> lrde.epita.fr>
Cc: 10852 <at> debbugs.gnu.org
Subject: Re: bug#10852: VPATH builds cannot recover from missing parser header
Date: Mon, 20 Feb 2012 14:58:13 +0100

Hi Akim.

On 02/20/2012 02:24 PM, Akim Demaille wrote:
> I am having problems in Bison (current master) to recover
> from a lost parse-gram.h, generated from parse-gram.y
> with regular Automake (1.11.3) handling:
> 
>> AM_YFLAGS = -d -v --warnings=all,error --report=all
>>
>> src_bison_SOURCES =                             \
>>   ...
>>   src/output.h                                  \
>>   src/parse-gram.y                              \
>>   src/print-xml.c                               \
>>   ...
> 
> 
> Makefile.in:
> 
>> src/parse-gram.h: src/parse-gram.c
>> 	@if test ! -f $@; then rm -f src/parse-gram.c; else :; fi
>> 	@if test ! -f $@; then $(MAKE) $(AM_MAKEFLAGS) src/parse-gram.c; else :; fi
> 
> The problem is that src/parse-gram.c is in $(srcdir), so
> its occurrences in the commands should be prefixed by
> $(srcdir)/.  Actually, I don't understand why we don't
> simply use $< (lemme guess: it's not portable for non
> generic recipes,
>
Exactly; see the first node here:

  http://www.gnu.org/software/autoconf/manual/html_node/Portable-Make.html

> ISTR some horrors in this area that my brains decided to quickly forget :).
>
;-)

> Also, why two "if"?
>
For the sake of "make -n": at least GNU make and Solaris make execute
recipes containing the $(MAKE) string even when they are running in dry
mode; so if we didn't break the recipe above in two invocations, the
file 'src/parse-gram.c' would be removed even upon "make -n".  Not nice.

> In case some concurrent execution of Make would already have
> provided $@ in the meanwhile?
>
Nope; see above.

> The following patch extends a test which is aimed at checking
> this, but does it in a non-vpath build :)
>
But the test is wrong, because it checks that the Yacc-generated .h and .c
files are placed in the $srcdir, while we expect them to be placed in the
$builddir.  Do the tests added by your patch work for you?  They don't work
for me (as I would have expected BTW).

> I have a question though.  I don't understand how come
> Make realizes it must provide $@ in srcdir too.
> 
> The Makefile.in features:
> 
>> .y.c:
>> 	$(AM_V_YACC)$(am__skipyacc) $(SHELL) $(YLWRAP) $< y.tab.c $@ y.tab.h $*.h y.output $*.output -- $(YACCCOMPILE)
> 
> and at runtime I have:
> 
>> /bin/sh ../../build-aux/ylwrap ../../src/parse-gram.y y.tab.c ../../src/parse-gram.c y.tab.h ../../src/parse-gram.h y.output ../../src/parse-gram.output -- ./tests/bison -y -d -v --warnings=all,error --report=all 
>> updating ../../src/parse-gram.h
>> ../../src/parse-gram.output is unchanged
> 
> which is what is expected, but I don't understand why
> it works: that $< is prefixed with $(srcdir), this I
> understand, but why does it appear on $@?
>
Unless I'm somehow sorely mistaken, that is not automake's doing; it's the
make implementation that does such a rewrite.  Which is highly unexpected
BTW.  Which make implementation are you using?

> I have to specify $(srcdir)/ by hand on similar patterns,
> why does it work here?
>
No idea.

Regards,
  Stefano

Message #11 received at 10852 <at> debbugs.gnu.org (full text, mbox):

From: Akim Demaille <akim <at> lrde.epita.fr>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: 10852 <at> debbugs.gnu.org
Subject: Re: bug#10852: VPATH builds cannot recover from missing parser header
Date: Mon, 20 Feb 2012 15:23:25 +0100

Le 20 févr. 2012 à 14:58, Stefano Lattarini a écrit :

> Hi Akim.

Hi Stefano,

Thanks for the quick answer!

> The following patch extends a test which is aimed at checking
>> 
>> this, but does it in a non-vpath build :)
>> 
> But the test is wrong, because it checks that the Yacc-generated .h and .c
> files are placed in the $srcdir, while we expect them to be placed in the
> $builddir.  Do the tests added by your patch work for you?  They don't work
> for me (as I would have expected BTW).

OK, so it is my expectations which are wrong.  Probably
in an attempt to have simple minded picture in my head,
I expected the maintainer source tree to be alike an end
user tarball.  Since Automake ships the generated parsers
and scanners, I expected it to generate these guys in srcdir,
not builddir.

So indeed the patch I proposed wanted to exhibit the failure,
but it's my understanding that is incorrect.  Hence the test
failed, as _I_ expected.

>> I have a question though.  I don't understand how come
>> Make realizes it must provide $@ in srcdir too.
>> 
>> The Makefile.in features:
>> 
>>> .y.c:
>>> 	$(AM_V_YACC)$(am__skipyacc) $(SHELL) $(YLWRAP) $< y.tab.c $@ y.tab.h $*.h y.output $*.output -- $(YACCCOMPILE)
>> 
>> and at runtime I have:
>> 
>>> /bin/sh ../../build-aux/ylwrap ../../src/parse-gram.y y.tab.c ../../src/parse-gram.c y.tab.h ../../src/parse-gram.h y.output ../../src/parse-gram.output -- ./tests/bison -y -d -v --warnings=all,error --report=all 
>>> updating ../../src/parse-gram.h
>>> ../../src/parse-gram.output is unchanged
>> 
>> which is what is expected, but I don't understand why
>> it works: that $< is prefixed with $(srcdir), this I
>> understand, but why does it appear on $@?
>> 
> Unless I'm somehow sorely mistaken, that is not automake's doing; it's the
> make implementation that does such a rewrite.  Which is highly unexpected
> BTW.  Which make implementation are you using?

It's GNU Make 3.81/3.82.

Got it.  It's the dep file that introduces the $(srcdir)
part:

src/src_bison-parse-gram.o: ../../src/parse-gram.c lib/config.h \
  ../../src/system.h \
  /usr/lib/gcc/i686-apple-darwin10/4.2.1/include/limits.h \
  /usr/lib/gcc/i686-apple-darwin10/4.2.1/include/syslimits.h \
...

and it continues being propagated this way.

Is there is a way to tell Automake I want the generation
to be performed in srcdir?  In the case of Bison, we do
want to keep a copy in the repository.  I guess I cannot
use Automake for this.

Message #14 received at 10852 <at> debbugs.gnu.org (full text, mbox):

From: Akim Demaille <akim <at> lrde.epita.fr>
To: Akim Demaille <akim <at> lrde.epita.fr>
Cc: Stefano Lattarini <stefano.lattarini <at> gmail.com>, 10852 <at> debbugs.gnu.org
Subject: Re: bug#10852: VPATH builds cannot recover from missing parser header
Date: Tue, 21 Feb 2012 11:25:15 +0100

Le 20 févr. 2012 à 15:23, Akim Demaille a écrit :

>> But the test is wrong, because it checks that the Yacc-generated .h and .c
>> files are placed in the $srcdir, while we expect them to be placed in the
>> $builddir.  Do the tests added by your patch work for you?  They don't work
>> for me (as I would have expected BTW).
> 
> OK, so it is my expectations which are wrong.  Probably
> in an attempt to have simple minded picture in my head,
> I expected the maintainer source tree to be alike an end
> user tarball.  Since Automake ships the generated parsers
> and scanners, I expected it to generate these guys in srcdir,
> not builddir.
> 
> So indeed the patch I proposed wanted to exhibit the failure,
> but it's my understanding that is incorrect.  Hence the test
> failed, as _I_ expected.

The more I think about this, the more it troubles me.  It feels
wrong to me that the maintainer-source-tree is so different from
the user-source-tree.  It's easy to imagine situations where the
user, in a vpath-build, would have one parser.c in srcdir and
another in builddir.  Then, who knows what will happen...  As I
showed in my previous message, it can even behave extremely weirdly
because of pretty much hidden things, such as the dependencies
in .deps.

Besides, it prevents the maintainers from checking in their generated
files.  While it is most of the time not right to save them, I do
know several project that save them, for instance to be
able to check the diffs between revisions.  Others, such as Bison,
_must_ save these files.  Whether not to save them under VCS is
a matter of .gitignore, not of Automake's rules.

It basically boils down to whether the maintainer-src-tree should
look like a fresh check-out, or a fresh tarball.  I think it should
look like a tarball.

FWIW, Alexandre Duret-Lutz just told me that he was also troubled
by this.

Message #17 received at 10852 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Akim Demaille <akim <at> lrde.epita.fr>
Cc: 10852 <at> debbugs.gnu.org
Subject: Re: bug#10852: VPATH builds cannot recover from missing parser header
Date: Tue, 21 Feb 2012 13:00:01 +0100

On 02/21/2012 11:25 AM, Akim Demaille wrote:
> 
> Le 20 févr. 2012 à 15:23, Akim Demaille a écrit :
> 
>>> But the test is wrong, because it checks that the Yacc-generated .h and .c
>>> files are placed in the $srcdir, while we expect them to be placed in the
>>> $builddir.  Do the tests added by your patch work for you?  They don't work
>>> for me (as I would have expected BTW).
>>
>> OK, so it is my expectations which are wrong.  Probably
>> in an attempt to have simple minded picture in my head,
>> I expected the maintainer source tree to be alike an end
>> user tarball.  Since Automake ships the generated parsers
>> and scanners, I expected it to generate these guys in srcdir,
>> not builddir.
>>
>> So indeed the patch I proposed wanted to exhibit the failure,
>> but it's my understanding that is incorrect.  Hence the test
>> failed, as _I_ expected.
> 
> The more I think about this, the more it troubles me.  It feels
> wrong to me that the maintainer-source-tree is so different from
> the user-source-tree.  It's easy to imagine situations where the
> user, in a vpath-build, would have one parser.c in srcdir and
> another in builddir.  Then, who knows what will happen...
>
The one in builddir should "take precedence", in the spirit of VPATH
builds.

> As I showed in my previous message, it can even behave extremely
> weirdly because of pretty much hidden things, such as the dependencies
> in .deps.
>
This is a real problem, yes.

> Besides, it prevents the maintainers from checking in their generated
> files.
>
Well, saying that the current behaviour *prevents* them from doing so is
a bit of an exaggeration -- they could simply avoid VPATH builds during
development ;-)

> While it is most of the time not right to save them, I do
> know several project that save them, for instance to be
> able to check the diffs between revisions.  Others, such as Bison,
> _must_ save these files.  Whether not to save them under VCS is
> a matter of .gitignore, not of Automake's rules.
>

> It basically boils down to whether the maintainer-src-tree should
> look like a fresh check-out, or a fresh tarball.  I think it should
> look like a tarball.
>
Given your reasoning and concrete use cases, I'm starting to agree that
this might be the most sensible policy.  And something like this is
already being done for '.info' files (see the extensive explanations in
comments in automake.in:handle_texinfo_helper()).

Two points though:

  1. This is a somewhat invasive change IMO; so, if we decide to give
     it a try, we should do so after Automake 1.12 (which I plan
     to release in a couple of weeks, a month tops).

  2. This new policy "maintainer-src-tree should look like fresh tarball"
     should be documented/registered somewhere, preferably with a good
     rationale.

As usual, if someone wants to speed this process up, patches are very
welcome.

> FWIW, Alexandre Duret-Lutz just told me that he was also troubled
> by this.
> 

Thanks,
  Stefano

Message #20 received at 10852 <at> debbugs.gnu.org (full text, mbox):

From: Akim Demaille <akim <at> lrde.epita.fr>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: 10852 <at> debbugs.gnu.org
Subject: Re: bug#10852: VPATH builds cannot recover from missing parser header
Date: Wed, 22 Feb 2012 15:54:10 +0100

hi Stefano, Hello World!\n

Le 20 févr. 2012 à 14:58, Stefano Lattarini a écrit :

> Hi Akim.
> 
> On 02/20/2012 02:24 PM, Akim Demaille wrote:
>>> src/parse-gram.h: src/parse-gram.c
>>> 	@if test ! -f $@; then rm -f src/parse-gram.c; else :; fi
>>> 	@if test ! -f $@; then $(MAKE) $(AM_MAKEFLAGS) src/parse-gram.c; else :; fi
>> Also, why two "if"?
> For the sake of "make -n": at least GNU make and Solaris make execute
> recipes containing the $(MAKE) string even when they are running in dry
> mode; so if we didn't break the recipe above in two invocations, the
> file 'src/parse-gram.c' would be removed even upon "make -n".  Not nice.

Ah, I can understand the rationale, thanks for the explanation!

Another technical question: any reason for not using the much
shorter "||" here?

src/parse-gram.h: src/parse-gram.c
	test -f $@ || rm -f src/parse-gram.c
	test -f $@ || $(MAKE) $(AM_MAKEFLAGS) src/parse-gram.c

Message #23 received at 10852 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Akim Demaille <akim <at> lrde.epita.fr>
Cc: 10852 <at> debbugs.gnu.org
Subject: Re: bug#10852: VPATH builds cannot recover from missing parser header
Date: Thu, 23 Feb 2012 10:43:42 +0100

On 02/22/2012 03:54 PM, Akim Demaille wrote:
> hi Stefano, Hello World!\n
> 
>> On 02/20/2012 02:24 PM, Akim Demaille wrote:
>>>>
>>>> src/parse-gram.h: src/parse-gram.c
>>>> 	@if test ! -f $@; then rm -f src/parse-gram.c; else :; fi
>>>> 	@if test ! -f $@; then $(MAKE) $(AM_MAKEFLAGS) src/parse-gram.c; else :; fi
>>>
>>> Also, why two "if"?
>>
>> For the sake of "make -n": at least GNU make and Solaris make execute
>> recipes containing the $(MAKE) string even when they are running in dry
>> mode; so if we didn't break the recipe above in two invocations, the
>> file 'src/parse-gram.c' would be removed even upon "make -n".  Not nice.
> 
> Ah, I can understand the rationale, thanks for the explanation!
> 
> Another technical question: any reason for not using the much
> shorter "||" here?
>
I don't think so -- but I haven't written that code :-)

> src/parse-gram.h: src/parse-gram.c
> 	test -f $@ || rm -f src/parse-gram.c
> 	test -f $@ || $(MAKE) $(AM_MAKEFLAGS) src/parse-gram.c
> 
This seems nicer.  Care to write a patch to implement this simplification
(here and for other similar usages)?  Otherwise, I will get to do that
myself eventually.

Thanks,
  Stefano

Message #26 received at 10852 <at> debbugs.gnu.org (full text, mbox):

From: Akim Demaille <akim.demaille <at> gmail.com>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: 10852 <at> debbugs.gnu.org
Subject: Re: bug#10852: VPATH builds cannot recover from missing parser header
Date: Thu, 23 Feb 2012 16:20:23 +0100

Le 23 févr. 2012 à 10:43, Stefano Lattarini a écrit :

>> src/parse-gram.h: src/parse-gram.c
>> 	test -f $@ || rm -f src/parse-gram.c
>> 	test -f $@ || $(MAKE) $(AM_MAKEFLAGS) src/parse-gram.c
>> 
> This seems nicer.  Care to write a patch to implement this simplification
> (here and for other similar usages)?  Otherwise, I will get to do that
> myself eventually.

OK, I will try to address that.  But I have no idea
when :)  There's quite a backlog on Bison.

Message #29 received at 10852 <at> debbugs.gnu.org (full text, mbox):

From: Akim Demaille <akim <at> lrde.epita.fr>
To: Stefano Lattarini <stefano.lattarini <at> gmail.com>
Cc: 10852 <at> debbugs.gnu.org
Subject: Re: bug#10852: VPATH builds cannot recover from missing parser header
Date: Fri, 24 Feb 2012 13:55:18 +0100

[Message part 1 (text/plain, inline)]

Le 20 févr. 2012 à 14:58, Stefano Lattarini a écrit :

>> Also, why two "if"?
>> 
> For the sake of "make -n": at least GNU make and Solaris make execute
> recipes containing the $(MAKE) string even when they are running in dry
> mode; so if we didn't break the recipe above in two invocations, the
> file 'src/parse-gram.c' would be removed even upon "make -n".  Not nice.

Automake probably needs something like what follows.
Your wording is just perfect imho :)

[0001-doc-promote-Makefile-snippets-that-work-properly-wit.patch (application/octet-stream, attachment)]

[Message part 3 (text/plain, inline)]

Message #32 received at 10852 <at> debbugs.gnu.org (full text, mbox):

From: Stefano Lattarini <stefano.lattarini <at> gmail.com>
To: Akim Demaille <akim <at> lrde.epita.fr>
Cc: 10852 <at> debbugs.gnu.org
Subject: Re: bug#10852: VPATH builds cannot recover from missing parser header
Date: Fri, 24 Feb 2012 17:19:23 +0100

Hi Akim.

On 02/24/2012 01:55 PM, Akim Demaille wrote:
> 
> Le 20 févr. 2012 à 14:58, Stefano Lattarini a écrit :
> 
>>> Also, why two "if"?
>>>
>> For the sake of "make -n": at least GNU make and Solaris make execute
>> recipes containing the $(MAKE) string even when they are running in dry
>> mode; so if we didn't break the recipe above in two invocations, the
>> file 'src/parse-gram.c' would be removed even upon "make -n".  Not nice.
> 
> Automake probably needs something like what follows.
> Your wording is just perfect imho :)
> 
Thanks for preparing this patch.  ACK of course.  Can you push it yourself
(to the automake master branch), or should I apply it?

Regards,
  Stefano

Message #35 received at 10852 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: stefano.lattarini <at> gmail.com, akim <at> lrde.epita.fr, 10852 <at> debbugs.gnu.org
Subject: Re: bug#10852: VPATH builds cannot recover from missing parser header
Date: Thu, 28 May 2020 18:51:55 -0600

Back on this bug from 2012,
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=10852

    ak> Also, why two "if"?
    sl> For the sake of "make -n": 

    * doc/automake.texi (Multiple Outputs): Split commands than
    reinvoke $(MAKE) to avoid file removals during dry runs.

I pushed Akim's patch.

Regarding the bigger issue mentioned in the middle:

> It basically boils down to whether the maintainer-src-tree should
> look like a fresh check-out, or a fresh tarball.  I think it should
> look like a tarball.

Personally I agree, but unless I'm misunderstanding, GNU packages have
moved quite in the other direction, aggressively minimizing the source
tree to be as much like a checkout as possible. E.g., there is no
ChangeLog file in Automake or most other git-based GNU packages.

I certainly have no appetite for attempting to change this, and it would
fly in the face of the established "git practice" of the last N years,
so I doubt such a policy would be warmly welcomed. So I'm closing this
bug. If I'm missing the point, can discuss further, of course. --best, karl.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.