GNU bug report logs - #27532
getprogname: support for qemu

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 27532 in the body.
You can then email your comments to 27532 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: bug-grep <at> gnu.org
Subject: getprogname: support for qemu
Date: Thu, 29 Jun 2017 18:26:25 +0200

Hi,

When running the testsuite of grep-3.0 with qemu user mode, some tests fail.

How to reproduce:
- On a Debian or Ubuntu system, install package 'g++-5-aarch64-linux-gnu'.
- Install qemu-2.8.1 or qemu-2.9.0 from source.
- Prepare for executing aarch64 binaries:
  $ sudo update-binfmts --install qemu-aarch64 $HOME/inst-qemu/2.8.1/bin/qemu-aarch64 --magic '\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00' --mask '\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff' --offset 0 --credential no
  $ QEMU_LD_PREFIX=/usr/aarch64-linux-gnu; export QEMU_LD_PREFIX
- $ ../configure --host=aarch64-linux CC="aarch64-linux-gnu-gcc-5"
- $ make
- $ make check

Test failure example:
filename-lineno.pl: test invalid-re: stderr mismatch, comparing invalid-re.1 (expected) and invalid-re.E (actual)
*** invalid-re.1	Thu Jun 29 17:40:54 2017
--- invalid-re.E	Thu Jun 29 17:40:54 2017
***************
*** 1 ****
! grep: g:4: Unmatched [...
--- 1 ----
! /tmp/grep-3.0/build-arm64/src/grep: g:4: Unmatched [...

How come? The test suite invokes 'grep' from $PATH but:
1) When the x86_64 kernel is about to execute a native aarch64 binary, it
   prepares an argv with
     argv[0] = absolute file name of qemu-aarch64
     argv[1] = absolute file name of grep (it must be absolute, since it's not
               qemu's job to search for 'grep' in $PATH).
2) Inside grep, which is linked to glibc, the error() function used is the one
   from glibc. The one from gnulib, present in grep's source code, is not
   compiled. The error() function in glibc uses program_invocation_name,
   which is the absolute path of 'grep' by (1).
   The error() function in glibc does *not* use program_invocation_short_name,
   nor does it use gnulib's getprogname() which would also return
   program_invocation_short_name.

The following proof-of-concept patch fixes the problem - "make check" passes -,
but I don't know whether
  - you want something like this at all,
  - you want to limit it to the test situation. i.e. make the behaviour depend
    on some environment variable,
  - you prefer to fix the test suite instead (by removing the dirname of the
    program from the output before comparison with the expected result).

Bruno


--- grep-3.0/src/grep.c.bak	2017-02-09 02:37:33.000000000 +0100
+++ grep-3.0/src/grep.c	2017-06-29 18:07:53.072178500 +0200
@@ -2432,6 +2432,15 @@
   return result;
 }
 
+#if __GLIBC__ >= 2
+extern void (*error_print_progname) (void);
+static void
+error_print_program_invocation_short_name (void)
+{
+  fprintf (stderr, "%s: ", program_invocation_short_name);
+}
+#endif
+
 int
 main (int argc, char **argv)
 {
@@ -2445,6 +2454,10 @@
   int fread_errno;
   intmax_t default_context;
   FILE *fp;
+
+#if __GLIBC__ >= 2
+  error_print_progname = error_print_program_invocation_short_name;
+#endif
   exit_failure = EXIT_TROUBLE;
   initialize_main (&argc, &argv);
 

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: bug-grep <at> gnu.org
Subject: Re: grep test failures under qemu
Date: Thu, 29 Jun 2017 18:29:08 +0200

Oops, the title was wrong.
Corrected subject: "grep test failures under qemu"

Message #11 received at 27532 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Bruno Haible <bruno <at> clisp.org>
Cc: 27532 <at> debbugs.gnu.org
Subject: Re: bug#27532: getprogname: support for qemu
Date: Thu, 29 Jun 2017 12:50:11 -0400

Hello Bruno and all,

> On Jun 29, 2017, at 12:26, Bruno Haible <bruno <at> clisp.org> wrote:
> 
> When running the testsuite of grep-3.0 with qemu user mode, some tests fail.
> [...]
> *** 1 ****
> ! grep: g:4: Unmatched [...
> --- 1 ----
> ! /tmp/grep-3.0/build-arm64/src/grep: g:4: Unmatched [...
> [...]
>     argv[0] = absolute file name of qemu-aarch64
>     argv[1] = absolute file name of grep (it must be absolute, since it's not
>               qemu's job to search for 'grep' in $PATH).

Luckily, many GNU test scripts already use "$prog" perl variable
for the program's name in error message, so perhaps it would be possible
to accomodate qemu without code changes.

I've encountered the same for my program (datamash).
and as an ugly hack added an undocumented option to print the program's name
to set '$prog' accordingly:
https://git.savannah.gnu.org/cgit/datamash.git/tree/tests/datamash-tests.pl#n37
  ## Cross-Compiling portability hack:
  ##  under qemu/binfmt, argv[0] (which is used to report errors) will contain
  ##  the full path of the binary, if the binary is on the $PATH.
  ##  So we try to detect what is the actual returned value of the program
  ##  in case of an error.
  my $prog = `$prog_bin ---print-progname`;
  $prog = $prog_bin unless $prog;


But this hack can be avoided, if we just run 'grep' with invalid
arguments, triggering an error, then extracting the program name from STDERR.

E.g. for https://git.savannah.gnu.org/cgit/grep.git/tree/tests/filename-lineno.pl
change the following on line 26 from:
    my $prog = 'grep';
to
    my $prog = 'grep';
    my $full_prog_name = `$prog --invalid-option-for-testing 2>&1 | head -n1 | cut -f1 -d:`;
    $prog = $full_prog_name if $full_prog_name;

This is untested code, but should work more-or-less.
Once "$prog" is updated, all the tests should pass.

Hope this helps,
regards,
 - assaf

Message #14 received at 27532 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: Bruno Haible <bruno <at> clisp.org>, 27532 <at> debbugs.gnu.org
Subject: Re: bug#27532: getprogname: support for qemu
Date: Thu, 29 Jun 2017 18:10:53 -0700

[Message part 1 (text/plain, inline)]

On Thu, Jun 29, 2017 at 9:50 AM, Assaf Gordon <assafgordon <at> gmail.com> wrote:
> Hello Bruno and all,
>
>> On Jun 29, 2017, at 12:26, Bruno Haible <bruno <at> clisp.org> wrote:
>>
...
> Luckily, many GNU test scripts already use "$prog" perl variable
> for the program's name in error message, so perhaps it would be possible
> to accomodate qemu without code changes.
>
> I've encountered the same for my program (datamash).
> and as an ugly hack added an undocumented option to print the program's name
> to set '$prog' accordingly:
> https://git.savannah.gnu.org/cgit/datamash.git/tree/tests/datamash-tests.pl#n37
>   ## Cross-Compiling portability hack:
>   ##  under qemu/binfmt, argv[0] (which is used to report errors) will contain
>   ##  the full path of the binary, if the binary is on the $PATH.
>   ##  So we try to detect what is the actual returned value of the program
>   ##  in case of an error.
>   my $prog = `$prog_bin ---print-progname`;
>   $prog = $prog_bin unless $prog;
>
>
> But this hack can be avoided, if we just run 'grep' with invalid
> arguments, triggering an error, then extracting the program name from STDERR.
>
> E.g. for https://git.savannah.gnu.org/cgit/grep.git/tree/tests/filename-lineno.pl
> change the following on line 26 from:
>     my $prog = 'grep';
> to
>     my $prog = 'grep';
>     my $full_prog_name = `$prog --invalid-option-for-testing 2>&1 | head -n1 | cut -f1 -d:`;
>     $prog = $full_prog_name if $full_prog_name;
>
> This is untested code, but should work more-or-less.
> Once "$prog" is updated, all the tests should pass.

Thanks to both of you.
Does this patch solve the problem?

[grep-vs-qemu.diff (text/plain, attachment)]

Message #17 received at 27532 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: Jim Meyering <jim <at> meyering.net>
Cc: Assaf Gordon <assafgordon <at> gmail.com>, 27532 <at> debbugs.gnu.org
Subject: Re: bug#27532: getprogname: support for qemu
Date: Fri, 30 Jun 2017 18:04:24 +0200

[Message part 1 (text/plain, inline)]

Hi Jim,

> Does this patch solve the problem?

Yes, this patch fixes the filename-lineno.pl failure. Thanks.

There are 3 similar failures, still. Logs are attached.

Bruno

[in-eq-out-infloop.log.gz (application/gzip, attachment)]

[reversed-range-endpoints.log (text/x-log, attachment)]

[write-error-msg.log (text/x-log, attachment)]

Message #20 received at 27532 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Bruno Haible <bruno <at> clisp.org>
Cc: Assaf Gordon <assafgordon <at> gmail.com>, 27532 <at> debbugs.gnu.org
Subject: Re: bug#27532: getprogname: support for qemu
Date: Fri, 30 Jun 2017 11:13:23 -0700

[Message part 1 (text/plain, inline)]

On Fri, Jun 30, 2017 at 9:04 AM, Bruno Haible <bruno <at> clisp.org> wrote:
> Hi Jim,
>
>> Does this patch solve the problem?
>
> Yes, this patch fixes the filename-lineno.pl failure. Thanks.
>
> There are 3 similar failures, still. Logs are attached.

Thank you for the details.
Here's a patch that should address those others, too. Can you confirm
that it fixes those problems?

[grep-vs-qemu-v2.diff (text/plain, attachment)]

Message #23 received at 27532 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: Jim Meyering <jim <at> meyering.net>
Cc: Assaf Gordon <assafgordon <at> gmail.com>, 27532 <at> debbugs.gnu.org
Subject: Re: bug#27532: getprogname: support for qemu
Date: Sat, 01 Jul 2017 11:38:03 +0200

Hi Jim,

> Here's a patch that should address those others, too. Can you confirm
> that it fixes those problems?

Yes, with this patch, all 4 failures are gone, and "make check" proceeds
to the gnulib tests, which also all succeed.

Bruno

Message #28 received at 27532-done <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Bruno Haible <bruno <at> clisp.org>
Cc: Assaf Gordon <assafgordon <at> gmail.com>, 27532-done <at> debbugs.gnu.org
Subject: Re: bug#27532: getprogname: support for qemu
Date: Sat, 1 Jul 2017 10:56:06 -0700

On Sat, Jul 1, 2017 at 2:38 AM, Bruno Haible <bruno <at> clisp.org> wrote:
> Hi Jim,
>
>> Here's a patch that should address those others, too. Can you confirm
>> that it fixes those problems?
>
> Yes, with this patch, all 4 failures are gone, and "make check" proceeds
> to the gnulib tests, which also all succeed.

Thanks again. I've pushed it as
https://git.savannah.gnu.org/cgit/grep.git/commit/?id=9fbc2fb1823c71a9d3370e0b04886c5994000fa8

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.