GNU bug report logs - #29446
GNU grep 3.1 fails test suite when PCRE lacks JIT

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 29446 in the body.
You can then email your comments to 29446 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "A. Wilcox" <awilfox <at> adelielinux.org>
To: bug-grep <at> gnu.org
Subject: GNU grep 3.1 fails test suite when PCRE lacks JIT
Date: Sat, 25 Nov 2017 15:35:03 -0600

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi there grep developers,

I'm trying to package GNU grep 3.1 on 64-bit PowerPC (big-endian) for
our Linux distribution, Adélie Linux.  It seems to be working quite
well, except there is a failing test: pcre-jitstack.

"grep: internal PCRE error: -21" (which is PCRE_ERROR_RECURSIONLIMIT)
is printed.  I believe that this is because we have to disable JIT on
the ppc64 platform (because the JIT compiler is badly broken on it).

I am not very familiar with either of the grep or PCRE codebases so I
am not sure exactly how to best skip the test when PCRE does not have
JIT support enabled.  Otherwise, I would have included a patch.

Thank you for your time.

Best regards,
- --arw

- -- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJaGeIDAAoJEMspy1GSK50UYKoP/3xmewMMArDgld6jvXdpFEdG
GQeRly0BWxckUksE3FNRdfw9OBS2VAJKd8FpU+3wbRBp0Lnf9btTBGiHD9vB0cBX
kRQIADw87M6uDIBKedN1lGMjr9E8BcLOj/5GTRIvd6eKDVZspVRW7bHGdb+n4om4
uVlmpyP2niSfYkcE/F9Bv0BQrjjyHUXKbKP+u7O3nA/VJt27tOM1RuIQSnKE2sLE
9hOd8YY88zPP8xAXpSccvq1qnbsCH+X8Av/nzyMGc/xdcvlkbepweeLWZ9eF0+x8
oqBE/fg7EQ4VawUpCTlgwD7+K2c+pXvXIkhksxxNgq6YP1UccCiz8T6JDZsq6VAY
RU4MdlxRrKWMKjVf+GQXBR0tr+F+LeOT9QXC6+vlqISjoT2Ev9NM2HIUamii5uBU
I+9q3Yc8mw2adL07kiMCtKRCMVSoa5Qx+lNaMIsSGCYLOzNajZr/IjzZjbMKSTbC
jb2yx/c6zTshghl1+g/OJYyEikia8T4qXnLp7LwRgF5INp2ep+4rFGk3N5K8SD8X
Xg2vh6QQTqORQChGRHMI9fTWvj5zrxboSCFqinK7ASCZO3uRE/AGvkJcmO+LPPz8
ZPSksPvHvHZCZ76Wq1R8Fo3PAcOPtX4YCEUyNwMb4aOlNORZohFdaZJIMmTdkq9p
r1GLLYi8RjC6V3zaCDw7
=Of5f
-----END PGP SIGNATURE-----

Message #8 received at 29446 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: "A. Wilcox" <awilfox <at> adelielinux.org>, 29446 <at> debbugs.gnu.org
Subject: Re: bug#29446: GNU grep 3.1 fails test suite when PCRE lacks JIT
Date: Sat, 25 Nov 2017 20:15:51 -0800

[Message part 1 (text/plain, inline)]

I'd rather fix the problem than skip the test. I installed the attached into the 
master branch on Savannah: does it fix the problem for you?

PS. Please use the name "Adélie GNU/Linux" when referring to that distribution, 
to give proper credit to the GNU project. Thanks.

[0001-grep-port-better-to-Ad-lie-GNU-Linux-64-bit-ppc.txt (text/plain, attachment)]

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Thomas Deutschmann <whissi <at> gentoo.org>
To: bug-grep <at> gnu.org
Subject: bug#29446: GNU grep 3.1 fails test suite when PCRE lacks JIT
Date: Mon, 15 Oct 2018 18:52:20 +0200

[Message part 1 (text/plain, inline)]

Hi,

we have a similar problem on Gentoo Linux [1] where we allow to disable
JIT support in PCRE.

With the patch applied from previous mail [2], we are still seeing
failure of test "pcre-jitstack":

FAIL: pcre-jitstack
===================

++ initial_cwd_=/var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests
+++ testdir_prefix_
+++ printf gt
++ pfx_=gt
+++ mktempd_ /var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests gt-pcre-jitstack.XXXX
+++ case $# in
+++ destdir_=/var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests
+++ template_=gt-pcre-jitstack.XXXX
+++ MAX_TRIES_=4
+++ case $destdir_ in
+++ destdir_slash_=/var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests/
+++ case $template_ in
++++ unset TMPDIR
+++ d=/var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests/gt-pcre-jitstack.71Wt
+++ case $d in
+++ :
+++ test -d /var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests/gt-pcre-jitstack.71Wt
++++ ls -dgo /var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests/gt-pcre-jitstack.71Wt
+++ perms='drwx------ 2 4096 Oct 15 18:33 /var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests/gt-pcre-jitstack.71Wt'
+++ case $perms in
+++ :
+++ echo /var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests/gt-pcre-jitstack.71Wt
+++ return
++ test_dir_=/var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests/gt-pcre-jitstack.71Wt
++ cd /var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests/gt-pcre-jitstack.71Wt
++ gl_init_sh_nl_='
'
++ IFS=' 	
'
++ for sig_ in 1 2 3 13 15
+++ expr 1 + 128
++ eval 'trap '\''Exit 129'\'' 1'
+++ trap 'Exit 129' 1
++ for sig_ in 1 2 3 13 15
+++ expr 2 + 128
++ eval 'trap '\''Exit 130'\'' 2'
+++ trap 'Exit 130' 2
++ for sig_ in 1 2 3 13 15
+++ expr 3 + 128
++ eval 'trap '\''Exit 131'\'' 3'
+++ trap 'Exit 131' 3
++ for sig_ in 1 2 3 13 15
+++ expr 13 + 128
++ eval 'trap '\''Exit 141'\'' 13'
+++ trap 'Exit 141' 13
++ for sig_ in 1 2 3 13 15
+++ expr 15 + 128
++ eval 'trap '\''Exit 143'\'' 15'
+++ trap 'Exit 143' 15
++ trap remove_tmp_ 0
+ path_prepend_ ../src
+ test 1 '!=' 0
+ path_dir_=../src
+ case $path_dir_ in
+ abs_path_dir_=/var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests/../src
+ case $abs_path_dir_ in
+ PATH=/var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests/../src:/var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/src:./src:/usr/lib/portage/python3.6/ebuild-helpers/xattr:/usr/lib/portage/python3.6/ebuild-helpers:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/i686-pc-linux-gnu/gcc-bin/7.3.0:/usr/lib/llvm/6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin
+ create_exe_shims_ /var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests/../src
+ case $EXEEXT in
+ return 0
+ shift
+ test 0 '!=' 0
+ export PATH
+ require_pcre_
+ echo .
+ grep -P .
.
+ compare /dev/null err
+ compare_dev_null_ /dev/null err
+ test 2 = 2
+ test x/dev/null = x/dev/null
+ test -s err
+ return 0
+ return 0
+ for p in 'base64 -d' 'base64 -D' 'openssl base64 -d' "perl -MMIME::Base64 -0777ne 'print decode_base64(\$_)'" FAIL
+ test 'base64 -d' = FAIL
++ echo eA==
+ x=x
+ test Xx = Xx
+ eval 'b64_decode() { base64 -d; }'
+ break
+ foo=foo
+ test Xfoo = Xfoo
+ fail=0
+ b64_decode
+ base64 -d
+ gzip -d pcrejit.txt
+ LC_ALL=C
+ grep -P -n '^([/](?!/)|[^/])*~/.*' pcrejit.txt
grep: internal PCRE error: -21
+ test 2 '!=' 1
+ ulimit -s unlimited
+ ulimit -s unlimited
+ returns_ 1 env LC_ALL=C grep -P -n '^([/](?!/)|[^/])*~/.*' pcrejit.txt
grep: internal PCRE error: -21
+ fail=1
+ Exit 1
+ set +e
+ exit 1
+ exit 1
+ remove_tmp_
+ __st=1
+ cleanup_
+ :
+ test '' = yes
+ cd /var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests
+ chmod -R u+rwx /var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests/gt-pcre-jitstack.71Wt
+ rm -rf /var/tmp/portage/sys-apps/grep-3.1/work/grep-3.1/tests/gt-pcre-jitstack.71Wt
+ exit 1
FAIL pcre-jitstack (exit status: 1)


See also:
=========
[1] https://bugs.gentoo.org/569816

[2] https://www.mail-archive.com/bug-grep <at> gnu.org/msg06922.html


-- 
Regards,
Thomas Deutschmann / Gentoo Linux Developer
C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5

[signature.asc (application/pgp-signature, attachment)]

Message #14 received at 29446 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: "A. Wilcox" <awilfox <at> adelielinux.org>
Cc: 29446 <at> debbugs.gnu.org, Thomas Deutschmann <whissi <at> gentoo.org>
Subject: Re: GNU grep 3.1 fails test suite when PCRE lacks JIT
Date: Tue, 31 Dec 2019 12:41:52 -0800

Re this old bug report <https://bugs.gnu.org/29446>:

I attempted to reproduce this bug on Ubuntu 18.04.3 LTS x86-64, by building pcre
8.43 by hand with "./configure --disable-jit --prefix=/tmp/prefix" and then
compiling and building GNU grep master with the pcre that had JIT disabled. No
luck; I couldn't reproduce the bug. If you have any advice as to how to
reproduce the bug on Ubuntu or Fedora x86-64 (the systems I have easy access to)
please let me know.

Message #17 received at 29446 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Deutschmann <whissi <at> gentoo.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>, "A. Wilcox" <awilfox <at> adelielinux.org>
Cc: 29446 <at> debbugs.gnu.org
Subject: Re: GNU grep 3.1 fails test suite when PCRE lacks JIT
Date: Thu, 2 Jan 2020 19:02:20 +0100

[Message part 1 (text/plain, inline)]

Hi Paul,

On 2019-12-31 21:41, Paul Eggert wrote:
> Re this old bug report <https://bugs.gnu.org/29446>:
> 
> I attempted to reproduce this bug on Ubuntu 18.04.3 LTS x86-64, by building pcre
> 8.43 by hand with "./configure --disable-jit --prefix=/tmp/prefix" and then
> compiling and building GNU grep master with the pcre that had JIT disabled. No
> luck; I couldn't reproduce the bug. If you have any advice as to how to
> reproduce the bug on Ubuntu or Fedora x86-64 (the systems I have easy access to)
> please let me know.

Is it suitable for you to run Gentoo in a chrooted environment?

Please see the following howto which I just verified on latest Debian
version:

> # Setup Gentoo chroot using stage3
> cd /tmp
> mkdir /tmp/gentoo
> wget http://distfiles.gentoo.org/releases/amd64/autobuilds/20200101T214502Z/stage3-amd64-20200101T214502Z.tar.xz
> tar -xaf stage3-amd64-*.tar.xz -C /tmp/gentoo
> mount -t proc /proc gentoo/proc
> mount --rbind /sys gentoo/sys
> mount --make-rslave gentoo/sys
> mount --rbind /dev gentoo/dev
> mount --make-rslave gentoo/dev
> 
> # Chroot into Gentoo
> env -i HOME=/root TERM=$TERM $(which chroot) /tmp/gentoo /bin/bash --login
> 
> # In Gentoo chroot...
> export PS1="(chroot) $PS1"
> 
> # Make sure name resolution works...
> echo 'nameserver 8.8.8.8' > /etc/resolv.conf
> 
> # Get latest daily portage snapshot
> emerge-webrsync
> 
> # Tell portage we want FEATURES=test for sys-apps/grep
> mkdir /etc/portage/env
> echo 'FEATURES="test"' > /etc/portage/env/run-tests.conf
> echo 'sys-apps/grep run-tests.conf' >> /etc/portage/package.env
> 
> # Build dev-libs/libpcre without jit to trigger the error
> echo 'dev-libs/libpcre -jit' >> /etc/portage/package.use/gnu_bug29446
> emerge -1 dev-libs/libpcre
> 
> # Chdir to grep ebuild
> cd $(portageq get_repo_path / gentoo)/sys-apps/grep
> 
> # Normally you would use `emerge ...` to install package and all deps, but because we don't want to merge package on disk and are only interested in running tests manually, we will use `ebuild` command
> ebuild grep-3.3-r1.ebuild clean test
> 
> # You should now see test 'pcre-jitstack' failing.
> # Feel free to inspect source and run test again:
> su portage --shell /bin/bash
> 
> # As portage use, chdir into work dir
> cd ~/sys-apps/grep-3.3-r1/work/grep-3.3
> 
> make check TESTS=pcre-jitstack


-- 
Regards,
Thomas Deutschmann / Gentoo Linux Developer
C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5

[signature.asc (application/pgp-signature, attachment)]

Message #20 received at 29446 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Thomas Deutschmann <whissi <at> gentoo.org>,
 "A. Wilcox" <awilfox <at> adelielinux.org>
Cc: 29446 <at> debbugs.gnu.org
Subject: Re: GNU grep 3.1 fails test suite when PCRE lacks JIT
Date: Thu, 2 Jan 2020 12:38:25 -0800

On 1/2/20 10:02 AM, Thomas Deutschmann wrote:
> Is it suitable for you to run Gentoo in a chrooted environment?

Thanks, but I wouldn't consider that easy enough access. I wouldn't have 
decent debugging tools without further downloading and installing them 
into the chrooted environment, and I don't know Gentoo so this would 
take some time. (Similarly for my building a Gentoo VM.)

Message #23 received at 29446 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Deutschmann <whissi <at> gentoo.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>, "A. Wilcox" <awilfox <at> adelielinux.org>
Cc: 29446 <at> debbugs.gnu.org
Subject: Re: GNU grep 3.1 fails test suite when PCRE lacks JIT
Date: Thu, 2 Jan 2020 22:47:25 +0100

[Message part 1 (text/plain, inline)]

Hi,

OK, I managed to reproduce the error using pure Debian:

$ mkdir /tmp/pcre-test
$ cd /tmp/pcre-test

$ wget https://ftp.pcre.org/pub/pcre/pcre-8.43.tar.gz
$ tar -xaf pcre-8.43.tar.gz

$ cd pcre-8.43/
$ ./configure --prefix=/tmp/pcre-test/build \
--with-match-limit-recursion=8192 \
--enable-pcregrep-libbz2 \
--enable-cpp \
--disable-jit \
--disable-pcregrep-jit \
--disable-pcre16 \
--disable-pcre32 \
--disable-pcretest-libedit \
--enable-pcretest-libreadline \
--disable-static \
--enable-utf \
--enable-unicode-properties \
--enable-pcregrep-libz \
--enable-pcre8 \
--enable-shared

$ make
$ make install

$ cd ..
$ wget https://ftp.gnu.org/gnu/grep/grep-3.3.tar.xz
$ tar -xaf grep-3.3.tar.xz
$ cd grep-3.3/
$ PKG_CONFIG_PATH=/tmp/pcre-test/build/lib/pkgconfig ./configure
$ make

# Now check src/grep
$ lddtree src/grep
grep => src/grep (interpreter => /lib64/ld-linux-x86-64.so.2)
    libpcre.so.1 => not found
[...]

$ export LD_LIBRARY_PATH=/tmp/pcre-test/build/lib

# Check with lddtree again to be sure our local pcre build is being used
$ lddtree src/grep
grep => src/grep (interpreter => /lib64/ld-linux-x86-64.so.2)
    libpcre.so.1 => /tmp/pcre-test/build/lib/libpcre.so.1
[...]

# Test that src/grep works:
$ echo . | src/grep -P .

# Now run test which will fail:
$ make check TESTS=pcre-jitstack

> + gzip -d pcrejit.txt
> + LC_ALL=C
> + grep -P -n '^([/](?!/)|[^/])*~/.*' pcrejit.txt
> grep: internal PCRE error: -21
> + test 2 '!=' 1
> + ulimit -s unlimited
> + ulimit -s unlimited
> + returns_ 1 env LC_ALL=C grep -P -n '^([/](?!/)|[^/])*~/.*' pcrejit.txt
> grep: internal PCRE error: -21
> + fail=1
> + Exit 1


-- 
Regards,
Thomas Deutschmann / Gentoo Linux Developer
C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5

[signature.asc (application/pgp-signature, attachment)]

Message #28 received at 29446-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Thomas Deutschmann <whissi <at> gentoo.org>
Cc: "A. Wilcox" <awilfox <at> adelielinux.org>, 29446-done <at> debbugs.gnu.org
Subject: Re: GNU grep 3.1 fails test suite when PCRE lacks JIT
Date: Wed, 9 Sep 2020 16:10:41 -0700

[Message part 1 (text/plain, inline)]

On 1/2/20 1:47 PM, Thomas Deutschmann wrote:

> I managed to reproduce the error using pure Debian:

Thanks, with that recipe I reproduced the problem on Ubuntu 18.04.5, and 
installed the attached. I'll boldly close the bug report.

libpcre is quite a pain to use, and makes 'grep' less reliable. Perhaps somebody 
with some time could see whether libpcre2 is any better.

[0001-grep-fix-PCRE-JIT-test-when-JIT-not-available.patch (text/x-patch, attachment)]

[0002-grep-fix-logic-for-growing-PCRE-JIT-stack.patch (text/x-patch, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.