GNU bug report logs - #16444
grep -r fails with "Bad file descriptor" for mingw-w64

Previous Next

To reply to this bug, email your comments to 16444 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Stephan T. Lavavej" <stl <at> nuwen.net>
To: <bug-grep <at> gnu.org>
Subject: grep -r fails with "Bad file descriptor" for mingw-w64
Date: Tue, 14 Jan 2014 06:49:48 -0800

Hi,

I'm building grep 2.16 with mingw-w64, and it works (with color!) for
grepping individual files, but not recursively:

C:\Temp\gcc>build\src\grep --version
build\src\grep (GNU grep) 2.16
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others, see
<http://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.

C:\Temp\gcc>dir /s /b somedir
C:\Temp\gcc\somedir\somefile.txt

C:\Temp\gcc>type somedir\somefile.txt
one
two
three

C:\Temp\gcc>build\src\grep -r two somedir
build\src\grep: somedir: Bad file descriptor

My installer-free build environment is available from
http://nuwen.net/mingw.html ; if necessary, I can provide exact instructions
to replicate my build (but I'm just running configure and make without any
extra options or local patches).

I've observed this since grep 2.11, also with mingw.org, so I don't believe
this is specific to mingw-w64.

Thanks,
STL

Message #8 received at 16444 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: "Stephan T. Lavavej" <stl <at> nuwen.net>, 16444 <at> debbugs.gnu.org
Subject: Re: bug#16444: grep -r fails with "Bad file descriptor" for mingw-w64
Date: Tue, 14 Jan 2014 09:00:51 -0800

Stephan T. Lavavej wrote:
> I don't believe
> this is specific to mingw-w64.

Quite possibly it's specific to mingw, though.

Mingw is low priority for us,
but if someone like you with the motivation
and experience can come up with a patch
(ideally a patch to mingw), that might be nice.

Message #11 received at 16444 <at> debbugs.gnu.org (full text, mbox):

From: "Stephan T. Lavavej" <stl <at> nuwen.net>
To: "'Paul Eggert'" <eggert <at> cs.ucla.edu>,
	<16444 <at> debbugs.gnu.org>
Subject: RE: bug#16444: grep -r fails with "Bad file descriptor" for mingw-w64
Date: Tue, 14 Jan 2014 20:26:42 -0800

I've spent hours debugging this - it is definitely mingw/mingw-w64 specific. I don't really understand what's going on, but it appears that this is the same issue as http://lists.gnu.org/archive/html/bug-gnulib/2009-01/msg00370.html . I see that you committed http://lists.gnu.org/archive/html/bug-gnulib/2010-09/msg00230.html a while ago, but there are still uses of dirfd which always returns -1 for mingw.

Figuring out how to make this work is far beyond my area of expertise (if this involved crazy C++ templates I'd be right at home).

I know mingw is a second-class citizen, but if you or someone else who understands what gnulib is doing here could investigate, that would be wonderful for everyone on Windows who uses grep.

Thanks,
STL

-----Original Message-----
From: Paul Eggert [mailto:eggert <at> cs.ucla.edu] 
Sent: Tuesday, January 14, 2014 9:01 AM
To: Stephan T. Lavavej; 16444 <at> debbugs.gnu.org
Subject: Re: bug#16444: grep -r fails with "Bad file descriptor" for mingw-w64

Stephan T. Lavavej wrote:
> I don't believe
> this is specific to mingw-w64.

Quite possibly it's specific to mingw, though.

Mingw is low priority for us,
but if someone like you with the motivation
and experience can come up with a patch
(ideally a patch to mingw), that might be nice.

Message #18 received at 16444 <at> debbugs.gnu.org (full text, mbox):

From: David Grayson <davidegrayson <at> gmail.com>
To: 16444 <at> debbugs.gnu.org
Subject: bug #16444: grep -r fails with "Bad file descriptor" for mingw-w64
Date: Tue, 10 Mar 2015 22:20:28 -0700

[Message part 1 (text/plain, inline)]

Hello.

I am encountering this bug too.  I would like to compile grep 2.21 for
Windows using mingw-w64 and Git Bash.  But the grep that I end up compiling
exhibits several bugs related to recursive operation.  To reproduce the
bugs, I made a directory called "testdir" with two empty directories named
"a" and "b" inside it.  I used "cd" to go into the testdir.

Here is a shell session showing bug A:

  $ ../src/grep.exe a -r
  c:\Users\David\Documents\grep\grep-2.21\src\grep.exe: 9: Bad file
descriptor

Here is a shell session showing bug B:

  $ ../src/grep.exe a -R
  c:\Users\David\Documents\grep\grep-2.21\src\grep.exe: warning: a:
recursive directory loop
  c:\Users\David\Documents\grep\grep-2.21\src\grep.exe: warning: b:
recursive directory loop

Bug A is more important because it prevents grep from printing the desired
output, so I am focussing on it first.  I think I have narrowed it down
enough to determine that it is actually a bug in the fts implementation
from gnulib, or in one of its dependencies (like mingw-w64 of msvcrt.dll).
If I put this small block of code at the top of main, it causes the same
error:

  { // tmphax
    program_name = "silly";
    filename = "emptydir";

    char *fts_arg[2];
    fts_arg[0] = (char *)filename;
    fts_arg[1] = NULL;
    int fts_opts = FTS_CWDFD | FTS_NOSTAT | FTS_TIGHT_CYCLE_CHECK
      | FTS_COMFOLLOW | FTS_PHYSICAL;
    FTS *fts = fts_open(fts_arg, fts_opts, NULL);
    printf("fts_open returned %p\n", fts);

    FTSENT *ent;
    ent = fts_read(fts);
    ent = fts_read(fts);
    printf("  fts_path=   %s\n", ent->fts_path);
    printf("  fts_accpath=%s\n", ent->fts_accpath);
    printf("  fts_name=   %s\n", ent->fts_name);
    if(ent->fts_info == FTS_ERR)
    {
      printf("  fts_info=   FTS_ERR\n");
      suppressible_error(filename, ent->fts_errno);
    }
    return 36;
  }

Note that "emptydir" is a directory inside the CWD that is empty.  The
output from that block of code is:

  fts_open returned 008222B0
    fts_path=   emptydir
    fts_accpath=emptydir
    fts_name=   emptydir
    fts_info=   FTS_ERR
  silly: emptydir: Bad file descriptor

I noticed that the Git repository for gnulib has some recent bug fixes for
the fts implementation, so I tried copying the fts* files from gnulib
commit 0f4fa16 into grep's lib directory, but that did not make any
difference.  My next step will probably be to take grep out of the picture
and report this as a bug to the gnulib team (or find an existing bug
report, like the one mentioned earlier in this thread).

Note that for Bug A, grep doesn't print "9" as the filename in the error
message every time.  It seems to be printing a random character that is
different each time, because grep is reading beyond the bounds of a string
and probably doing undefined behavior!  We can call that Bug C.  If a grep
maintainer is interested in fixing Bug C, they should improve the logic at
this line in grepdirent in grep.c:

  filename = ent->fts_path + filename_prefix_len;

If you run "grep a -r" then it seems that fts_path can be ".", while
filename_prefix_len is 2.  So it is really bad to execute the line above in
that case, because filename will be set to point past the end of the string
and then later used when printing error messages.

I hope this was helpful.

--David Grayson

[Message part 2 (text/html, inline)]

Message #21 received at 16444 <at> debbugs.gnu.org (full text, mbox):

From: Norihiro Tanaka <noritnk <at> kcn.ne.jp>
To: David Grayson <davidegrayson <at> gmail.com>
Cc: 16444 <at> debbugs.gnu.org
Subject: Re: bug #16444: grep -r fails with "Bad file descriptor" for mingw-w64
Date: Wed, 11 Mar 2015 23:06:22 +0900

On Tue, 10 Mar 2015 22:20:28 -0700
David Grayson <davidegrayson <at> gmail.com> wrote:

> Hello.
> 
> I am encountering this bug too.  I would like to compile grep 2.21 for
> Windows using mingw-w64 and Git Bash.  But the grep that I end up compiling
> exhibits several bugs related to recursive operation.  To reproduce the
> bugs, I made a directory called "testdir" with two empty directories named
> "a" and "b" inside it.  I used "cd" to go into the testdir.
> 
> Here is a shell session showing bug A:
> 
>   $ ../src/grep.exe a -r
>   c:\Users\David\Documents\grep\grep-2.21\src\grep.exe: 9: Bad file
> descriptor
> 
> Here is a shell session showing bug B:
> 
>   $ ../src/grep.exe a -R
>   c:\Users\David\Documents\grep\grep-2.21\src\grep.exe: warning: a:
> recursive directory loop
>   c:\Users\David\Documents\grep\grep-2.21\src\grep.exe: warning: b:
> recursive directory loop

SAME_INODE macro defined in same-inode.h to be used by fts.c is as
following.

#  define SAME_INODE(a, b)    \
    ((a).st_ino == (b).st_ino \
     && (a).st_dev == (b).st_dev)

However, MinGW always returns ST_INO = 0, which is POSIX imcompatible.
I see that it leads to the result that all directories is same.

So if we compare the file path instead of st_ino, the bug may be fixed.

Message #24 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Jackson <pj <at> usa.net>
To: bug-grep <at> gnu.org
Subject: Re: bug#16444: bug #16444: grep -r fails with "Bad file descriptor"
 for mingw-w64
Date: Wed, 11 Mar 2015 19:23:03 -0500

Norihiro Tanaka wrote:
>> So if we compare the file path instead of st_ino, the bug may be fixed.

Or perhaps compare st_ino if it's non-zero, else compare path.

# define IS_SPECIAL_MinGW_Case(a) ((a).st_.ino == 0)

# define SAME_INODE(a, b)    \
    (IS_SPECIAL_MinGW_Case(a) ? \
            paths_compare_equal((a), (b)) : \
            (a).st_ino == (b).st_ino ) \
     && (a).st_dev == (b).st_dev)

Inode number compare works well when supported, and has different semantics
(such as in handling of hard links) in some cases, which semantics should not be
changed, in my view.  MinGW needs to be "quarantined" in this circumstance,
which is what the test for a non-zero inode would (apparently, from what you say)
accomplish.

In other words, our special handling for MinGW should not alter how we handle
everyone else.

-- 
                Paul Jackson
                pj <at> usa.net

Message #27 received at 16444 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: David Grayson <davidegrayson <at> gmail.com>, 16444 <at> debbugs.gnu.org
Subject: Re: bug#16444: bug #16444: grep -r fails with "Bad file descriptor"
 for mingw-w64
Date: Wed, 11 Mar 2015 22:46:56 -0700

[Message part 1 (text/plain, inline)]

David Grayson wrote:
> We can call that Bug C.  If a grep
> maintainer is interested in fixing Bug C, they should improve the logic at
> this line in grepdirent in grep.c:
>
>    filename = ent->fts_path + filename_prefix_len;

Thanks for reporting this bug.  I installed the attached patch for that.

[0001-grep-output-.-file-name-in-diagnostic.patch (text/x-patch, attachment)]

Message #30 received at 16444 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Norihiro Tanaka <noritnk <at> kcn.ne.jp>, 
 David Grayson <davidegrayson <at> gmail.com>
Cc: 16444 <at> debbugs.gnu.org
Subject: Re: bug#16444: bug #16444: grep -r fails with "Bad file descriptor"
 for mingw-w64
Date: Wed, 11 Mar 2015 23:15:12 -0700

Norihiro Tanaka wrote:
> MinGW always returns ST_INO = 0, which is POSIX incompatible.

Yes, this is a known deficiency in MinGW that is documented in Gnulib, here:

http://www.gnu.org/software/gnulib/manual/html_node/sys_002fstat_002eh.html

(search for "st_ino").

When we discovered this deficiency years ago, we decided that it'd be too much 
of a pain to work around it then.  Please see the thread starting here:

http://lists.gnu.org/archive/html/bug-gnulib/2009-09/msg00252.html

in which a patch was installed that tries to work around some of the MinGW 
deficiency; unfortunately the patch broke a lot of things and was backed out. 
I'm not optimistic about a fix this time either.

The deficiency should be fixed in MinGW, so that functions like 'stat' behave as 
programs expect.  In the meantime, I suggest not using recursive grep in MinGW.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.