GNU bug report logs - #26441
Gnulib’s ‘test-lock’ fails to complete on machines with >= 32 cores

Previous Next

Package: guix;

Reported by: ludo <at> gnu.org (Ludovic Courtès)

Date: Tue, 11 Apr 2017 08:18:01 UTC

Severity: important

Done: ludo <at> gnu.org (Ludovic Courtès)

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 26441 in the body.
You can then email your comments to 26441 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Tue, 11 Apr 2017 08:18:01 GMT) Full text and rfc822 format available.

Acknowledgement sent to ludo <at> gnu.org (Ludovic Courtès):
New bug report received and forwarded. Copy sent to bug-guix <at> gnu.org. (Tue, 11 Apr 2017 08:18:01 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: bug-guix <at> gnu.org
Cc: Mathieu Othacehe <m.othacehe <at> gmail.com>
Subject: Gnulib’s ‘test-lock’ fails to
 complete on machines with >= 32 cores
Date: Tue, 11 Apr 2017 10:16:54 +0200
With current ‘master’, Gnulib’s ‘test-lock’ (a test found in many GNU
packages that use Gnulib, such as gettext-0.19.8.1) fails to complete in
1 hour almost systematically on bayfront, which is a 32-core x86_64
machine.  The backtrace is like this:

--8<---------------cut here---------------start------------->8---
(gdb) thread apply all bt

Thread 21 (LWP 5636):
#0  0x00007ffff77a9602 in pthread_rwlock_wrlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x000000000040172f in rwlock_mutator_thread (arg=<optimized out>) at test-lock.c:348
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 20 (LWP 5635):
#0  0x00007ffff77a9602 in pthread_rwlock_wrlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x000000000040172f in rwlock_mutator_thread (arg=<optimized out>) at test-lock.c:348
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 19 (LWP 5634):
#0  0x00007ffff77a9602 in pthread_rwlock_wrlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x000000000040172f in rwlock_mutator_thread (arg=<optimized out>) at test-lock.c:348
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 18 (LWP 5633):
#0  0x00007ffff77a9602 in pthread_rwlock_wrlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x000000000040172f in rwlock_mutator_thread (arg=<optimized out>) at test-lock.c:348
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 17 (LWP 5632):
#0  0x00007ffff77a9602 in pthread_rwlock_wrlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x000000000040172f in rwlock_mutator_thread (arg=<optimized out>) at test-lock.c:348
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 16 (LWP 5631):
#0  0x00007ffff77a9602 in pthread_rwlock_wrlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x000000000040172f in rwlock_mutator_thread (arg=<optimized out>) at test-lock.c:348
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 15 (LWP 5630):
#0  0x00007ffff77a9602 in pthread_rwlock_wrlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x000000000040172f in rwlock_mutator_thread (arg=<optimized out>) at test-lock.c:348
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 14 (LWP 5629):
#0  0x00007ffff77a93d6 in pthread_rwlock_wrlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x000000000040172f in rwlock_mutator_thread (arg=<optimized out>) at test-lock.c:348
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 13 (LWP 5628):
#0  0x00007ffff77a9602 in pthread_rwlock_wrlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x000000000040172f in rwlock_mutator_thread (arg=<optimized out>) at test-lock.c:348
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 12 (LWP 5627):
#0  0x00007ffff77a9602 in pthread_rwlock_wrlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x000000000040172f in rwlock_mutator_thread (arg=<optimized out>) at test-lock.c:348
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 11 (LWP 5626):
#0  0x00007ffff74cf777 in sched_yield () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6
#1  0x0000000000401810 in gl_thread_yield () at test-lock.c:104
#2  rwlock_checker_thread (arg=<optimized out>) at test-lock.c:381
#3  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#4  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 10 (LWP 5625):
#0  0x00007ffff74cf777 in sched_yield () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6
#1  0x0000000000401810 in gl_thread_yield () at test-lock.c:104
#2  rwlock_checker_thread (arg=<optimized out>) at test-lock.c:381
#3  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#4  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 9 (LWP 5624):
#0  0x00007ffff77a9dd7 in pthread_rwlock_unlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x0000000000401807 in rwlock_checker_thread (arg=<optimized out>) at test-lock.c:378
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 8 (LWP 5623):
#0  0x00007ffff77a9dd7 in pthread_rwlock_unlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x0000000000401807 in rwlock_checker_thread (arg=<optimized out>) at test-lock.c:378
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 7 (LWP 5622):
#0  0x00007ffff77a9dd7 in pthread_rwlock_unlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x0000000000401807 in rwlock_checker_thread (arg=<optimized out>) at test-lock.c:378
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 6 (LWP 5621):
#0  0x00007ffff77a9dd7 in pthread_rwlock_unlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x0000000000401807 in rwlock_checker_thread (arg=<optimized out>) at test-lock.c:378
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 5 (LWP 5620):
#0  0x00007ffff74cf777 in sched_yield () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6
#1  0x0000000000401810 in gl_thread_yield () at test-lock.c:104
#2  rwlock_checker_thread (arg=<optimized out>) at test-lock.c:381
#3  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#4  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 4 (LWP 5619):
#0  0x00007ffff74cf777 in sched_yield () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6
#1  0x0000000000401810 in gl_thread_yield () at test-lock.c:104
#2  rwlock_checker_thread (arg=<optimized out>) at test-lock.c:381
#3  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#4  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 3 (LWP 5618):
#0  0x00007ffff77a9dd7 in pthread_rwlock_unlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x0000000000401807 in rwlock_checker_thread (arg=<optimized out>) at test-lock.c:378
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 2 (LWP 5617):
#0  0x00007ffff77a9dd7 in pthread_rwlock_unlock () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x0000000000401807 in rwlock_checker_thread (arg=<optimized out>) at test-lock.c:378
#2  0x00007ffff77a4454 in start_thread () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#3  0x00007ffff74e67bf in clone () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libc.so.6

Thread 1 (LWP 5605):
#0  0x00007ffff77a568d in pthread_join () from target:/gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25/lib/libpthread.so.0
#1  0x0000000000400e29 in gl_thread_join (retvalp=0x0, thread=<optimized out>) at test-lock.c:99
#2  test_rwlock () at test-lock.c:408
#3  main () at test-lock.c:682
--8<---------------cut here---------------end--------------->8---

Mathieu Othacehe reported the same issue on a 32-core machine.

This may be fixed by Gnulib commit
480d374e596a0ee3fed168ab42cd84c313ad3c89 (not present in
gettext-0.19.8.1), which introduces this:

  --- a/tests/test-lock.c
  +++ b/tests/test-lock.c
  @@ -50,6 +50,13 @@
      Uncomment this to see if the operating system has a fair scheduler.  */
   #define EXPLICIT_YIELD 1

  +/* Whether to use 'volatile' on some variables that communicate information
  +   between threads.  If set to 0, a lock is used to protect these variables.
  +   If set to 1, 'volatile' is used; this is theoretically equivalent but can
  +   lead to much slower execution (e.g. 30x slower total run time on a 40-core
  +   machine.  */
  +#define USE_VOLATILE 0

30x slower could exceed the default max-silent-timeout, which is 1 hour.

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Wed, 12 Apr 2017 09:02:01 GMT) Full text and rfc822 format available.

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: bug-guix <at> gnu.org
Subject: Re: Gnulib’s ‘test-lock’ fails
 to complete on machines with >= 32 cores
Date: Wed, 12 Apr 2017 11:01:16 +0200
Hi,

> This may be fixed by Gnulib commit
> 480d374e596a0ee3fed168ab42cd84c313ad3c89 (not present in

On gettext, commit 1afbcb06fded2a427b761dd1615b1e48e1e853cc seems to fix
the problem. I ran three consecutive tests :

--8<---------------cut here---------------start------------->8---
mathieu <at> elbruz /tmp/guix-build-gettext-next-0.19.8.1.drv-0/gettext-0.19.8.1$ time ./gettext-tools/gnulib-tests/.libs/test-lock 
Starting test_lock ... OK
Starting test_rwlock ... OK
Starting test_recursive_lock ... OK
Starting test_once ... OK

real    3m50.493s
user    31m15.128s
sys     6m15.440s

mathieu <at> elbruz /tmp/guix-build-gettext-next-0.19.8.1.drv-0/gettext-0.19.8.1$ time ./gettext-tools/gnulib-tests/.libs/test-lock 
Starting test_lock ... OK
Starting test_rwlock ... OK
Starting test_recursive_lock ... OK
Starting test_once ... OK

real    4m11.575s
user    34m10.304s
sys     6m46.288s

mathieu <at> elbruz /tmp/guix-build-gettext-next-0.19.8.1.drv-0/gettext-0.19.8.1$ time ./gettext-tools/gnulib-tests/.libs/test-lock 
Starting test_lock ... OK
Starting test_rwlock ... OK
Starting test_recursive_lock ... OK
Starting test_once ... OK

real    3m21.155s
user    27m6.328s
sys     5m27.248s
--8<---------------cut here---------------end--------------->8---

Tests succeeds in a reasonable time and it can be backported to our
version of gettext.

However, I don't fully understand the difference between commit
1afbcb06fded2a427b761dd1615b1e48e1e853cc in gettext and commit
480d374e596a0ee3fed168ab42cd84c313ad3c89 in gnulib.

Mathieu




Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Wed, 12 Apr 2017 11:55:01 GMT) Full text and rfc822 format available.

Message #11 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: 26441 <at> debbugs.gnu.org
Cc: Mathieu Othacehe <m.othacehe <at> gmail.com>
Subject: Re: bug#26441: Gnulib’s ‘test-lock’ fails to complete on machines with >= 32 cores
Date: Wed, 12 Apr 2017 13:53:57 +0200
ludo <at> gnu.org (Ludovic Courtès) skribis:

> This may be fixed by Gnulib commit
> 480d374e596a0ee3fed168ab42cd84c313ad3c89 (not present in
> gettext-0.19.8.1), which introduces this:
>
>   --- a/tests/test-lock.c
>   +++ b/tests/test-lock.c
>   @@ -50,6 +50,13 @@
>       Uncomment this to see if the operating system has a fair scheduler.  */
>    #define EXPLICIT_YIELD 1
>
>   +/* Whether to use 'volatile' on some variables that communicate information
>   +   between threads.  If set to 0, a lock is used to protect these variables.
>   +   If set to 1, 'volatile' is used; this is theoretically equivalent but can
>   +   lead to much slower execution (e.g. 30x slower total run time on a 40-core
>   +   machine.  */
>   +#define USE_VOLATILE 0
>
> 30x slower could exceed the default max-silent-timeout, which is 1 hour.

Maybe not: building with --max-silent-time=18000 didn’t help…

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Wed, 12 Apr 2017 12:02:01 GMT) Full text and rfc822 format available.

Message #14 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Bruno Haible <bruno <at> clisp.org>
Cc: Mathieu Othacehe <m.othacehe <at> gmail.com>, bug-gnulib <at> gnu.org,
 26441 <at> debbugs.gnu.org
Subject: Re: bug#26441: Gnulib’s ‘test-lock’ fails to complete on machines with >= 32 cores
Date: Wed, 12 Apr 2017 14:00:50 +0200
Hello Gnulib,

The ‘test-lock’ test as shipped with gettext-0.19.8.1 fails to complete
(or hangs?) on machines with 32+ cores, as reported here:

  https://bugs.gnu.org/26441

Does that ring a bell?

Mathieu Othacehe found that this is fixed by gettext commit
1afbcb06fded2a427b761dd1615b1e48e1e853cc but writes:

> However, I don't fully understand the difference between commit
> 1afbcb06fded2a427b761dd1615b1e48e1e853cc in gettext and commit
> 480d374e596a0ee3fed168ab42cd84c313ad3c89 in gnulib.

Could you clarify this?

Thanks in advance,
Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Wed, 12 Apr 2017 14:38:02 GMT) Full text and rfc822 format available.

Message #17 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Mathieu Othacehe <m.othacehe <at> gmail.com>, bug-gnulib <at> gnu.org,
 26441 <at> debbugs.gnu.org
Subject: Re: bug#26441: Gnulib’s ‘test-lock’ fails
 to complete on machines with >= 32 cores
Date: Wed, 12 Apr 2017 16:37:03 +0200
Hi Ludo,

> The ‘test-lock’ test as shipped with gettext-0.19.8.1 fails to complete
> (or hangs?) on machines with 32+ cores, as reported here:
> 
>   https://bugs.gnu.org/26441
> 
> Does that ring a bell?

Yes, this is the bug that was investigated in December 2016 [1] and
subsequently fixed [2].

> Mathieu Othacehe found that this is fixed by gettext commit
> 1afbcb06fded2a427b761dd1615b1e48e1e853cc but writes:
> 
> > However, I don't fully understand the difference between commit
> > 1afbcb06fded2a427b761dd1615b1e48e1e853cc in gettext and commit
> > 480d374e596a0ee3fed168ab42cd84c313ad3c89 in gnulib.

gettext contains two copies of test-lock.c, one from gnulib and
a another one that doesn't use gnulib.
The commit 480d374e596a0ee3fed168ab42cd84c313ad3c89 in gnulib fixed
the first one.
The commit 1afbcb06fded2a427b761dd1615b1e48e1e853cc in gettext fixed
the second one, in the same way.

Bruno

[1] https://lists.gnu.org/archive/html/bug-gnulib/2016-12/msg00113.html
[2] http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=commitdiff;h=480d374e596a0ee3fed168ab42cd84c313ad3c89





Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Wed, 12 Apr 2017 16:12:01 GMT) Full text and rfc822 format available.

Message #20 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Bruno Haible <bruno <at> clisp.org>
Cc: Mathieu Othacehe <m.othacehe <at> gmail.com>, bug-gnulib <at> gnu.org,
 26441 <at> debbugs.gnu.org
Subject: Re: bug#26441: Gnulib’s ‘test-lock’ fails to complete on machines with >= 32 cores
Date: Wed, 12 Apr 2017 18:11:22 +0200
Hi Bruno,

Bruno Haible <bruno <at> clisp.org> skribis:

>> The ‘test-lock’ test as shipped with gettext-0.19.8.1 fails to complete
>> (or hangs?) on machines with 32+ cores, as reported here:
>> 
>>   https://bugs.gnu.org/26441
>> 
>> Does that ring a bell?
>
> Yes, this is the bug that was investigated in December 2016 [1] and
> subsequently fixed [2].
>
>> Mathieu Othacehe found that this is fixed by gettext commit
>> 1afbcb06fded2a427b761dd1615b1e48e1e853cc but writes:
>> 
>> > However, I don't fully understand the difference between commit
>> > 1afbcb06fded2a427b761dd1615b1e48e1e853cc in gettext and commit
>> > 480d374e596a0ee3fed168ab42cd84c313ad3c89 in gnulib.
>
> gettext contains two copies of test-lock.c, one from gnulib and
> a another one that doesn't use gnulib.
> The commit 480d374e596a0ee3fed168ab42cd84c313ad3c89 in gnulib fixed
> the first one.
> The commit 1afbcb06fded2a427b761dd1615b1e48e1e853cc in gettext fixed
> the second one, in the same way.

OK, thanks for the explanation.  I guess we’ll borrow the relevant
patches.

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Thu, 13 Apr 2017 06:40:01 GMT) Full text and rfc822 format available.

Message #23 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>, 26441 <at> debbugs.gnu.org
Subject: Re: bug#26441: Gnulib’s ‘test-lock’ fails to complete on machines with >= 32 cores
Date: Thu, 13 Apr 2017 08:39:55 +0200
[Message part 1 (text/plain, inline)]
Hi,

Here's a patch for gettext, to solve test-lock performance issues.

While rebuilding all gettext dependencies, I noticed that there are
other programs impacted by this bug :

* libunistring
* coreutils
* findutils

But the list may be bigger :

http://git.gnu.org.ua/cgit/gnulib.git/tree/users.txt

Those users are embedding gnulib source code (via git submodules for
example), so I'm not sure what to do here ...

We can consider that it's not such a big deal and ignore this problem
waiting for programs to integrate the fix, or we can patch them on
core-updates.

WDYT ?

Thank you,

Mathieu
[0001-gnu-gettext-Fix-make-check-issues-on-multi-core-mach.patch (text/x-diff, inline)]
From e4ad9aa61fa75afa4417616de36cd25e9631fd67 Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <m.othacehe <at> gmail.com>
Date: Wed, 12 Apr 2017 21:17:24 +0200
Subject: [PATCH] gnu: gettext: Fix make check issues on multi-core machines.

* gnu/packages/patches/gettext-gnulib-multi-core.patch: New file.
* gnu/packages/patches/gettext-multi-core.patch: New file.
* gnu/packages/gettext.scm (gettext-minimal)[patches]: Add a reference
to the two previous patches.
---
 gnu/packages/gettext.scm                           |   9 +-
 .../patches/gettext-gnulib-multi-core.patch        | 176 ++++++++++++++++++++
 gnu/packages/patches/gettext-multi-core.patch      | 183 +++++++++++++++++++++
 3 files changed, 367 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/gettext-gnulib-multi-core.patch
 create mode 100644 gnu/packages/patches/gettext-multi-core.patch

diff --git a/gnu/packages/gettext.scm b/gnu/packages/gettext.scm
index f583d1c2c..2a749e3a6 100644
--- a/gnu/packages/gettext.scm
+++ b/gnu/packages/gettext.scm
@@ -5,6 +5,7 @@
 ;;; Copyright © 2016 Efraim Flashner <efraim <at> flashner.co.il>
 ;;; Copyright © 2016 Alex Kost <alezost <at> gmail.com>
 ;;; Copyright © 2017 Marius Bakke <mbakke <at> fastmail.com>
+;;; Copyright © 2017 Mathieu Othacehe <m.othacehe <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -45,7 +46,13 @@
                                  version ".tar.gz"))
              (sha256
               (base32
-               "0hsw28f9q9xaggjlsdp2qmbp2rbd1mp0njzan2ld9kiqwkq2m57z"))))
+               "0hsw28f9q9xaggjlsdp2qmbp2rbd1mp0njzan2ld9kiqwkq2m57z"))
+             ;; test-lock has performance issues on multi-core machines,
+             ;; it hangs or takes a long time to complete.
+             ;; There is one commit in gettext and one commit
+             ;; in gettext's embedded gnulib to fix this issue.
+             (patches (search-patches "gettext-multi-core.patch"
+                                      "gettext-gnulib-multi-core.patch"))))
     (build-system gnu-build-system)
     (outputs '("out"
                "doc"))                            ;8 MiB of HTML
diff --git a/gnu/packages/patches/gettext-gnulib-multi-core.patch b/gnu/packages/patches/gettext-gnulib-multi-core.patch
new file mode 100644
index 000000000..4393f3a1b
--- /dev/null
+++ b/gnu/packages/patches/gettext-gnulib-multi-core.patch
@@ -0,0 +1,176 @@
+This patch fixes performance problems on multi-core machines.
+See commit 480d374e596a0ee3fed168ab42cd84c313ad3c89 in gnulib
+from Bruno Haible <bruno <at> clisp.org>.
+
+diff --git a/gettext-tools/gnulib-tests/test-lock.c b/gettext-tools/gnulib-tests/test-lock.c
+index cb734b4e6..aa6de2739 100644
+--- a/gettext-tools/gnulib-tests/test-lock.c
++++ b/gettext-tools/gnulib-tests/test-lock.c
+@@ -50,6 +50,13 @@
+    Uncomment this to see if the operating system has a fair scheduler.  */
+ #define EXPLICIT_YIELD 1
+ 
++/* Whether to use 'volatile' on some variables that communicate information
++   between threads.  If set to 0, a lock is used to protect these variables.
++   If set to 1, 'volatile' is used; this is theoretically equivalent but can
++   lead to much slower execution (e.g. 30x slower total run time on a 40-core
++   machine.  */
++#define USE_VOLATILE 0
++
+ /* Whether to print debugging messages.  */
+ #define ENABLE_DEBUGGING 0
+ 
+@@ -103,6 +110,51 @@
+ # define yield()
+ #endif
+ 
++#if USE_VOLATILE
++struct atomic_int {
++  volatile int value;
++};
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++}
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  return ai->value;
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  ai->value = new_value;
++}
++#else
++struct atomic_int {
++  gl_lock_define (, lock)
++  int value;
++};
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++  gl_lock_init (ai->lock);
++}
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  gl_lock_lock (ai->lock);
++  int ret = ai->value;
++  gl_lock_unlock (ai->lock);
++  return ret;
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  gl_lock_lock (ai->lock);
++  ai->value = new_value;
++  gl_lock_unlock (ai->lock);
++}
++#endif
++
+ #define ACCOUNT_COUNT 4
+ 
+ static int account[ACCOUNT_COUNT];
+@@ -170,12 +222,12 @@ lock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int lock_checker_done;
++static struct atomic_int lock_checker_done;
+ 
+ static void *
+ lock_checker_thread (void *arg)
+ {
+-  while (!lock_checker_done)
++  while (get_atomic_int_value (&lock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check lock\n", gl_thread_self_pointer ());
+       gl_lock_lock (my_lock);
+@@ -200,7 +252,8 @@ test_lock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  lock_checker_done = 0;
++  init_atomic_int (&lock_checker_done);
++  set_atomic_int_value (&lock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   checkerthread = gl_thread_create (lock_checker_thread, NULL);
+@@ -210,7 +263,7 @@ test_lock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  lock_checker_done = 1;
++  set_atomic_int_value (&lock_checker_done, 1);
+   gl_thread_join (checkerthread, NULL);
+   check_accounts ();
+ }
+@@ -254,12 +307,12 @@ rwlock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int rwlock_checker_done;
++static struct atomic_int rwlock_checker_done;
+ 
+ static void *
+ rwlock_checker_thread (void *arg)
+ {
+-  while (!rwlock_checker_done)
++  while (get_atomic_int_value (&rwlock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check rdlock\n", gl_thread_self_pointer ());
+       gl_rwlock_rdlock (my_rwlock);
+@@ -284,7 +337,8 @@ test_rwlock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  rwlock_checker_done = 0;
++  init_atomic_int (&rwlock_checker_done);
++  set_atomic_int_value (&rwlock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+@@ -295,7 +349,7 @@ test_rwlock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  rwlock_checker_done = 1;
++  set_atomic_int_value (&rwlock_checker_done, 1);
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (checkerthreads[i], NULL);
+   check_accounts ();
+@@ -356,12 +410,12 @@ reclock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int reclock_checker_done;
++static struct atomic_int reclock_checker_done;
+ 
+ static void *
+ reclock_checker_thread (void *arg)
+ {
+-  while (!reclock_checker_done)
++  while (get_atomic_int_value (&reclock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check lock\n", gl_thread_self_pointer ());
+       gl_recursive_lock_lock (my_reclock);
+@@ -386,7 +440,8 @@ test_recursive_lock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  reclock_checker_done = 0;
++  init_atomic_int (&reclock_checker_done);
++  set_atomic_int_value (&reclock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   checkerthread = gl_thread_create (reclock_checker_thread, NULL);
+@@ -396,7 +451,7 @@ test_recursive_lock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  reclock_checker_done = 1;
++  set_atomic_int_value (&reclock_checker_done, 1);
+   gl_thread_join (checkerthread, NULL);
+   check_accounts ();
+ }
diff --git a/gnu/packages/patches/gettext-multi-core.patch b/gnu/packages/patches/gettext-multi-core.patch
new file mode 100644
index 000000000..3c7f0c759
--- /dev/null
+++ b/gnu/packages/patches/gettext-multi-core.patch
@@ -0,0 +1,183 @@
+This patch fixes performance problems on multi-core machines.
+See commit 1afbcb06fded2a427b761dd1615b1e48e1e853cc in gettext
+from Bruno Haible <bruno <at> clisp.org>.
+
+diff --git a/gettext-runtime/tests/test-lock.c b/gettext-runtime/tests/test-lock.c
+index d279d1d60..51cec3d6b 100644
+--- a/gettext-runtime/tests/test-lock.c
++++ b/gettext-runtime/tests/test-lock.c
+@@ -1,5 +1,5 @@
+ /* Test of locking in multithreaded situations.
+-   Copyright (C) 2005, 2008-2016 Free Software Foundation, Inc.
++   Copyright (C) 2005, 2008-2017 Free Software Foundation, Inc.
+ 
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Lesser General Public License as published by
+@@ -50,6 +50,13 @@
+    Uncomment this to see if the operating system has a fair scheduler.  */
+ #define EXPLICIT_YIELD 1
+ 
++/* Whether to use 'volatile' on some variables that communicate information
++   between threads.  If set to 0, a lock is used to protect these variables.
++   If set to 1, 'volatile' is used; this is theoretically equivalent but can
++   lead to much slower execution (e.g. 30x slower total run time on a 40-core
++   machine.  */
++#define USE_VOLATILE 0
++
+ /* Whether to print debugging messages.  */
+ #define ENABLE_DEBUGGING 0
+ 
+@@ -214,6 +221,51 @@ static inline void * gl_thread_self_pointer (void)
+ # define yield()
+ #endif
+ 
++#if USE_VOLATILE
++struct atomic_int {
++  volatile int value;
++};
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++}
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  return ai->value;
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  ai->value = new_value;
++}
++#else
++struct atomic_int {
++  gl_lock_define (, lock)
++  int value;
++};
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++  gl_lock_init (ai->lock);
++}
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  gl_lock_lock (ai->lock);
++  int ret = ai->value;
++  gl_lock_unlock (ai->lock);
++  return ret;
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  gl_lock_lock (ai->lock);
++  ai->value = new_value;
++  gl_lock_unlock (ai->lock);
++}
++#endif
++
+ #define ACCOUNT_COUNT 4
+ 
+ static int account[ACCOUNT_COUNT];
+@@ -281,12 +333,12 @@ lock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int lock_checker_done;
++static struct atomic_int lock_checker_done;
+ 
+ static void *
+ lock_checker_thread (void *arg)
+ {
+-  while (!lock_checker_done)
++  while (get_atomic_int_value (&lock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check lock\n", gl_thread_self_pointer ());
+       gl_lock_lock (my_lock);
+@@ -311,7 +363,8 @@ test_lock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  lock_checker_done = 0;
++  init_atomic_int (&lock_checker_done);
++  set_atomic_int_value (&lock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   checkerthread = gl_thread_create (lock_checker_thread, NULL);
+@@ -321,7 +374,7 @@ test_lock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  lock_checker_done = 1;
++  set_atomic_int_value (&lock_checker_done, 1);
+   gl_thread_join (checkerthread, NULL);
+   check_accounts ();
+ }
+@@ -365,12 +418,12 @@ rwlock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int rwlock_checker_done;
++static struct atomic_int rwlock_checker_done;
+ 
+ static void *
+ rwlock_checker_thread (void *arg)
+ {
+-  while (!rwlock_checker_done)
++  while (get_atomic_int_value (&rwlock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check rdlock\n", gl_thread_self_pointer ());
+       gl_rwlock_rdlock (my_rwlock);
+@@ -395,7 +448,8 @@ test_rwlock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  rwlock_checker_done = 0;
++  init_atomic_int (&rwlock_checker_done);
++  set_atomic_int_value (&rwlock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+@@ -406,7 +460,7 @@ test_rwlock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  rwlock_checker_done = 1;
++  set_atomic_int_value (&rwlock_checker_done, 1);
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (checkerthreads[i], NULL);
+   check_accounts ();
+@@ -467,12 +521,12 @@ reclock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int reclock_checker_done;
++static struct atomic_int reclock_checker_done;
+ 
+ static void *
+ reclock_checker_thread (void *arg)
+ {
+-  while (!reclock_checker_done)
++  while (get_atomic_int_value (&reclock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check lock\n", gl_thread_self_pointer ());
+       gl_recursive_lock_lock (my_reclock);
+@@ -497,7 +551,8 @@ test_recursive_lock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  reclock_checker_done = 0;
++  init_atomic_int (&reclock_checker_done);
++  set_atomic_int_value (&reclock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   checkerthread = gl_thread_create (reclock_checker_thread, NULL);
+@@ -507,7 +562,7 @@ test_recursive_lock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  reclock_checker_done = 1;
++  set_atomic_int_value (&reclock_checker_done, 1);
+   gl_thread_join (checkerthread, NULL);
+   check_accounts ();
+ }
-- 
2.11.1


Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Thu, 13 Apr 2017 12:25:01 GMT) Full text and rfc822 format available.

Message #26 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Mathieu Othacehe <m.othacehe <at> gmail.com>
Cc: 26441 <at> debbugs.gnu.org
Subject: Re: bug#26441: Gnulib’s ‘test-lock’ fails to complete on machines with >= 32 cores
Date: Thu, 13 Apr 2017 14:23:56 +0200
Hi Mathieu,

Mathieu Othacehe <m.othacehe <at> gmail.com> skribis:

> Here's a patch for gettext, to solve test-lock performance issues.

Awesome.

> While rebuilding all gettext dependencies, I noticed that there are
> other programs impacted by this bug :
>
> * libunistring
> * coreutils
> * findutils

Coreutils 8.27 was released very recently so it probably already has the
‘test-lock’ fix.  But yeah, there are many other GNU packages that
possibly include it, although not all of them depend on this specific
Gnulib module.

> We can consider that it's not such a big deal and ignore this problem
> waiting for programs to integrate the fix, or we can patch them on
> core-updates.
>
> WDYT ?

Maybe we can simply patch the important packages (findutils,
libunistring) and wait and see if others need the patch.  Not ideal, but
I can’t think of a better way.

> From e4ad9aa61fa75afa4417616de36cd25e9631fd67 Mon Sep 17 00:00:00 2001
> From: Mathieu Othacehe <m.othacehe <at> gmail.com>
> Date: Wed, 12 Apr 2017 21:17:24 +0200
> Subject: [PATCH] gnu: gettext: Fix make check issues on multi-core machines.
>
> * gnu/packages/patches/gettext-gnulib-multi-core.patch: New file.
> * gnu/packages/patches/gettext-multi-core.patch: New file.
> * gnu/packages/gettext.scm (gettext-minimal)[patches]: Add a reference
> to the two previous patches.

[...]

> --- /dev/null
> +++ b/gnu/packages/patches/gettext-gnulib-multi-core.patch
> @@ -0,0 +1,176 @@
> +This patch fixes performance problems on multi-core machines.
> +See commit 480d374e596a0ee3fed168ab42cd84c313ad3c89 in gnulib
> +from Bruno Haible <bruno <at> clisp.org>.

I added the reference to <https://bugs.gnu.org/26441> and pushed.

Thank you!

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Thu, 13 Apr 2017 16:32:02 GMT) Full text and rfc822 format available.

Message #29 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 26441 <at> debbugs.gnu.org
Subject: Re: bug#26441: Gnulib’s ‘test-lock’ fails to complete on machines with >= 32 cores
Date: Thu, 13 Apr 2017 18:31:20 +0200
> Maybe we can simply patch the important packages (findutils,
> libunistring) and wait and see if others need the patch.  Not ideal, but
> I can’t think of a better way.

Ok fine for me.

I'll send findutils and libunistring patches for now.

Thanks,

Mathieu




Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Thu, 13 Apr 2017 16:35:02 GMT) Full text and rfc822 format available.

Message #32 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: 26441 <at> debbugs.gnu.org
Cc: Mathieu Othacehe <m.othacehe <at> gmail.com>
Subject: [PATCH] gnu: libunistring: Fix make check issues on multi-core
 machines.
Date: Thu, 13 Apr 2017 18:33:40 +0200
* gnu/packages/patches/libunistring-gnulib-multi-core.patch: New file.
* gnu/packages/libunistring.scm (libunistring)[patches]: Add a reference
to previous patch.
---
 gnu/packages/libunistring.scm                      |   8 +-
 .../patches/libunistring-gnulib-multi-core.patch   | 178 +++++++++++++++++++++
 2 files changed, 185 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/libunistring-gnulib-multi-core.patch

diff --git a/gnu/packages/libunistring.scm b/gnu/packages/libunistring.scm
index 212bec4b4..df02f68ce 100644
--- a/gnu/packages/libunistring.scm
+++ b/gnu/packages/libunistring.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2015 Mark H Weaver <mhw <at> netris.org>
 ;;; Copyright © 2016 Efraim Flashner <efraim <at> flashner.co.il>
 ;;; Copyright © 2016 Jan Nieuwenhuizen <janneke <at> gnu.org>
+;;; Copyright © 2017 Mathieu Othacehe <m.othacehe <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -24,6 +25,7 @@
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix build-system gnu)
+  #:use-module (gnu packages)
   #:use-module (gnu packages base))
 
 (define-public libunistring
@@ -37,7 +39,11 @@
                   version ".tar.xz"))
             (sha256
              (base32
-              "15z76qrmrvkc3c6hfq2lzzqysgd21s682f2smycfab5g598n8drf"))))
+              "15z76qrmrvkc3c6hfq2lzzqysgd21s682f2smycfab5g598n8drf"))
+             ;; test-lock has performance issues on multi-core machines,
+             ;; it hangs or takes a long time to complete.
+             ;; This is a commit from gnulib to fix this issue.
+            (patches (search-patches "libunistring-gnulib-multi-core.patch"))))
    (propagated-inputs (libiconv-if-needed))
    (build-system gnu-build-system)
    (arguments
diff --git a/gnu/packages/patches/libunistring-gnulib-multi-core.patch b/gnu/packages/patches/libunistring-gnulib-multi-core.patch
new file mode 100644
index 000000000..709b20c6d
--- /dev/null
+++ b/gnu/packages/patches/libunistring-gnulib-multi-core.patch
@@ -0,0 +1,178 @@
+This patch fixes performance problems on multi-core machines
+as reported at <https://bugs.gnu.org/26441>.
+
+See commit 480d374e596a0ee3fed168ab42cd84c313ad3c89 in Gnulib
+by Bruno Haible <bruno <at> clisp.org>.
+
+diff --git a/tests/test-lock.c b/tests/test-lock.c
+index cb734b4e6..aa6de2739 100644
+--- a/tests/test-lock.c
++++ b/tests/test-lock.c
+@@ -50,6 +50,13 @@
+    Uncomment this to see if the operating system has a fair scheduler.  */
+ #define EXPLICIT_YIELD 1
+ 
++/* Whether to use 'volatile' on some variables that communicate information
++   between threads.  If set to 0, a lock is used to protect these variables.
++   If set to 1, 'volatile' is used; this is theoretically equivalent but can
++   lead to much slower execution (e.g. 30x slower total run time on a 40-core
++   machine.  */
++#define USE_VOLATILE 0
++
+ /* Whether to print debugging messages.  */
+ #define ENABLE_DEBUGGING 0
+ 
+@@ -103,6 +110,51 @@
+ # define yield()
+ #endif
+ 
++#if USE_VOLATILE
++struct atomic_int {
++  volatile int value;
++};
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++}
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  return ai->value;
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  ai->value = new_value;
++}
++#else
++struct atomic_int {
++  gl_lock_define (, lock)
++  int value;
++};
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++  gl_lock_init (ai->lock);
++}
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  gl_lock_lock (ai->lock);
++  int ret = ai->value;
++  gl_lock_unlock (ai->lock);
++  return ret;
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  gl_lock_lock (ai->lock);
++  ai->value = new_value;
++  gl_lock_unlock (ai->lock);
++}
++#endif
++
+ #define ACCOUNT_COUNT 4
+ 
+ static int account[ACCOUNT_COUNT];
+@@ -170,12 +222,12 @@ lock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int lock_checker_done;
++static struct atomic_int lock_checker_done;
+ 
+ static void *
+ lock_checker_thread (void *arg)
+ {
+-  while (!lock_checker_done)
++  while (get_atomic_int_value (&lock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check lock\n", gl_thread_self_pointer ());
+       gl_lock_lock (my_lock);
+@@ -200,7 +252,8 @@ test_lock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  lock_checker_done = 0;
++  init_atomic_int (&lock_checker_done);
++  set_atomic_int_value (&lock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   checkerthread = gl_thread_create (lock_checker_thread, NULL);
+@@ -210,7 +263,7 @@ test_lock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  lock_checker_done = 1;
++  set_atomic_int_value (&lock_checker_done, 1);
+   gl_thread_join (checkerthread, NULL);
+   check_accounts ();
+ }
+@@ -254,12 +307,12 @@ rwlock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int rwlock_checker_done;
++static struct atomic_int rwlock_checker_done;
+ 
+ static void *
+ rwlock_checker_thread (void *arg)
+ {
+-  while (!rwlock_checker_done)
++  while (get_atomic_int_value (&rwlock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check rdlock\n", gl_thread_self_pointer ());
+       gl_rwlock_rdlock (my_rwlock);
+@@ -284,7 +337,8 @@ test_rwlock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  rwlock_checker_done = 0;
++  init_atomic_int (&rwlock_checker_done);
++  set_atomic_int_value (&rwlock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+@@ -295,7 +349,7 @@ test_rwlock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  rwlock_checker_done = 1;
++  set_atomic_int_value (&rwlock_checker_done, 1);
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (checkerthreads[i], NULL);
+   check_accounts ();
+@@ -356,12 +410,12 @@ reclock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int reclock_checker_done;
++static struct atomic_int reclock_checker_done;
+ 
+ static void *
+ reclock_checker_thread (void *arg)
+ {
+-  while (!reclock_checker_done)
++  while (get_atomic_int_value (&reclock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check lock\n", gl_thread_self_pointer ());
+       gl_recursive_lock_lock (my_reclock);
+@@ -386,7 +440,8 @@ test_recursive_lock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  reclock_checker_done = 0;
++  init_atomic_int (&reclock_checker_done);
++  set_atomic_int_value (&reclock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   checkerthread = gl_thread_create (reclock_checker_thread, NULL);
+@@ -396,7 +451,7 @@ test_recursive_lock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  reclock_checker_done = 1;
++  set_atomic_int_value (&reclock_checker_done, 1);
+   gl_thread_join (checkerthread, NULL);
+   check_accounts ();
+ }
-- 
2.12.2





Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Thu, 13 Apr 2017 16:35:02 GMT) Full text and rfc822 format available.

Message #35 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: 26441 <at> debbugs.gnu.org
Cc: Mathieu Othacehe <m.othacehe <at> gmail.com>
Subject: [PATCH] gnu: findutils: Fix make check issues on multi-core machines.
Date: Thu, 13 Apr 2017 18:34:08 +0200
* gnu/packages/patches/findutils-gnulib-multi-core.patch: New file.
* gnu/packages/base.scm (findutils)[patches]: Add a reference
to the previous patch.
---
 gnu/packages/base.scm                              |  60 +++--
 .../patches/findutils-gnulib-multi-core.patch      | 294 +++++++++++++++++++++
 2 files changed, 327 insertions(+), 27 deletions(-)
 create mode 100644 gnu/packages/patches/findutils-gnulib-multi-core.patch

diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index 7bcfd7a2e..e620d9cea 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2016 Efraim Flashner <efraim <at> flashner.co.il>
 ;;; Copyright © 2016 Jan Nieuwenhuizen <janneke <at> gnu.org>
 ;;; Copyright © 2017 Rene Saavedra <rennes <at> openmailbox.org>
+;;; Copyright © 2017 Mathieu Othacehe <m.othacehe <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -250,38 +251,43 @@ interactive means to merge two files.")
 
 (define-public findutils
   (package
-   (name "findutils")
-   (version "4.6.0")
-   (source (origin
-            (method url-fetch)
-            (uri (string-append "mirror://gnu/findutils/findutils-"
-                                version ".tar.gz"))
-            (sha256
-             (base32
-              "178nn4dl7wbcw499czikirnkniwnx36argdnqgz4ik9i6zvwkm6y"))
-            (patches (search-patches "findutils-localstatedir.patch"
-                                     "findutils-test-xargs.patch"))))
-   (build-system gnu-build-system)
-   (arguments
-    `(#:configure-flags (list
-                         ;; Tell 'updatedb' to write to /var.
-                         "--localstatedir=/var"
-
-                         ;; Work around cross-compilation failure.  See
-                         ;; <http://savannah.gnu.org/bugs/?27299#comment1>.
-                         ,@(if (%current-target-system)
-                               '("gl_cv_func_wcwidth_works=yes")
-                               '()))))
-   (synopsis "Operating on files matching given criteria")
-   (description
-    "Findutils supplies the basic file directory searching utilities of the
+    (name "findutils")
+    (version "4.6.0")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "mirror://gnu/findutils/findutils-"
+                                  version ".tar.gz"))
+              (sha256
+               (base32
+                "178nn4dl7wbcw499czikirnkniwnx36argdnqgz4ik9i6zvwkm6y"))
+              (patches (search-patches
+                        "findutils-localstatedir.patch"
+                        "findutils-test-xargs.patch"
+                        ;; test-lock has performance issues on multi-core
+                        ;; machines, it hangs or takes a long time to complete.
+                        ;; This is a commit from gnulib to fix this issue.
+                        "findutils-gnulib-multi-core.patch"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:configure-flags (list
+                          ;; Tell 'updatedb' to write to /var.
+                          "--localstatedir=/var"
+
+                          ;; Work around cross-compilation failure.  See
+                          ;; <http://savannah.gnu.org/bugs/?27299#comment1>.
+                          ,@(if (%current-target-system)
+                                '("gl_cv_func_wcwidth_works=yes")
+                                '()))))
+    (synopsis "Operating on files matching given criteria")
+    (description
+     "Findutils supplies the basic file directory searching utilities of the
 GNU system.  It consists of two primary searching utilities: \"find\"
 recursively searches for files in a directory according to given criteria and
 \"locate\" lists files in a database that match a query.  Two auxiliary tools
 are included: \"updatedb\" updates the file name database and \"xargs\" may be
 used to apply commands with arbitrarily long arguments.")
-   (license gpl3+)
-   (home-page "https://www.gnu.org/software/findutils/")))
+    (license gpl3+)
+    (home-page "https://www.gnu.org/software/findutils/")))
 
 (define-public coreutils
   (package
diff --git a/gnu/packages/patches/findutils-gnulib-multi-core.patch b/gnu/packages/patches/findutils-gnulib-multi-core.patch
new file mode 100644
index 000000000..5a37f4f1f
--- /dev/null
+++ b/gnu/packages/patches/findutils-gnulib-multi-core.patch
@@ -0,0 +1,294 @@
+This patch fixes performance problems on multi-core machines
+as reported at <https://bugs.gnu.org/26441>.
+
+See commit 480d374e596a0ee3fed168ab42cd84c313ad3c89 in Gnulib
+by Bruno Haible <bruno <at> clisp.org>.
+
+diff --git a/tests/test-lock.c b/tests/test-lock.c
+index a992f64..fb18dee 100644
+--- a/tests/test-lock.c
++++ b/tests/test-lock.c
+@@ -1,5 +1,5 @@
+ /* Test of locking in multithreaded situations.
+-   Copyright (C) 2005, 2008-2015 Free Software Foundation, Inc.
++   Copyright (C) 2005, 2008-2017 Free Software Foundation, Inc.
+ 
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+@@ -50,6 +50,28 @@
+    Uncomment this to see if the operating system has a fair scheduler.  */
+ #define EXPLICIT_YIELD 1
+ 
++/* Whether to use 'volatile' on some variables that communicate information
++   between threads.  If set to 0, a semaphore or a lock is used to protect
++   these variables.  If set to 1, 'volatile' is used; this is theoretically
++   equivalent but can lead to much slower execution (e.g. 30x slower total
++   run time on a 40-core machine), because 'volatile' does not imply any
++   synchronization/communication between different CPUs.  */
++#define USE_VOLATILE 0
++
++#if USE_POSIX_THREADS && HAVE_SEMAPHORE_H
++/* Whether to use a semaphore to communicate information between threads.
++   If set to 0, a lock is used. If set to 1, a semaphore is used.
++   Uncomment this to reduce the dependencies of this test.  */
++# define USE_SEMAPHORE 1
++/* Mac OS X provides only named semaphores (sem_open); its facility for
++   unnamed semaphores (sem_init) does not work.  */
++# if defined __APPLE__ && defined __MACH__
++#  define USE_NAMED_SEMAPHORE 1
++# else
++#  define USE_UNNAMED_SEMAPHORE 1
++# endif
++#endif
++
+ /* Whether to print debugging messages.  */
+ #define ENABLE_DEBUGGING 0
+ 
+@@ -90,6 +112,12 @@
+ 
+ #include "glthread/thread.h"
+ #include "glthread/yield.h"
++#if USE_SEMAPHORE
++# include <errno.h>
++# include <fcntl.h>
++# include <semaphore.h>
++# include <unistd.h>
++#endif
+ 
+ #if ENABLE_DEBUGGING
+ # define dbgprintf printf
+@@ -103,6 +131,132 @@
+ # define yield()
+ #endif
+ 
++#if USE_VOLATILE
++struct atomic_int {
++  volatile int value;
++};
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++}
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  return ai->value;
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  ai->value = new_value;
++}
++#elif USE_SEMAPHORE
++/* This atomic_int implementation can only support the values 0 and 1.
++   It is initially 0 and can be set to 1 only once.  */
++# if USE_UNNAMED_SEMAPHORE
++struct atomic_int {
++  sem_t semaphore;
++};
++#define atomic_int_semaphore(ai) (&(ai)->semaphore)
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++  sem_init (&ai->semaphore, 0, 0);
++}
++# endif
++# if USE_NAMED_SEMAPHORE
++struct atomic_int {
++  sem_t *semaphore;
++};
++#define atomic_int_semaphore(ai) ((ai)->semaphore)
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++  sem_t *s;
++  unsigned int count;
++  for (count = 0; ; count++)
++    {
++      char name[80];
++      /* Use getpid() in the name, so that different processes running at the
++         same time will not interfere.  Use ai in the name, so that different
++         atomic_int in the same process will not interfere.  Use a count in
++         the name, so that even in the (unlikely) case that a semaphore with
++         the specified name already exists, we can try a different name.  */
++      sprintf (name, "test-lock-%lu-%p-%u",
++               (unsigned long) getpid (), ai, count);
++      s = sem_open (name, O_CREAT | O_EXCL, 0600, 0);
++      if (s == SEM_FAILED)
++        {
++          if (errno == EEXIST)
++            /* Retry with a different name.  */
++            continue;
++          else
++            {
++              perror ("sem_open failed");
++              abort ();
++            }
++        }
++      else
++        {
++          /* Try not to leave a semaphore hanging around on the file system
++             eternally, if we can avoid it.  */
++          sem_unlink (name);
++          break;
++        }
++    }
++  ai->semaphore = s;
++}
++# endif
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  if (sem_trywait (atomic_int_semaphore (ai)) == 0)
++    {
++      if (sem_post (atomic_int_semaphore (ai)))
++        abort ();
++      return 1;
++    }
++  else if (errno == EAGAIN)
++    return 0;
++  else
++    abort ();
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  if (new_value == 0)
++    /* It's already initialized with 0.  */
++    return;
++  /* To set the value 1: */
++  if (sem_post (atomic_int_semaphore (ai)))
++    abort ();
++}
++#else
++struct atomic_int {
++  gl_lock_define (, lock)
++  int value;
++};
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++  gl_lock_init (ai->lock);
++}
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  gl_lock_lock (ai->lock);
++  int ret = ai->value;
++  gl_lock_unlock (ai->lock);
++  return ret;
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  gl_lock_lock (ai->lock);
++  ai->value = new_value;
++  gl_lock_unlock (ai->lock);
++}
++#endif
++
+ #define ACCOUNT_COUNT 4
+ 
+ static int account[ACCOUNT_COUNT];
+@@ -170,12 +324,12 @@ lock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int lock_checker_done;
++static struct atomic_int lock_checker_done;
+ 
+ static void *
+ lock_checker_thread (void *arg)
+ {
+-  while (!lock_checker_done)
++  while (get_atomic_int_value (&lock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check lock\n", gl_thread_self_pointer ());
+       gl_lock_lock (my_lock);
+@@ -200,7 +354,8 @@ test_lock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  lock_checker_done = 0;
++  init_atomic_int (&lock_checker_done);
++  set_atomic_int_value (&lock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   checkerthread = gl_thread_create (lock_checker_thread, NULL);
+@@ -210,7 +365,7 @@ test_lock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  lock_checker_done = 1;
++  set_atomic_int_value (&lock_checker_done, 1);
+   gl_thread_join (checkerthread, NULL);
+   check_accounts ();
+ }
+@@ -254,12 +409,12 @@ rwlock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int rwlock_checker_done;
++static struct atomic_int rwlock_checker_done;
+ 
+ static void *
+ rwlock_checker_thread (void *arg)
+ {
+-  while (!rwlock_checker_done)
++  while (get_atomic_int_value (&rwlock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check rdlock\n", gl_thread_self_pointer ());
+       gl_rwlock_rdlock (my_rwlock);
+@@ -284,7 +439,8 @@ test_rwlock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  rwlock_checker_done = 0;
++  init_atomic_int (&rwlock_checker_done);
++  set_atomic_int_value (&rwlock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+@@ -295,7 +451,7 @@ test_rwlock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  rwlock_checker_done = 1;
++  set_atomic_int_value (&rwlock_checker_done, 1);
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (checkerthreads[i], NULL);
+   check_accounts ();
+@@ -356,12 +512,12 @@ reclock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int reclock_checker_done;
++static struct atomic_int reclock_checker_done;
+ 
+ static void *
+ reclock_checker_thread (void *arg)
+ {
+-  while (!reclock_checker_done)
++  while (get_atomic_int_value (&reclock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check lock\n", gl_thread_self_pointer ());
+       gl_recursive_lock_lock (my_reclock);
+@@ -386,7 +542,8 @@ test_recursive_lock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  reclock_checker_done = 0;
++  init_atomic_int (&reclock_checker_done);
++  set_atomic_int_value (&reclock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   checkerthread = gl_thread_create (reclock_checker_thread, NULL);
+@@ -396,7 +553,7 @@ test_recursive_lock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  reclock_checker_done = 1;
++  set_atomic_int_value (&reclock_checker_done, 1);
+   gl_thread_join (checkerthread, NULL);
+   check_accounts ();
+ }
-- 
2.11.1





Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Thu, 13 Apr 2017 19:39:01 GMT) Full text and rfc822 format available.

Message #38 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Mathieu Othacehe <m.othacehe <at> gmail.com>, 26441 <at> debbugs.gnu.org
Subject: Re: bug#26441: [PATCH] gnu: findutils: Fix make check issues
 on	multi-core machines.
Date: Thu, 13 Apr 2017 21:38:46 +0200
[Message part 1 (text/plain, inline)]
Hi, thanks for working on these!

[...]
  
>  (define-public findutils
>    (package
> -   (name "findutils")
> -   (version "4.6.0")
> -   (source (origin
> -            (method url-fetch)
> -            (uri (string-append "mirror://gnu/findutils/findutils-"
> -                                version ".tar.gz"))
> -            (sha256
> -             (base32
> -              "178nn4dl7wbcw499czikirnkniwnx36argdnqgz4ik9i6zvwkm6y"))
> -            (patches (search-patches "findutils-localstatedir.patch"
> -                                     "findutils-test-xargs.patch"))))
> -   (build-system gnu-build-system)
> -   (arguments
> -    `(#:configure-flags (list
> -                         ;; Tell 'updatedb' to write to /var.
> -                         "--localstatedir=/var"
> -
> -                         ;; Work around cross-compilation failure.  See
> -                         ;; <http://savannah.gnu.org/bugs/?27299#comment1>.
> -                         ,@(if (%current-target-system)
> -                               '("gl_cv_func_wcwidth_works=yes")
> -                               '()))))
> -   (synopsis "Operating on files matching given criteria")
> -   (description
> -    "Findutils supplies the basic file directory searching utilities of the
> +    (name "findutils")
> +    (version "4.6.0")
> +    (source (origin
> +              (method url-fetch)
> +              (uri (string-append "mirror://gnu/findutils/findutils-"
> +                                  version ".tar.gz"))
> +              (sha256
> +               (base32
> +                "178nn4dl7wbcw499czikirnkniwnx36argdnqgz4ik9i6zvwkm6y"))
> +              (patches (search-patches
> +                        "findutils-localstatedir.patch"
> +                        "findutils-test-xargs.patch"
> +                        ;; test-lock has performance issues on multi-core
> +                        ;; machines, it hangs or takes a long time to complete.
> +                        ;; This is a commit from gnulib to fix this issue.
> +                        "findutils-gnulib-multi-core.patch"))))

Please don't mix large indentation changes with functional changes. It
is really difficult to tell what this commit does. Can you split the
indentation change out in a separate commit?
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Thu, 13 Apr 2017 20:05:02 GMT) Full text and rfc822 format available.

Message #41 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: 26441 <at> debbugs.gnu.org
Subject: Re: bug#26441: [PATCH] gnu: findutils: Fix make check issues on
 multi-core machines.
Date: Thu, 13 Apr 2017 22:04:20 +0200
Hi Marius,

> Please don't mix large indentation changes with functional changes. It
> is really difficult to tell what this commit does. Can you split the
> indentation change out in a separate commit?

Sure, I'll send new patches.

Mathieu




Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Thu, 13 Apr 2017 20:07:02 GMT) Full text and rfc822 format available.

Message #44 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: 26441 <at> debbugs.gnu.org
Cc: Mathieu Othacehe <m.othacehe <at> gmail.com>
Subject: [PATCH 1/2] gnu: findutils: Fix make check issues on multi-core
 machines.
Date: Thu, 13 Apr 2017 22:06:35 +0200
* gnu/packages/patches/findutils-gnulib-multi-core.patch: New file.
* gnu/packages/base.scm (findutils)[patches]: Add a reference
to the previous patch.
---
 gnu/packages/base.scm | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index 7bcfd7a2e..1ae3c8911 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2016 Efraim Flashner <efraim <at> flashner.co.il>
 ;;; Copyright © 2016 Jan Nieuwenhuizen <janneke <at> gnu.org>
 ;;; Copyright © 2017 Rene Saavedra <rennes <at> openmailbox.org>
+;;; Copyright © 2017 Mathieu Othacehe <m.othacehe <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -259,8 +260,13 @@ interactive means to merge two files.")
             (sha256
              (base32
               "178nn4dl7wbcw499czikirnkniwnx36argdnqgz4ik9i6zvwkm6y"))
-            (patches (search-patches "findutils-localstatedir.patch"
-                                     "findutils-test-xargs.patch"))))
+            (patches (search-patches
+                      "findutils-localstatedir.patch"
+                      "findutils-test-xargs.patch"
+                      ;; test-lock has performance issues on multi-core
+                      ;; machines, it hangs or takes a long time to complete.
+                      ;; This is a commit from gnulib to fix this issue.
+                      "findutils-gnulib-multi-core.patch"))))
    (build-system gnu-build-system)
    (arguments
     `(#:configure-flags (list
-- 
2.12.2





Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Thu, 13 Apr 2017 20:08:01 GMT) Full text and rfc822 format available.

Message #47 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: 26441 <at> debbugs.gnu.org
Cc: Mathieu Othacehe <m.othacehe <at> gmail.com>
Subject: [PATCH 2/2] gnu: findutils: Reindent package definition.
Date: Thu, 13 Apr 2017 22:06:36 +0200
* gnu/packages/base.scm (findutils): Reindent, no functional change.
---
 gnu/packages/base.scm | 64 +++++++++++++++++++++++++--------------------------
 1 file changed, 32 insertions(+), 32 deletions(-)

diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index 1ae3c8911..e620d9cea 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -251,43 +251,43 @@ interactive means to merge two files.")
 
 (define-public findutils
   (package
-   (name "findutils")
-   (version "4.6.0")
-   (source (origin
-            (method url-fetch)
-            (uri (string-append "mirror://gnu/findutils/findutils-"
-                                version ".tar.gz"))
-            (sha256
-             (base32
-              "178nn4dl7wbcw499czikirnkniwnx36argdnqgz4ik9i6zvwkm6y"))
-            (patches (search-patches
-                      "findutils-localstatedir.patch"
-                      "findutils-test-xargs.patch"
-                      ;; test-lock has performance issues on multi-core
-                      ;; machines, it hangs or takes a long time to complete.
-                      ;; This is a commit from gnulib to fix this issue.
-                      "findutils-gnulib-multi-core.patch"))))
-   (build-system gnu-build-system)
-   (arguments
-    `(#:configure-flags (list
-                         ;; Tell 'updatedb' to write to /var.
-                         "--localstatedir=/var"
-
-                         ;; Work around cross-compilation failure.  See
-                         ;; <http://savannah.gnu.org/bugs/?27299#comment1>.
-                         ,@(if (%current-target-system)
-                               '("gl_cv_func_wcwidth_works=yes")
-                               '()))))
-   (synopsis "Operating on files matching given criteria")
-   (description
-    "Findutils supplies the basic file directory searching utilities of the
+    (name "findutils")
+    (version "4.6.0")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "mirror://gnu/findutils/findutils-"
+                                  version ".tar.gz"))
+              (sha256
+               (base32
+                "178nn4dl7wbcw499czikirnkniwnx36argdnqgz4ik9i6zvwkm6y"))
+              (patches (search-patches
+                        "findutils-localstatedir.patch"
+                        "findutils-test-xargs.patch"
+                        ;; test-lock has performance issues on multi-core
+                        ;; machines, it hangs or takes a long time to complete.
+                        ;; This is a commit from gnulib to fix this issue.
+                        "findutils-gnulib-multi-core.patch"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:configure-flags (list
+                          ;; Tell 'updatedb' to write to /var.
+                          "--localstatedir=/var"
+
+                          ;; Work around cross-compilation failure.  See
+                          ;; <http://savannah.gnu.org/bugs/?27299#comment1>.
+                          ,@(if (%current-target-system)
+                                '("gl_cv_func_wcwidth_works=yes")
+                                '()))))
+    (synopsis "Operating on files matching given criteria")
+    (description
+     "Findutils supplies the basic file directory searching utilities of the
 GNU system.  It consists of two primary searching utilities: \"find\"
 recursively searches for files in a directory according to given criteria and
 \"locate\" lists files in a database that match a query.  Two auxiliary tools
 are included: \"updatedb\" updates the file name database and \"xargs\" may be
 used to apply commands with arbitrarily long arguments.")
-   (license gpl3+)
-   (home-page "https://www.gnu.org/software/findutils/")))
+    (license gpl3+)
+    (home-page "https://www.gnu.org/software/findutils/")))
 
 (define-public coreutils
   (package
-- 
2.12.2





Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Sat, 15 Apr 2017 18:12:02 GMT) Full text and rfc822 format available.

Message #50 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Mathieu Othacehe <m.othacehe <at> gmail.com>, 26441 <at> debbugs.gnu.org
Subject: Re: bug#26441: [PATCH 1/2] gnu: findutils: Fix make check issues
 on	multi-core machines.
Date: Sat, 15 Apr 2017 20:11:12 +0200
[Message part 1 (text/plain, inline)]
Mathieu Othacehe <m.othacehe <at> gmail.com> writes:

> * gnu/packages/patches/findutils-gnulib-multi-core.patch: New file.
> * gnu/packages/base.scm (findutils)[patches]: Add a reference
> to the previous patch.
> ---
>  gnu/packages/base.scm | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)

It looks like you forgot the actual patch :)

Please also add it to gnu/local.mk. TIA!

[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Mon, 17 Apr 2017 10:19:02 GMT) Full text and rfc822 format available.

Message #53 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: 26441 <at> debbugs.gnu.org
Subject: Re: bug#26441: [PATCH 1/2] gnu: findutils: Fix make check issues on
 multi-core machines.
Date: Mon, 17 Apr 2017 12:18:02 +0200
Hi Marius !

> It looks like you forgot the actual patch :)

Ha ! You're right :)

>
> Please also add it to gnu/local.mk. TIA!

Ok, I'll a new batch, to be applied to core-updates.

Thanks,

Mathieu




Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Mon, 17 Apr 2017 10:19:02 GMT) Full text and rfc822 format available.

Message #56 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: 26441 <at> debbugs.gnu.org
Cc: Mathieu Othacehe <m.othacehe <at> gmail.com>
Subject: [PATCH 3/3] gnu: libunistring: Fix make check issues on multi-core
 machines.
Date: Mon, 17 Apr 2017 12:18:29 +0200
* gnu/packages/patches/libunistring-gnulib-multi-core.patch: New file.
* gnu/local.mk (dist_patch): Add previous patch.
* gnu/packages/libunistring.scm (libunistring)[patches]: Add a reference
to previous patch.
---
 gnu/local.mk                                       |   1 +
 gnu/packages/libunistring.scm                      |   8 +-
 .../patches/libunistring-gnulib-multi-core.patch   | 178 +++++++++++++++++++++
 3 files changed, 186 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/libunistring-gnulib-multi-core.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index b1dbfec5e..a0d7cfd0a 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -732,6 +732,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/libtorrent-rasterbar-boost-compat.patch	\
   %D%/packages/patches/libtool-skip-tests2.patch		\
   %D%/packages/patches/libunwind-CVE-2015-3239.patch		\
+  %D%/packages/patches/libunistring-gnulib-multi-core.patch	\
   %D%/packages/patches/libvpx-CVE-2016-2818.patch		\
   %D%/packages/patches/libwmf-CAN-2004-0941.patch		\
   %D%/packages/patches/libwmf-CVE-2006-3376.patch		\
diff --git a/gnu/packages/libunistring.scm b/gnu/packages/libunistring.scm
index 212bec4b4..df02f68ce 100644
--- a/gnu/packages/libunistring.scm
+++ b/gnu/packages/libunistring.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2015 Mark H Weaver <mhw <at> netris.org>
 ;;; Copyright © 2016 Efraim Flashner <efraim <at> flashner.co.il>
 ;;; Copyright © 2016 Jan Nieuwenhuizen <janneke <at> gnu.org>
+;;; Copyright © 2017 Mathieu Othacehe <m.othacehe <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -24,6 +25,7 @@
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix build-system gnu)
+  #:use-module (gnu packages)
   #:use-module (gnu packages base))
 
 (define-public libunistring
@@ -37,7 +39,11 @@
                   version ".tar.xz"))
             (sha256
              (base32
-              "15z76qrmrvkc3c6hfq2lzzqysgd21s682f2smycfab5g598n8drf"))))
+              "15z76qrmrvkc3c6hfq2lzzqysgd21s682f2smycfab5g598n8drf"))
+             ;; test-lock has performance issues on multi-core machines,
+             ;; it hangs or takes a long time to complete.
+             ;; This is a commit from gnulib to fix this issue.
+            (patches (search-patches "libunistring-gnulib-multi-core.patch"))))
    (propagated-inputs (libiconv-if-needed))
    (build-system gnu-build-system)
    (arguments
diff --git a/gnu/packages/patches/libunistring-gnulib-multi-core.patch b/gnu/packages/patches/libunistring-gnulib-multi-core.patch
new file mode 100644
index 000000000..709b20c6d
--- /dev/null
+++ b/gnu/packages/patches/libunistring-gnulib-multi-core.patch
@@ -0,0 +1,178 @@
+This patch fixes performance problems on multi-core machines
+as reported at <https://bugs.gnu.org/26441>.
+
+See commit 480d374e596a0ee3fed168ab42cd84c313ad3c89 in Gnulib
+by Bruno Haible <bruno <at> clisp.org>.
+
+diff --git a/tests/test-lock.c b/tests/test-lock.c
+index cb734b4e6..aa6de2739 100644
+--- a/tests/test-lock.c
++++ b/tests/test-lock.c
+@@ -50,6 +50,13 @@
+    Uncomment this to see if the operating system has a fair scheduler.  */
+ #define EXPLICIT_YIELD 1
+ 
++/* Whether to use 'volatile' on some variables that communicate information
++   between threads.  If set to 0, a lock is used to protect these variables.
++   If set to 1, 'volatile' is used; this is theoretically equivalent but can
++   lead to much slower execution (e.g. 30x slower total run time on a 40-core
++   machine.  */
++#define USE_VOLATILE 0
++
+ /* Whether to print debugging messages.  */
+ #define ENABLE_DEBUGGING 0
+ 
+@@ -103,6 +110,51 @@
+ # define yield()
+ #endif
+ 
++#if USE_VOLATILE
++struct atomic_int {
++  volatile int value;
++};
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++}
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  return ai->value;
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  ai->value = new_value;
++}
++#else
++struct atomic_int {
++  gl_lock_define (, lock)
++  int value;
++};
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++  gl_lock_init (ai->lock);
++}
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  gl_lock_lock (ai->lock);
++  int ret = ai->value;
++  gl_lock_unlock (ai->lock);
++  return ret;
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  gl_lock_lock (ai->lock);
++  ai->value = new_value;
++  gl_lock_unlock (ai->lock);
++}
++#endif
++
+ #define ACCOUNT_COUNT 4
+ 
+ static int account[ACCOUNT_COUNT];
+@@ -170,12 +222,12 @@ lock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int lock_checker_done;
++static struct atomic_int lock_checker_done;
+ 
+ static void *
+ lock_checker_thread (void *arg)
+ {
+-  while (!lock_checker_done)
++  while (get_atomic_int_value (&lock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check lock\n", gl_thread_self_pointer ());
+       gl_lock_lock (my_lock);
+@@ -200,7 +252,8 @@ test_lock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  lock_checker_done = 0;
++  init_atomic_int (&lock_checker_done);
++  set_atomic_int_value (&lock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   checkerthread = gl_thread_create (lock_checker_thread, NULL);
+@@ -210,7 +263,7 @@ test_lock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  lock_checker_done = 1;
++  set_atomic_int_value (&lock_checker_done, 1);
+   gl_thread_join (checkerthread, NULL);
+   check_accounts ();
+ }
+@@ -254,12 +307,12 @@ rwlock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int rwlock_checker_done;
++static struct atomic_int rwlock_checker_done;
+ 
+ static void *
+ rwlock_checker_thread (void *arg)
+ {
+-  while (!rwlock_checker_done)
++  while (get_atomic_int_value (&rwlock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check rdlock\n", gl_thread_self_pointer ());
+       gl_rwlock_rdlock (my_rwlock);
+@@ -284,7 +337,8 @@ test_rwlock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  rwlock_checker_done = 0;
++  init_atomic_int (&rwlock_checker_done);
++  set_atomic_int_value (&rwlock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+@@ -295,7 +349,7 @@ test_rwlock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  rwlock_checker_done = 1;
++  set_atomic_int_value (&rwlock_checker_done, 1);
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (checkerthreads[i], NULL);
+   check_accounts ();
+@@ -356,12 +410,12 @@ reclock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int reclock_checker_done;
++static struct atomic_int reclock_checker_done;
+ 
+ static void *
+ reclock_checker_thread (void *arg)
+ {
+-  while (!reclock_checker_done)
++  while (get_atomic_int_value (&reclock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check lock\n", gl_thread_self_pointer ());
+       gl_recursive_lock_lock (my_reclock);
+@@ -386,7 +440,8 @@ test_recursive_lock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  reclock_checker_done = 0;
++  init_atomic_int (&reclock_checker_done);
++  set_atomic_int_value (&reclock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   checkerthread = gl_thread_create (reclock_checker_thread, NULL);
+@@ -396,7 +451,7 @@ test_recursive_lock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  reclock_checker_done = 1;
++  set_atomic_int_value (&reclock_checker_done, 1);
+   gl_thread_join (checkerthread, NULL);
+   check_accounts ();
+ }
-- 
2.12.2





Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Mon, 17 Apr 2017 10:19:03 GMT) Full text and rfc822 format available.

Message #59 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: 26441 <at> debbugs.gnu.org
Cc: Mathieu Othacehe <m.othacehe <at> gmail.com>
Subject: [PATCH 2/3] gnu: findutils: Fix make check issues on multi-core
 machines.
Date: Mon, 17 Apr 2017 12:18:28 +0200
* gnu/packages/patches/findutils-gnulib-multi-core.patch: New file.
* gnu/local.mk (dist_patch): Add previous patch.
* gnu/packages/base.scm (findutils)[patches]: Add a reference
to the previous patch.
---
 gnu/local.mk                                       |   1 +
 gnu/packages/base.scm                              |  10 +-
 .../patches/findutils-gnulib-multi-core.patch      | 294 +++++++++++++++++++++
 3 files changed, 303 insertions(+), 2 deletions(-)
 create mode 100644 gnu/packages/patches/findutils-gnulib-multi-core.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 620fb07f3..b1dbfec5e 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -562,6 +562,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/fcgi-2.4.0-gcc44-fixes.patch		\
   %D%/packages/patches/fcgi-2.4.0-poll.patch			\
   %D%/packages/patches/findutils-localstatedir.patch		\
+  %D%/packages/patches/findutils-gnulib-multi-core.patch	\
   %D%/packages/patches/findutils-test-xargs.patch		\
   %D%/packages/patches/flint-ldconfig.patch			\
   %D%/packages/patches/fltk-shared-lib-defines.patch		\
diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index c56859021..004aacfa0 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -8,6 +8,7 @@
 ;;; Copyright © 2016 Efraim Flashner <efraim <at> flashner.co.il>
 ;;; Copyright © 2016 Jan Nieuwenhuizen <janneke <at> gnu.org>
 ;;; Copyright © 2017 Rene Saavedra <rennes <at> openmailbox.org>
+;;; Copyright © 2017 Mathieu Othacehe <m.othacehe <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -259,8 +260,13 @@ interactive means to merge two files.")
             (sha256
              (base32
               "178nn4dl7wbcw499czikirnkniwnx36argdnqgz4ik9i6zvwkm6y"))
-            (patches (search-patches "findutils-localstatedir.patch"
-                                     "findutils-test-xargs.patch"))))
+            (patches (search-patches
+                      "findutils-localstatedir.patch"
+                      "findutils-test-xargs.patch"
+                      ;; test-lock has performance issues on multi-core
+                      ;; machines, it hangs or takes a long time to complete.
+                      ;; This is a commit from gnulib to fix this issue.
+                      "findutils-gnulib-multi-core.patch"))))
    (build-system gnu-build-system)
    (arguments
     `(#:configure-flags (list
diff --git a/gnu/packages/patches/findutils-gnulib-multi-core.patch b/gnu/packages/patches/findutils-gnulib-multi-core.patch
new file mode 100644
index 000000000..5a37f4f1f
--- /dev/null
+++ b/gnu/packages/patches/findutils-gnulib-multi-core.patch
@@ -0,0 +1,294 @@
+This patch fixes performance problems on multi-core machines
+as reported at <https://bugs.gnu.org/26441>.
+
+See commit 480d374e596a0ee3fed168ab42cd84c313ad3c89 in Gnulib
+by Bruno Haible <bruno <at> clisp.org>.
+
+diff --git a/tests/test-lock.c b/tests/test-lock.c
+index a992f64..fb18dee 100644
+--- a/tests/test-lock.c
++++ b/tests/test-lock.c
+@@ -1,5 +1,5 @@
+ /* Test of locking in multithreaded situations.
+-   Copyright (C) 2005, 2008-2015 Free Software Foundation, Inc.
++   Copyright (C) 2005, 2008-2017 Free Software Foundation, Inc.
+ 
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+@@ -50,6 +50,28 @@
+    Uncomment this to see if the operating system has a fair scheduler.  */
+ #define EXPLICIT_YIELD 1
+ 
++/* Whether to use 'volatile' on some variables that communicate information
++   between threads.  If set to 0, a semaphore or a lock is used to protect
++   these variables.  If set to 1, 'volatile' is used; this is theoretically
++   equivalent but can lead to much slower execution (e.g. 30x slower total
++   run time on a 40-core machine), because 'volatile' does not imply any
++   synchronization/communication between different CPUs.  */
++#define USE_VOLATILE 0
++
++#if USE_POSIX_THREADS && HAVE_SEMAPHORE_H
++/* Whether to use a semaphore to communicate information between threads.
++   If set to 0, a lock is used. If set to 1, a semaphore is used.
++   Uncomment this to reduce the dependencies of this test.  */
++# define USE_SEMAPHORE 1
++/* Mac OS X provides only named semaphores (sem_open); its facility for
++   unnamed semaphores (sem_init) does not work.  */
++# if defined __APPLE__ && defined __MACH__
++#  define USE_NAMED_SEMAPHORE 1
++# else
++#  define USE_UNNAMED_SEMAPHORE 1
++# endif
++#endif
++
+ /* Whether to print debugging messages.  */
+ #define ENABLE_DEBUGGING 0
+ 
+@@ -90,6 +112,12 @@
+ 
+ #include "glthread/thread.h"
+ #include "glthread/yield.h"
++#if USE_SEMAPHORE
++# include <errno.h>
++# include <fcntl.h>
++# include <semaphore.h>
++# include <unistd.h>
++#endif
+ 
+ #if ENABLE_DEBUGGING
+ # define dbgprintf printf
+@@ -103,6 +131,132 @@
+ # define yield()
+ #endif
+ 
++#if USE_VOLATILE
++struct atomic_int {
++  volatile int value;
++};
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++}
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  return ai->value;
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  ai->value = new_value;
++}
++#elif USE_SEMAPHORE
++/* This atomic_int implementation can only support the values 0 and 1.
++   It is initially 0 and can be set to 1 only once.  */
++# if USE_UNNAMED_SEMAPHORE
++struct atomic_int {
++  sem_t semaphore;
++};
++#define atomic_int_semaphore(ai) (&(ai)->semaphore)
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++  sem_init (&ai->semaphore, 0, 0);
++}
++# endif
++# if USE_NAMED_SEMAPHORE
++struct atomic_int {
++  sem_t *semaphore;
++};
++#define atomic_int_semaphore(ai) ((ai)->semaphore)
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++  sem_t *s;
++  unsigned int count;
++  for (count = 0; ; count++)
++    {
++      char name[80];
++      /* Use getpid() in the name, so that different processes running at the
++         same time will not interfere.  Use ai in the name, so that different
++         atomic_int in the same process will not interfere.  Use a count in
++         the name, so that even in the (unlikely) case that a semaphore with
++         the specified name already exists, we can try a different name.  */
++      sprintf (name, "test-lock-%lu-%p-%u",
++               (unsigned long) getpid (), ai, count);
++      s = sem_open (name, O_CREAT | O_EXCL, 0600, 0);
++      if (s == SEM_FAILED)
++        {
++          if (errno == EEXIST)
++            /* Retry with a different name.  */
++            continue;
++          else
++            {
++              perror ("sem_open failed");
++              abort ();
++            }
++        }
++      else
++        {
++          /* Try not to leave a semaphore hanging around on the file system
++             eternally, if we can avoid it.  */
++          sem_unlink (name);
++          break;
++        }
++    }
++  ai->semaphore = s;
++}
++# endif
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  if (sem_trywait (atomic_int_semaphore (ai)) == 0)
++    {
++      if (sem_post (atomic_int_semaphore (ai)))
++        abort ();
++      return 1;
++    }
++  else if (errno == EAGAIN)
++    return 0;
++  else
++    abort ();
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  if (new_value == 0)
++    /* It's already initialized with 0.  */
++    return;
++  /* To set the value 1: */
++  if (sem_post (atomic_int_semaphore (ai)))
++    abort ();
++}
++#else
++struct atomic_int {
++  gl_lock_define (, lock)
++  int value;
++};
++static void
++init_atomic_int (struct atomic_int *ai)
++{
++  gl_lock_init (ai->lock);
++}
++static int
++get_atomic_int_value (struct atomic_int *ai)
++{
++  gl_lock_lock (ai->lock);
++  int ret = ai->value;
++  gl_lock_unlock (ai->lock);
++  return ret;
++}
++static void
++set_atomic_int_value (struct atomic_int *ai, int new_value)
++{
++  gl_lock_lock (ai->lock);
++  ai->value = new_value;
++  gl_lock_unlock (ai->lock);
++}
++#endif
++
+ #define ACCOUNT_COUNT 4
+ 
+ static int account[ACCOUNT_COUNT];
+@@ -170,12 +324,12 @@ lock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int lock_checker_done;
++static struct atomic_int lock_checker_done;
+ 
+ static void *
+ lock_checker_thread (void *arg)
+ {
+-  while (!lock_checker_done)
++  while (get_atomic_int_value (&lock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check lock\n", gl_thread_self_pointer ());
+       gl_lock_lock (my_lock);
+@@ -200,7 +354,8 @@ test_lock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  lock_checker_done = 0;
++  init_atomic_int (&lock_checker_done);
++  set_atomic_int_value (&lock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   checkerthread = gl_thread_create (lock_checker_thread, NULL);
+@@ -210,7 +365,7 @@ test_lock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  lock_checker_done = 1;
++  set_atomic_int_value (&lock_checker_done, 1);
+   gl_thread_join (checkerthread, NULL);
+   check_accounts ();
+ }
+@@ -254,12 +409,12 @@ rwlock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int rwlock_checker_done;
++static struct atomic_int rwlock_checker_done;
+ 
+ static void *
+ rwlock_checker_thread (void *arg)
+ {
+-  while (!rwlock_checker_done)
++  while (get_atomic_int_value (&rwlock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check rdlock\n", gl_thread_self_pointer ());
+       gl_rwlock_rdlock (my_rwlock);
+@@ -284,7 +439,8 @@ test_rwlock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  rwlock_checker_done = 0;
++  init_atomic_int (&rwlock_checker_done);
++  set_atomic_int_value (&rwlock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+@@ -295,7 +451,7 @@ test_rwlock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  rwlock_checker_done = 1;
++  set_atomic_int_value (&rwlock_checker_done, 1);
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (checkerthreads[i], NULL);
+   check_accounts ();
+@@ -356,12 +512,12 @@ reclock_mutator_thread (void *arg)
+   return NULL;
+ }
+ 
+-static volatile int reclock_checker_done;
++static struct atomic_int reclock_checker_done;
+ 
+ static void *
+ reclock_checker_thread (void *arg)
+ {
+-  while (!reclock_checker_done)
++  while (get_atomic_int_value (&reclock_checker_done) == 0)
+     {
+       dbgprintf ("Checker %p before check lock\n", gl_thread_self_pointer ());
+       gl_recursive_lock_lock (my_reclock);
+@@ -386,7 +542,8 @@ test_recursive_lock (void)
+   /* Initialization.  */
+   for (i = 0; i < ACCOUNT_COUNT; i++)
+     account[i] = 1000;
+-  reclock_checker_done = 0;
++  init_atomic_int (&reclock_checker_done);
++  set_atomic_int_value (&reclock_checker_done, 0);
+ 
+   /* Spawn the threads.  */
+   checkerthread = gl_thread_create (reclock_checker_thread, NULL);
+@@ -396,7 +553,7 @@ test_recursive_lock (void)
+   /* Wait for the threads to terminate.  */
+   for (i = 0; i < THREAD_COUNT; i++)
+     gl_thread_join (threads[i], NULL);
+-  reclock_checker_done = 1;
++  set_atomic_int_value (&reclock_checker_done, 1);
+   gl_thread_join (checkerthread, NULL);
+   check_accounts ();
+ }
-- 
2.12.2





Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Mon, 17 Apr 2017 10:19:03 GMT) Full text and rfc822 format available.

Message #62 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: 26441 <at> debbugs.gnu.org
Cc: Mathieu Othacehe <m.othacehe <at> gmail.com>
Subject: [PATCH 1/3] build: Add two missing patches to local.mk.
Date: Mon, 17 Apr 2017 12:18:27 +0200
gnu/local.mk (dist_patch): Add gettext-multi-core.patch and
gettext-gnulib-multi-core.patch.

Commit 480da86d0969a667e8d2a564de535cb73a6a2229 ommited them.
---
 gnu/local.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/gnu/local.mk b/gnu/local.mk
index 51f92f088..620fb07f3 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -593,6 +593,8 @@ dist_patch_DATA =						\
   %D%/packages/patches/gd-php-73968-Fix-109-XBM-reading.patch		\
   %D%/packages/patches/gegl-CVE-2012-4433.patch			\
   %D%/packages/patches/geoclue-config.patch			\
+  %D%/packages/patches/gettext-multi-core.patch          	\
+  %D%/packages/patches/gettext-gnulib-multi-core.patch          \
   %D%/packages/patches/ghc-dont-pass-linker-flags-via-response-files.patch	\
   %D%/packages/patches/ghostscript-CVE-2013-5653.patch		\
   %D%/packages/patches/ghostscript-CVE-2015-3228.patch		\
-- 
2.12.2





Information forwarded to bug-guix <at> gnu.org:
bug#26441; Package guix. (Mon, 17 Apr 2017 19:48:01 GMT) Full text and rfc822 format available.

Message #65 received at 26441 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Mathieu Othacehe <m.othacehe <at> gmail.com>
Cc: 26441 <at> debbugs.gnu.org
Subject: Re: bug#26441: [PATCH 1/2] gnu: findutils: Fix make check issues on
 multi-core machines.
Date: Mon, 17 Apr 2017 21:47:17 +0200
[Message part 1 (text/plain, inline)]
Mathieu Othacehe <m.othacehe <at> gmail.com> writes:

> Hi Marius !
>
>> It looks like you forgot the actual patch :)
>
> Ha ! You're right :)
>
>>
>> Please also add it to gnu/local.mk. TIA!
>
> Ok, I'll a new batch, to be applied to core-updates.

Applied, thank you! Can this be closed now?
[signature.asc (application/pgp-signature, inline)]

Severity set to 'important' from 'normal' Request was from ludo <at> gnu.org (Ludovic Courtès) to control <at> debbugs.gnu.org. (Wed, 19 Apr 2017 19:46:02 GMT) Full text and rfc822 format available.

Reply sent to ludo <at> gnu.org (Ludovic Courtès):
You have taken responsibility. (Sat, 22 Apr 2017 23:31:02 GMT) Full text and rfc822 format available.

Notification sent to ludo <at> gnu.org (Ludovic Courtès):
bug acknowledged by developer. (Sat, 22 Apr 2017 23:31:02 GMT) Full text and rfc822 format available.

Message #72 received at 26441-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Mathieu Othacehe <m.othacehe <at> gmail.com>
Cc: 26441-done <at> debbugs.gnu.org
Subject: Re: bug#26441: Gnulib’s ‘test-lock’ fails to complete on machines with >= 32 cores
Date: Sun, 23 Apr 2017 01:29:48 +0200
Mathieu Othacehe <m.othacehe <at> gmail.com> skribis:

>> Maybe we can simply patch the important packages (findutils,
>> libunistring) and wait and see if others need the patch.  Not ideal, but
>> I can’t think of a better way.
>
> Ok fine for me.
>
> I'll send findutils and libunistring patches for now.

I see these have been committed to core-updates:

  commit b100e0e89ebd1a6e73d6be4e354dd684abb057a4
  Author: Mathieu Othacehe <m.othacehe <at> gmail.com>
  Date:   Mon Apr 17 12:18:29 2017 +0200

      gnu: libunistring: Fix make check issues on multi-core machines.

      * gnu/packages/patches/libunistring-gnulib-multi-core.patch: New file.
      * gnu/local.mk (dist_patch): Add previous patch.
      * gnu/packages/libunistring.scm (libunistring)[patches]: Add a reference
      to previous patch.

      Signed-off-by: Marius Bakke <mbakke <at> fastmail.com>

  commit da8e256a527a21ad88005c1d84514d910b964c19
  Author: Mathieu Othacehe <m.othacehe <at> gmail.com>
  Date:   Mon Apr 17 12:18:28 2017 +0200

      gnu: findutils: Fix make check issues on multi-core machines.

      * gnu/packages/patches/findutils-gnulib-multi-core.patch: New file.
      * gnu/local.mk (dist_patch): Add previous patch.
      * gnu/packages/base.scm (findutils)[patches]: Add a reference
      to the previous patch.

      Signed-off-by: Marius Bakke <mbakke <at> fastmail.com>

Closing this bug now.  Thank you Mathieu!

Ludo’.




bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Sun, 21 May 2017 11:24:04 GMT) Full text and rfc822 format available.

Did not alter fixed versions and reopened. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Wed, 30 Aug 2017 14:16:02 GMT) Full text and rfc822 format available.

This bug report was last modified 6 years and 212 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.