GNU bug report logs -
#63986
Julia is very slow
Previous Next
Reported by: csantosb <at> inventati.org
Date: Fri, 9 Jun 2023 21:43:01 UTC
Severity: normal
Tags: patch
Done: Efraim Flashner <efraim <at> flashner.co.il>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 63986 in the body.
You can then email your comments to 63986 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Fri, 09 Jun 2023 21:43:01 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
csantosb <at> inventati.org:
New bug report received and forwarded. Copy sent to
bug-guix <at> gnu.org.
(Fri, 09 Jun 2023 21:43:01 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Hi guix,
I just noticed that the following line
julia -e 'using Pkg; Pkg.add("BenchmarkTools"); using BenchmarkTools; N = 1000; A = rand(N, N); B = rand(N, N); @btime $A*$B'
gives around 530 ms in my laptop when using guix provided julia. Same
behavior when running within a guix container.
This very same code, using the binary one may download from the julia
site gives 15 ms.
I’m using here an up to date guix. As a reference, julia binary
provided by archlinux takes also 15 ms.
Thanks,
Cayetano Santos
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Wed, 14 Jun 2023 16:01:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hi Cayetano,
Cc’in Efraim, Simon, and Nicolas who’ve looked into Julia packaging in
the past. Hopefully we can get inspiration from Arch’s build recipe!
Thanks,
Ludo’.
Cayetano Santos <csantosb <at> inventati.org> skribis:
> Hi guix,
>
> I just noticed that the following line
>
> julia -e 'using Pkg; Pkg.add("BenchmarkTools"); using BenchmarkTools; N = 1000; A = rand(N, N); B = rand(N, N); @btime $A*$B'
>
> gives around 530 ms in my laptop when using guix provided julia. Same
> behavior when running within a guix container.
>
> This very same code, using the binary one may download from the julia
> site gives 15 ms.
>
> I’m using here an up to date guix. As a reference, julia binary
> provided by archlinux takes also 15 ms.
>
> Thanks,
>
> Cayetano Santos
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Mon, 19 Jun 2023 16:37:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hi,
On Wed, 14 Jun 2023 at 18:00, Ludovic Courtès <ludo <at> gnu.org> wrote:
> Cc’in Efraim, Simon, and Nicolas who’ve looked into Julia packaging in
> the past. Hopefully we can get inspiration from Arch’s build recipe!
Hum, the difference seems:
USE_SYSTEM_DSFMT=0
USE_SYSTEM_LIBUV=0
which are set to 1 in our Guix recipe; I guess it does not come from
that. And we link against OpenBLAS and they link against some NETLIB; I
guess it does not come from that. Well, the other difference could be
“make release” that we do not run. Maybe?
Somehow, it seems from the ability to exploit the multicore, IIUC.
Using the binary generated by upstream:
--8<---------------cut here---------------start------------->8---
$ ldd julia-1.9.1/bin/julia
linux-vdso.so.1 (0x00007fffd83f1000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f08fb274000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f08fb251000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f08fb05f000)
libjulia.so.1 => /tmp/julia-1.9.1/bin/../lib/libjulia.so.1 (0x00007f08fb03c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f08fb28e000)
$ ldd julia-1.6.7/bin/julia
linux-vdso.so.1 (0x00007fffcdbd7000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f822423f000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f822421c000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f822402a000)
libjulia.so.1 => /tmp/julia-1.6.7/bin/../lib/libjulia.so.1 (0x00007f8223e04000)
/lib64/ld-linux-x86-64.so.2 (0x00007f8224259000)
--8<---------------cut here---------------end--------------->8---
And please note it also contain all these binaries:
--8<---------------cut here---------------start------------->8---
libamd.so -> libamd.so.2.4.6
libamd.so.2 -> libamd.so.2.4.6
libamd.so.2.4.6
libatomic.so -> libatomic.so.1.2.0
libatomic.so.1 -> libatomic.so.1.2.0
libatomic.so.1.2.0
libblastrampoline.so -> libblastrampoline.so.5
libblastrampoline.so.5
libblastrampoline.so.5.4.0 -> libblastrampoline.so.5
[...]
libsuitesparseconfig.so -> libsuitesparseconfig.so.5.10.1
libsuitesparseconfig.so.5 -> libsuitesparseconfig.so.5.10.1
libsuitesparseconfig.so.5.10.1
libumfpack.so -> libumfpack.so.5.7.9
libumfpack.so.5 -> libumfpack.so.5.7.9
libumfpack.so.5.7.9
libunwind.so -> libunwind.so.8.0.1
libunwind.so.8 -> libunwind.so.8.0.1
libunwind.so.8.0.1
libuv.so -> libuv.so.2.0.0
libuv.so.2 -> libuv.so.2.0.0
libuv.so.2.0.0
libz.so -> libz.so.1
libz.so.1 -> libz.so.1.2.13
libz.so.1.2.13
sys.so
--8<---------------cut here---------------end--------------->8---
I get these time:
+ ~7ms v1.9.1
+ ~18ms v1.6.7
compared to ~500ms of v1.8.3 provided by Guix.
I guess the issue is about “threading”. Most of the time is about
’futex’ in the Guix version.
--8<---------------cut here---------------start------------->8---
$ head v1.9.1.txt
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
83.48 5.682198 6 926675 sched_yield
14.87 1.011882 987 1025 futex
0.42 0.028269 2 12308 rt_sigprocmask
0.35 0.023628 9 2592 madvise
0.18 0.012532 6266 2 wait4
0.18 0.012353 10 1227 epoll_wait
0.17 0.011707 2 5015 13 read
0.09 0.006235 13 448 brk
$ head v1.6.7.txt
245193 ????( <detached ...>
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
86.41 5.870043 5 1008236 sched_yield
12.85 0.872865 467 1869 futex
0.26 0.017538 3 5165 madvise
0.09 0.006171 2 2173 12 read
0.07 0.004486 13 321 brk
0.06 0.004242 2 1772 rt_sigprocmask
0.05 0.003197 2 1554 456 statx
$ head vguix.txt
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
88.96 7.843293 4621 1697 2 futex
6.88 0.606245 31 19080 sched_yield
1.74 0.153092 153092 1 1 rt_sigtimedwait
0.48 0.041975 1 26317 1 read
0.36 0.032148 1 27602 rt_sigprocmask
0.34 0.030236 10078 3 1 wait4
0.33 0.028833 1 20780 mincore
0.33 0.028801 1 22424 write
--8<---------------cut here---------------end--------------->8---
Hum, I do not know… It needs some investigations.
Thanks for the report.
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Wed, 21 Jun 2023 14:37:01 GMT)
Full text and
rfc822 format available.
Message #14 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hey!
The benchmark you posted, Cayetano, is:
julia -e 'using Pkg; Pkg.add("BenchmarkTools"); using BenchmarkTools; N = 1000; A = rand(N, N); B = rand(N, N); @btime $A*$B'
This is a matrix multiplication that gets delegated to the underlying
BLAS right. Running it under ‘perf record’ confirms it:
--8<---------------cut here---------------start------------->8---
Samples: 139K of event 'cycles:u', Event count (approx.): 99624880590
Overhead Command Shared Object Symbol
35.27% .julia-real libblas.so.3.9.0 [.] dgemm_
3.99% .julia-real libjulia-internal.so.1.8 [.] gc_mark_loop
2.60% .julia-real libjulia-internal.so.1.8 [.] apply_cl
1.06% .julia-real libjulia-internal.so.1.8 [.] jl_get_binding_
--8<---------------cut here---------------end--------------->8---
We’re using libblas.so (presumably from the ‘lapack’ package) and not
OpenBLAS, so no wonder it’s slow.
Could it be that:
"LIBBLAS=-lopenblas"
"LIBBLASNAME=libopenblas"
is ineffective? I think we have a lead!
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Wed, 21 Jun 2023 20:50:02 GMT)
Full text and
rfc822 format available.
Message #17 received at 63986 <at> debbugs.gnu.org (full text, mbox):
>mer. 21 juin 2023 at 16:36, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:
> Hey!
>
> The benchmark you posted, Cayetano, is:
>
> julia -e 'using Pkg; Pkg.add("BenchmarkTools"); using BenchmarkTools; N = 1000; A = rand(N, N); B = rand(N, N); @btime $A*$B'
>
> This is a matrix multiplication that gets delegated to the underlying
> BLAS right. Running it under ‘perf record’ confirms it:
>
> Samples: 139K of event 'cycles:u', Event count (approx.): 99624880590
> Overhead Command Shared Object Symbol
> 35.27% .julia-real libblas.so.3.9.0 [.] dgemm_
> 3.99% .julia-real libjulia-internal.so.1.8 [.] gc_mark_loop
> 2.60% .julia-real libjulia-internal.so.1.8 [.] apply_cl
> 1.06% .julia-real libjulia-internal.so.1.8 [.] jl_get_binding_
>
> We’re using libblas.so (presumably from the ‘lapack’ package) and not
> OpenBLAS, so no wonder it’s slow.
>
> Could it be that:
>
> "LIBBLAS=-lopenblas"
> "LIBBLASNAME=libopenblas"
>
> is ineffective? I think we have a lead!
Are we following all instructions here ?
https://docs.julialang.org/en/v1.8/devdocs/build/distributing/#Notes-on-BLAS-and-LAPACK
I’m thinking about the variables LIBLAPACK and LIBLAPACKNAME.
C.
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Thu, 22 Jun 2023 13:28:02 GMT)
Full text and
rfc822 format available.
Message #20 received at 63986 <at> debbugs.gnu.org (full text, mbox):
>mer. 21 juin 2023 at 22:39, Cayetano Santos <csantosb <at> inventati.org> wrote:
>>mer. 21 juin 2023 at 16:36, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:
>
>> Hey!
>>
>> The benchmark you posted, Cayetano, is:
>>
>> julia -e 'using Pkg; Pkg.add("BenchmarkTools"); using BenchmarkTools; N = 1000; A = rand(N, N); B = rand(N, N); @btime $A*$B'
>>
>> This is a matrix multiplication that gets delegated to the underlying
>> BLAS right. Running it under ‘perf record’ confirms it:
>>
>> Samples: 139K of event 'cycles:u', Event count (approx.): 99624880590
>> Overhead Command Shared Object Symbol
>> 35.27% .julia-real libblas.so.3.9.0 [.] dgemm_
>> 3.99% .julia-real libjulia-internal.so.1.8 [.] gc_mark_loop
>> 2.60% .julia-real libjulia-internal.so.1.8 [.] apply_cl
>> 1.06% .julia-real libjulia-internal.so.1.8 [.] jl_get_binding_
>>
>> We’re using libblas.so (presumably from the ‘lapack’ package) and not
>> OpenBLAS, so no wonder it’s slow.
>>
>> Could it be that:
>>
>> "LIBBLAS=-lopenblas"
>> "LIBBLASNAME=libopenblas"
>>
>> is ineffective? I think we have a lead!
>
> Are we following all instructions here ?
>
> https://docs.julialang.org/en/v1.8/devdocs/build/distributing/#Notes-on-BLAS-and-LAPACK
>
> I’m thinking about the variables LIBLAPACK and LIBLAPACKNAME.
To complete my previous comment, I just realised that
Base.USE_BLAS64
gives "true" when running fast. Guix julia gives "false".
C.
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Thu, 22 Jun 2023 14:38:02 GMT)
Full text and
rfc822 format available.
Message #23 received at 63986 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Thu, Jun 22, 2023 at 03:26:30PM +0200, Cayetano Santos wrote:
>
>
> >mer. 21 juin 2023 at 22:39, Cayetano Santos <csantosb <at> inventati.org> wrote:
>
> >>mer. 21 juin 2023 at 16:36, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:
> >
> >> Hey!
> >>
> >> The benchmark you posted, Cayetano, is:
> >>
> >> julia -e 'using Pkg; Pkg.add("BenchmarkTools"); using BenchmarkTools; N = 1000; A = rand(N, N); B = rand(N, N); @btime $A*$B'
I've been having a hard time with that command, so I've been running
julia -e 'using Pkg; Pkg.add("BenchmarkTools"); using BenchmarkTools; N = 1000; A = rand(N, N); B = rand(N, N); @time A*B'
I'm not sure if that's useful or not but I didn't see a different with
LIBBLAS* set.
> >>
> >> This is a matrix multiplication that gets delegated to the underlying
> >> BLAS right. Running it under ‘perf record’ confirms it:
> >>
> >> Samples: 139K of event 'cycles:u', Event count (approx.): 99624880590
> >> Overhead Command Shared Object Symbol
> >> 35.27% .julia-real libblas.so.3.9.0 [.] dgemm_
> >> 3.99% .julia-real libjulia-internal.so.1.8 [.] gc_mark_loop
> >> 2.60% .julia-real libjulia-internal.so.1.8 [.] apply_cl
> >> 1.06% .julia-real libjulia-internal.so.1.8 [.] jl_get_binding_
> >>
> >> We’re using libblas.so (presumably from the ‘lapack’ package) and not
> >> OpenBLAS, so no wonder it’s slow.
> >>
> >> Could it be that:
> >>
> >> "LIBBLAS=-lopenblas"
> >> "LIBBLASNAME=libopenblas"
> >>
> >> is ineffective? I think we have a lead!
> >
> > Are we following all instructions here ?
> >
> > https://docs.julialang.org/en/v1.8/devdocs/build/distributing/#Notes-on-BLAS-and-LAPACK
> >
> > I’m thinking about the variables LIBLAPACK and LIBLAPACKNAME.
>
> To complete my previous comment, I just realised that
>
> Base.USE_BLAS64
>
> gives "true" when running fast. Guix julia gives "false".
>
> C.
I'll see what comes with switching it to blas64
--
Efraim Flashner <efraim <at> flashner.co.il> רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Thu, 22 Jun 2023 15:53:01 GMT)
Full text and
rfc822 format available.
Message #26 received at 63986 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi,
On Thu, 22 Jun 2023 at 15:26, Cayetano Santos via Bug reports for GNU Guix <bug-guix <at> gnu.org> wrote:
>> Are we following all instructions here ?
>>
>> https://docs.julialang.org/en/v1.8/devdocs/build/distributing/#Notes-on-BLAS-and-LAPACK
[...]
> Base.USE_BLAS64
>
> gives "true" when running fast. Guix julia gives "false".
When I try USE_BLAS64=1, then I get:
--8<---------------cut here---------------start------------->8---
┌ Error: No loaded BLAS libraries were built with ILP64 support
└ @ LinearAlgebra.BLAS /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/share/julia/stdlib/v1.8/LinearAlgebra/src/blas.jl:155
Quitting.
--8<---------------cut here---------------end--------------->8---
And from the documentation above, it reads:
[...] while on 64-bit architectures, Julia builds OpenBLAS to
use 64-bit integers (ILP64). It is essential that all Julia
functions that call BLAS and LAPACK API routines use integers of
the correct width.
Well using the patch attached, I get:
6.884 ms (2 allocations: 7.63 MiB)
compared to the previous
494.345 ms (2 allocations: 7.63 MiB)
WDYT about this patch?
[0001-gnu-julia-Use-openblas-with-ILP64-support.patch (text/x-diff, inline)]
From 024c92fac091f59dcdbd3a78eb6ea77bb15b2170 Mon Sep 17 00:00:00 2001
Message-Id: <024c92fac091f59dcdbd3a78eb6ea77bb15b2170.1687449033.git.zimon.toutoune <at> gmail.com>
From: Simon Tournier <zimon.toutoune <at> gmail.com>
Date: Thu, 22 Jun 2023 17:45:50 +0200
Subject: [PATCH] gnu: julia: Use openblas with ILP64 support.
Fixes <https://bugs.gnu.org/63986>.
Reported by Cayetano Santos <csantosb <at> inventati.org>.
* gnu/packages/julia.scm (julia)[arguments]<#:make-flags>: Use OpenBLAS with
ILP64 support.
[inputs]: Replace openblas by openblas-ilp64.
---
gnu/packages/julia.scm | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
index ba54175822..a034cbf543 100644
--- a/gnu/packages/julia.scm
+++ b/gnu/packages/julia.scm
@@ -481,10 +481,10 @@ (define-public julia
,@(if (target-aarch64?)
`("USE_BLAS64=0")
- '())
+ `("USE_BLAS64=1"))
- "LIBBLAS=-lopenblas"
- "LIBBLASNAME=libopenblas"
+ "LIBBLAS=-lopenblas_ilp64"
+ "LIBBLASNAME=libopenblas_ilp64"
(string-append "UTF8PROC_INC="
(assoc-ref %build-inputs "utf8proc")
@@ -513,7 +513,7 @@ (define-public julia
("llvm" ,llvm-julia)
("mbedtls-apache" ,mbedtls-apache)
("mpfr" ,mpfr)
- ("openblas" ,openblas)
+ ("openblas" ,openblas-ilp64)
("openlibm" ,openlibm)
("p7zip" ,p7zip)
("pcre2" ,pcre2)
base-commit: 37c2e94cec6cb8b5e0e93e7b6c712c3b187ca5db
--
2.38.1
[Message part 3 (text/plain, inline)]
Well, I need to do more tests but I guess that’s the good direction. :-)
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Thu, 22 Jun 2023 16:26:02 GMT)
Full text and
rfc822 format available.
Message #29 received at 63986 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Thu, Jun 22, 2023 at 05:52:15PM +0200, Simon Tournier wrote:
> Hi,
>
> On Thu, 22 Jun 2023 at 15:26, Cayetano Santos via Bug reports for GNU Guix <bug-guix <at> gnu.org> wrote:
> >> Are we following all instructions here ?
> >>
> >> https://docs.julialang.org/en/v1.8/devdocs/build/distributing/#Notes-on-BLAS-and-LAPACK
>
> [...]
>
> > Base.USE_BLAS64
> >
> > gives "true" when running fast. Guix julia gives "false".
>
> When I try USE_BLAS64=1, then I get:
>
> --8<---------------cut here---------------start------------->8---
> ┌ Error: No loaded BLAS libraries were built with ILP64 support
> └ @ LinearAlgebra.BLAS /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/share/julia/stdlib/v1.8/LinearAlgebra/src/blas.jl:155
> Quitting.
> --8<---------------cut here---------------end--------------->8---
>
> And from the documentation above, it reads:
>
> [...] while on 64-bit architectures, Julia builds OpenBLAS to
> use 64-bit integers (ILP64). It is essential that all Julia
> functions that call BLAS and LAPACK API routines use integers of
> the correct width.
>
> Well using the patch attached, I get:
>
> 6.884 ms (2 allocations: 7.63 MiB)
>
> compared to the previous
>
> 494.345 ms (2 allocations: 7.63 MiB)
>
> WDYT about this patch?
(ins)efraim <at> 3900XT ~/workspace/guix$ cat /gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/lib/pkgconfig/openblas.pc
libdir=/gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/lib
includedir=/gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/include
openblas_config= USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS= NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= generic MAX_THREADS=128
version=0.3.20
extralib=-lm -lpthread -lgfortran -lm -lpthread -lgfortran
Name: openblas
Description: OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version
Version: ${version}
URL: https://github.com/xianyi/OpenBLAS
Libs: -L${libdir} -lopenblas
Libs.private: ${extralib}
Cflags: -I${includedir}
Looks like it should be "LIBBLAS=-lopenblas"
It might need some tuning anyway, currently we have julia for i686 and
switching to solely openblas-ilp64 we'd lose 32-bit support.
I also noticed the julia expects the 64-bit openblas to be libopenblas64
(which happens to be what Debian¹ has). Would we need to adapt anything
in stdlib/OpenBLAS_jll/src/OpenBLAS_jll.jl?
Also, are we supposed to build lapack with our openblas as an input?
¹ https://sources.debian.org/src/openblas/0.3.21%2Bds-4/debian/rules/#L71
> From 024c92fac091f59dcdbd3a78eb6ea77bb15b2170 Mon Sep 17 00:00:00 2001
> Message-Id: <024c92fac091f59dcdbd3a78eb6ea77bb15b2170.1687449033.git.zimon.toutoune <at> gmail.com>
> From: Simon Tournier <zimon.toutoune <at> gmail.com>
> Date: Thu, 22 Jun 2023 17:45:50 +0200
> Subject: [PATCH] gnu: julia: Use openblas with ILP64 support.
>
> Fixes <https://bugs.gnu.org/63986>.
> Reported by Cayetano Santos <csantosb <at> inventati.org>.
>
> * gnu/packages/julia.scm (julia)[arguments]<#:make-flags>: Use OpenBLAS with
> ILP64 support.
> [inputs]: Replace openblas by openblas-ilp64.
> ---
> gnu/packages/julia.scm | 8 ++++----
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
> index ba54175822..a034cbf543 100644
> --- a/gnu/packages/julia.scm
> +++ b/gnu/packages/julia.scm
> @@ -481,10 +481,10 @@ (define-public julia
>
> ,@(if (target-aarch64?)
> `("USE_BLAS64=0")
> - '())
> + `("USE_BLAS64=1"))
>
> - "LIBBLAS=-lopenblas"
> - "LIBBLASNAME=libopenblas"
> + "LIBBLAS=-lopenblas_ilp64"
> + "LIBBLASNAME=libopenblas_ilp64"
>
> (string-append "UTF8PROC_INC="
> (assoc-ref %build-inputs "utf8proc")
> @@ -513,7 +513,7 @@ (define-public julia
> ("llvm" ,llvm-julia)
> ("mbedtls-apache" ,mbedtls-apache)
> ("mpfr" ,mpfr)
> - ("openblas" ,openblas)
> + ("openblas" ,openblas-ilp64)
> ("openlibm" ,openlibm)
> ("p7zip" ,p7zip)
> ("pcre2" ,pcre2)
>
> base-commit: 37c2e94cec6cb8b5e0e93e7b6c712c3b187ca5db
> --
> 2.38.1
>
>
> Well, I need to do more tests but I guess that’s the good direction. :-)
>
> Cheers,
> simon
--
Efraim Flashner <efraim <at> flashner.co.il> רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Thu, 22 Jun 2023 17:00:02 GMT)
Full text and
rfc822 format available.
Message #32 received at 63986 <at> debbugs.gnu.org (full text, mbox):
> Looks like it should be "LIBBLAS=-lopenblas"
>
> It might need some tuning anyway, currently we have julia for i686 and
> switching to solely openblas-ilp64 we'd lose 32-bit support.
>
> I also noticed the julia expects the 64-bit openblas to be libopenblas64
> (which happens to be what Debian¹ has). Would we need to adapt anything
> in stdlib/OpenBLAS_jll/src/OpenBLAS_jll.jl?
>
> Also, are we supposed to build lapack with our openblas as an input?
Being used to Arch, it seems to me the way they do things is the way to
go, or at least, a good reference (other than the support for 32-bit)
https://archlinux.org/packages/?sort=&q=blas&maintainer=&flagged=
https://gitlab.archlinux.org/archlinux/packaging/packages/julia/-/blob/main/PKGBUILD
C.
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Thu, 22 Jun 2023 17:14:02 GMT)
Full text and
rfc822 format available.
Message #35 received at 63986 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi,
On Thu, 22 Jun 2023 at 19:25, Efraim Flashner <efraim <at> flashner.co.il> wrote:
> It might need some tuning anyway, currently we have julia for i686 and
> switching to solely openblas-ilp64 we'd lose 32-bit support.
Yes, indeed. This attached patch conditionally switches.
[v2-0001-gnu-julia-Conditionally-use-openblas-with-ILP64-s.patch (text/x-diff, inline)]
From 9e1912148d4a691a5484fc7f87597b579ba61785 Mon Sep 17 00:00:00 2001
Message-Id: <9e1912148d4a691a5484fc7f87597b579ba61785.1687453627.git.zimon.toutoune <at> gmail.com>
From: Simon Tournier <zimon.toutoune <at> gmail.com>
Date: Thu, 22 Jun 2023 17:45:50 +0200
Subject: [PATCH v2] gnu: julia: Conditionally use openblas with ILP64 support.
Fixes <https://bugs.gnu.org/63986>.
Reported by Cayetano Santos <csantosb <at> inventati.org>.
* gnu/packages/julia.scm (julia)[arguments]<#:make-flags>: Conditionally use
OpenBLAS with ILP64 support for x86-64 target.
[inputs]: Conditionally replace openblas by openblas-ilp64 for x86-64 target.
---
gnu/packages/julia.scm | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
index ba54175822..3ffd1fa4cb 100644
--- a/gnu/packages/julia.scm
+++ b/gnu/packages/julia.scm
@@ -483,8 +483,12 @@ (define-public julia
`("USE_BLAS64=0")
'())
- "LIBBLAS=-lopenblas"
- "LIBBLASNAME=libopenblas"
+ ,@(if (target-x86-64?)
+ `("USE_BLAS64=1"
+ "LIBBLAS=-lopenblas_ilp64"
+ "LIBBLASNAME=libopenblas_ilp64")
+ `("LIBBLAS=-lopenblas"
+ "LIBBLASNAME=libopenblas"))
(string-append "UTF8PROC_INC="
(assoc-ref %build-inputs "utf8proc")
@@ -513,7 +517,9 @@ (define-public julia
("llvm" ,llvm-julia)
("mbedtls-apache" ,mbedtls-apache)
("mpfr" ,mpfr)
- ("openblas" ,openblas)
+ ,@(if (target-x86-64?)
+ `(("openblas" ,openblas-ilp64))
+ `(("openblas" ,openblas)))
("openlibm" ,openlibm)
("p7zip" ,p7zip)
("pcre2" ,pcre2)
base-commit: 37c2e94cec6cb8b5e0e93e7b6c712c3b187ca5db
--
2.38.1
[Message part 3 (text/plain, inline)]
Well, the current recipe could be improved but let focus on the current
bug report. And I guess the slowness come from an incorrect use of BLAS
for x86-64. That seems fixed by the patch, no?
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Thu, 22 Jun 2023 18:48:02 GMT)
Full text and
rfc822 format available.
Message #38 received at 63986 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi,
On Thu, 22 Jun 2023 at 19:25, Efraim Flashner <efraim <at> flashner.co.il> wrote:
> (ins)efraim <at> 3900XT ~/workspace/guix$ cat /gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/lib/pkgconfig/openblas.pc
> libdir=/gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/lib
> includedir=/gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/include
> openblas_config= USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS= NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= generic MAX_THREADS=128
> version=0.3.20
> extralib=-lm -lpthread -lgfortran -lm -lpthread -lgfortran
> Name: openblas
> Description: OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version
> Version: ${version}
> URL: https://github.com/xianyi/OpenBLAS
> Libs: -L${libdir} -lopenblas
> Libs.private: ${extralib}
> Cflags: -I${includedir}
>
> Looks like it should be "LIBBLAS=-lopenblas"
I propose to tweak openblas-ilp64. Currently it looks like:
--8<---------------cut here---------------start------------->8---
$ tree $(guix build openblas-ilp64)/lib
/gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/lib
├── cmake
│ └── openblas
│ ├── OpenBLASConfig.cmake
│ └── OpenBLASConfigVersion.cmake
├── libopenblas_ilp64p-r0.3.20.so
├── libopenblas_ilp64.so -> libopenblas_ilp64p-r0.3.20.so
├── libopenblas_ilp64.so.0 -> libopenblas_ilp64p-r0.3.20.so
└── pkgconfig
└── openblas.pc
--8<---------------cut here---------------end--------------->8---
which is inconsistent with pkgconfig as you noticed above. Therefore, I
am proposing the addition of a symlink of libopenblas_ilp64p.so to
libopenblas.so. For instance this attached patch.
[v3-0001-gnu-openblas-ilp64-Install-symlink-to-libopenblas.patch (text/x-diff, inline)]
From 0c8c7e9371d11972f4a6012ef503ef3057c91364 Mon Sep 17 00:00:00 2001
Message-Id: <0c8c7e9371d11972f4a6012ef503ef3057c91364.1687459454.git.zimon.toutoune <at> gmail.com>
From: Simon Tournier <zimon.toutoune <at> gmail.com>
Date: Thu, 22 Jun 2023 20:31:26 +0200
Subject: [PATCH v3 1/2] gnu: openblas-ilp64: Install symlink to libopenblas.
* gnu/packages/maths.scm (openblas-ilp64)[arguments]: Add phases for
installing symlink to libopenblas.
---
gnu/packages/maths.scm | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index f5a2181905..5c39ab8b94 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -4646,7 +4646,13 @@ (define-public openblas-ilp64
(substitute-keyword-arguments (package-arguments openblas)
((#:make-flags flags #~'())
#~(append (list "INTERFACE64=1" "LIBNAMESUFFIX=ilp64")
- #$flags))))
+ #$flags))
+ ((#:phases phases)
+ #~(modify-phases #$phases
+ (add-after 'install 'install-symlink
+ (lambda _
+ (symlink "libopenblas_ilp64.so"
+ (string-append #$output "/lib/libopenblas.so"))))))))
(synopsis "Optimized BLAS library based on GotoBLAS (ILP64 version)")
(license license:bsd-3)))
base-commit: 37c2e94cec6cb8b5e0e93e7b6c712c3b187ca5db
--
2.38.1
[Message part 3 (text/plain, inline)]
Then, the patch for Julia looks like the other attached patch.
[v3-0002-gnu-julia-Conditionally-use-openblas-with-ILP64-s.patch (text/x-diff, inline)]
From 8563a738703d133b44ae05b91b7448ca56d280b5 Mon Sep 17 00:00:00 2001
Message-Id: <8563a738703d133b44ae05b91b7448ca56d280b5.1687459454.git.zimon.toutoune <at> gmail.com>
In-Reply-To: <0c8c7e9371d11972f4a6012ef503ef3057c91364.1687459454.git.zimon.toutoune <at> gmail.com>
References: <0c8c7e9371d11972f4a6012ef503ef3057c91364.1687459454.git.zimon.toutoune <at> gmail.com>
From: Simon Tournier <zimon.toutoune <at> gmail.com>
Date: Thu, 22 Jun 2023 17:45:50 +0200
Subject: [PATCH v3 2/2] gnu: julia: Conditionally use openblas with ILP64
support.
Fixes <https://bugs.gnu.org/63986>.
Reported by Cayetano Santos <csantosb <at> inventati.org>.
* gnu/packages/julia.scm (julia)[arguments]<#:make-flags>: Conditionally use
64-bit BLAS for x86-64 target.
[inputs]: Conditionally replace openblas by openblas-ilp64 for x86-64 target.
---
gnu/packages/julia.scm | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
index ba54175822..17a27f4928 100644
--- a/gnu/packages/julia.scm
+++ b/gnu/packages/julia.scm
@@ -479,9 +479,9 @@ (define-public julia
"NO_GIT=1" ; build from release tarball.
"USE_GPL_LIBS=1" ; proudly
- ,@(if (target-aarch64?)
- `("USE_BLAS64=0")
- '())
+ ,@(if (target-x86-64?)
+ `("USE_BLAS64=1")
+ '())
"LIBBLAS=-lopenblas"
"LIBBLASNAME=libopenblas"
@@ -513,7 +513,9 @@ (define-public julia
("llvm" ,llvm-julia)
("mbedtls-apache" ,mbedtls-apache)
("mpfr" ,mpfr)
- ("openblas" ,openblas)
+ ,@(if (target-x86-64?)
+ `(("openblas" ,openblas-ilp64))
+ `(("openblas" ,openblas)))
("openlibm" ,openlibm)
("p7zip" ,p7zip)
("pcre2" ,pcre2)
--
2.38.1
[Message part 5 (text/plain, inline)]
WDYT?
Well, I am running all the testsuite for checking if all is correct.
Somehow, I think this is direction. Otherwise, what would you suggest?
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Thu, 22 Jun 2023 18:56:02 GMT)
Full text and
rfc822 format available.
Message #41 received at 63986 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Thu, Jun 22, 2023 at 08:47:48PM +0200, Simon Tournier wrote:
> Hi,
>
> On Thu, 22 Jun 2023 at 19:25, Efraim Flashner <efraim <at> flashner.co.il> wrote:
>
> > (ins)efraim <at> 3900XT ~/workspace/guix$ cat /gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/lib/pkgconfig/openblas.pc
> > libdir=/gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/lib
> > includedir=/gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/include
> > openblas_config= USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS= NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= generic MAX_THREADS=128
> > version=0.3.20
> > extralib=-lm -lpthread -lgfortran -lm -lpthread -lgfortran
> > Name: openblas
> > Description: OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version
> > Version: ${version}
> > URL: https://github.com/xianyi/OpenBLAS
> > Libs: -L${libdir} -lopenblas
> > Libs.private: ${extralib}
> > Cflags: -I${includedir}
> >
> > Looks like it should be "LIBBLAS=-lopenblas"
>
> I propose to tweak openblas-ilp64. Currently it looks like:
>
> --8<---------------cut here---------------start------------->8---
> $ tree $(guix build openblas-ilp64)/lib
> /gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/lib
> ├── cmake
> │ └── openblas
> │ ├── OpenBLASConfig.cmake
> │ └── OpenBLASConfigVersion.cmake
> ├── libopenblas_ilp64p-r0.3.20.so
> ├── libopenblas_ilp64.so -> libopenblas_ilp64p-r0.3.20.so
> ├── libopenblas_ilp64.so.0 -> libopenblas_ilp64p-r0.3.20.so
> └── pkgconfig
> └── openblas.pc
> --8<---------------cut here---------------end--------------->8---
>
> which is inconsistent with pkgconfig as you noticed above. Therefore, I
> am proposing the addition of a symlink of libopenblas_ilp64p.so to
> libopenblas.so. For instance this attached patch.
If we drop the "LIBNAMESUFFIX=ilp64" from openblas-ilp64 then we get
libopenblas.so by default without needing to also symlink it into place.
One benefit of this is we'd be able to easily do some package
transformations between openblas and openblas-ilp64.
Currently I'm looking around online to see what the consensus seems to
be with naming openblas-ilp64, and there seems to be a lot of options.
> From 0c8c7e9371d11972f4a6012ef503ef3057c91364 Mon Sep 17 00:00:00 2001
> Message-Id: <0c8c7e9371d11972f4a6012ef503ef3057c91364.1687459454.git.zimon.toutoune <at> gmail.com>
> From: Simon Tournier <zimon.toutoune <at> gmail.com>
> Date: Thu, 22 Jun 2023 20:31:26 +0200
> Subject: [PATCH v3 1/2] gnu: openblas-ilp64: Install symlink to libopenblas.
>
> * gnu/packages/maths.scm (openblas-ilp64)[arguments]: Add phases for
> installing symlink to libopenblas.
> ---
> gnu/packages/maths.scm | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
> index f5a2181905..5c39ab8b94 100644
> --- a/gnu/packages/maths.scm
> +++ b/gnu/packages/maths.scm
> @@ -4646,7 +4646,13 @@ (define-public openblas-ilp64
> (substitute-keyword-arguments (package-arguments openblas)
> ((#:make-flags flags #~'())
> #~(append (list "INTERFACE64=1" "LIBNAMESUFFIX=ilp64")
> - #$flags))))
> + #$flags))
> + ((#:phases phases)
> + #~(modify-phases #$phases
> + (add-after 'install 'install-symlink
> + (lambda _
> + (symlink "libopenblas_ilp64.so"
> + (string-append #$output "/lib/libopenblas.so"))))))))
> (synopsis "Optimized BLAS library based on GotoBLAS (ILP64 version)")
> (license license:bsd-3)))
>
>
> base-commit: 37c2e94cec6cb8b5e0e93e7b6c712c3b187ca5db
> --
> 2.38.1
>
>
> Then, the patch for Julia looks like the other attached patch.
>
> From 8563a738703d133b44ae05b91b7448ca56d280b5 Mon Sep 17 00:00:00 2001
> Message-Id: <8563a738703d133b44ae05b91b7448ca56d280b5.1687459454.git.zimon.toutoune <at> gmail.com>
> In-Reply-To: <0c8c7e9371d11972f4a6012ef503ef3057c91364.1687459454.git.zimon.toutoune <at> gmail.com>
> References: <0c8c7e9371d11972f4a6012ef503ef3057c91364.1687459454.git.zimon.toutoune <at> gmail.com>
> From: Simon Tournier <zimon.toutoune <at> gmail.com>
> Date: Thu, 22 Jun 2023 17:45:50 +0200
> Subject: [PATCH v3 2/2] gnu: julia: Conditionally use openblas with ILP64
> support.
>
> Fixes <https://bugs.gnu.org/63986>.
> Reported by Cayetano Santos <csantosb <at> inventati.org>.
>
> * gnu/packages/julia.scm (julia)[arguments]<#:make-flags>: Conditionally use
> 64-bit BLAS for x86-64 target.
> [inputs]: Conditionally replace openblas by openblas-ilp64 for x86-64 target.
> ---
> gnu/packages/julia.scm | 10 ++++++----
> 1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
> index ba54175822..17a27f4928 100644
> --- a/gnu/packages/julia.scm
> +++ b/gnu/packages/julia.scm
> @@ -479,9 +479,9 @@ (define-public julia
> "NO_GIT=1" ; build from release tarball.
> "USE_GPL_LIBS=1" ; proudly
>
> - ,@(if (target-aarch64?)
> - `("USE_BLAS64=0")
> - '())
> + ,@(if (target-x86-64?)
> + `("USE_BLAS64=1")
> + '())
>
> "LIBBLAS=-lopenblas"
> "LIBBLASNAME=libopenblas"
> @@ -513,7 +513,9 @@ (define-public julia
> ("llvm" ,llvm-julia)
> ("mbedtls-apache" ,mbedtls-apache)
> ("mpfr" ,mpfr)
> - ("openblas" ,openblas)
> + ,@(if (target-x86-64?)
> + `(("openblas" ,openblas-ilp64))
> + `(("openblas" ,openblas)))
> ("openlibm" ,openlibm)
> ("p7zip" ,p7zip)
> ("pcre2" ,pcre2)
> --
> 2.38.1
>
>
> WDYT?
>
> Well, I am running all the testsuite for checking if all is correct.
> Somehow, I think this is direction. Otherwise, what would you suggest?
>
> Cheers,
> simon
>
>
>
--
Efraim Flashner <efraim <at> flashner.co.il> רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Thu, 22 Jun 2023 20:14:01 GMT)
Full text and
rfc822 format available.
Message #44 received at 63986 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Thu, Jun 22, 2023 at 08:47:48PM +0200, Simon Tournier wrote:
> Hi,
>
> On Thu, 22 Jun 2023 at 19:25, Efraim Flashner <efraim <at> flashner.co.il> wrote:
>
> > (ins)efraim <at> 3900XT ~/workspace/guix$ cat /gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/lib/pkgconfig/openblas.pc
> > libdir=/gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/lib
> > includedir=/gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/include
> > openblas_config= USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS= NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= generic MAX_THREADS=128
> > version=0.3.20
> > extralib=-lm -lpthread -lgfortran -lm -lpthread -lgfortran
> > Name: openblas
> > Description: OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version
> > Version: ${version}
> > URL: https://github.com/xianyi/OpenBLAS
> > Libs: -L${libdir} -lopenblas
> > Libs.private: ${extralib}
> > Cflags: -I${includedir}
> >
> > Looks like it should be "LIBBLAS=-lopenblas"
>
> I propose to tweak openblas-ilp64. Currently it looks like:
>
> --8<---------------cut here---------------start------------->8---
> $ tree $(guix build openblas-ilp64)/lib
> /gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/lib
> ├── cmake
> │ └── openblas
> │ ├── OpenBLASConfig.cmake
> │ └── OpenBLASConfigVersion.cmake
> ├── libopenblas_ilp64p-r0.3.20.so
> ├── libopenblas_ilp64.so -> libopenblas_ilp64p-r0.3.20.so
> ├── libopenblas_ilp64.so.0 -> libopenblas_ilp64p-r0.3.20.so
> └── pkgconfig
> └── openblas.pc
> --8<---------------cut here---------------end--------------->8---
>
> which is inconsistent with pkgconfig as you noticed above. Therefore, I
> am proposing the addition of a symlink of libopenblas_ilp64p.so to
> libopenblas.so. For instance this attached patch.
I've attached the patch that I've made it to. julia is built with
openblas-ilp64 on 64-bit architectures, openblas-ilp64 has its
configure-flags adjusted to match what other programs are expecting, and
I've also patched python-numpy to use openblas-ilp64 as another test
target.
--
Efraim Flashner <efraim <at> flashner.co.il> רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[julia-openblas-ilp64.patch (text/plain, attachment)]
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Fri, 23 Jun 2023 09:10:02 GMT)
Full text and
rfc822 format available.
Message #47 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hi!
Simon Tournier <zimon.toutoune <at> gmail.com> skribis:
> + ,@(if (target-x86-64?)
> + `("USE_BLAS64=1"
> + "LIBBLAS=-lopenblas_ilp64"
> + "LIBBLASNAME=libopenblas_ilp64")
> + `("LIBBLAS=-lopenblas"
> + "LIBBLASNAME=libopenblas"))
>
> (string-append "UTF8PROC_INC="
> (assoc-ref %build-inputs "utf8proc")
> @@ -513,7 +517,9 @@ (define-public julia
> ("llvm" ,llvm-julia)
> ("mbedtls-apache" ,mbedtls-apache)
> ("mpfr" ,mpfr)
> - ("openblas" ,openblas)
> + ,@(if (target-x86-64?)
> + `(("openblas" ,openblas-ilp64))
> + `(("openblas" ,openblas)))
Should it be ‘target-64bit?’ instead?
Thumbs up for finding the solution!
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Fri, 23 Jun 2023 09:17:02 GMT)
Full text and
rfc822 format available.
Message #50 received at 63986 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Fri, Jun 23, 2023 at 11:09:44AM +0200, Ludovic Courtès wrote:
> Hi!
>
> Simon Tournier <zimon.toutoune <at> gmail.com> skribis:
>
> > + ,@(if (target-x86-64?)
> > + `("USE_BLAS64=1"
> > + "LIBBLAS=-lopenblas_ilp64"
> > + "LIBBLASNAME=libopenblas_ilp64")
> > + `("LIBBLAS=-lopenblas"
> > + "LIBBLASNAME=libopenblas"))
> >
> > (string-append "UTF8PROC_INC="
> > (assoc-ref %build-inputs "utf8proc")
> > @@ -513,7 +517,9 @@ (define-public julia
> > ("llvm" ,llvm-julia)
> > ("mbedtls-apache" ,mbedtls-apache)
> > ("mpfr" ,mpfr)
> > - ("openblas" ,openblas)
> > + ,@(if (target-x86-64?)
> > + `(("openblas" ,openblas-ilp64))
> > + `(("openblas" ,openblas)))
>
> Should it be ‘target-64bit?’ instead?
It should be, but julia fails the precompile stage with openblas-ilp64
Precompilation complete. Summary:
Total ─────── 1006.632176 seconds
Generation ── 690.274180 seconds 68.5726%
Execution ─── 316.357997 seconds 31.4274%
signal (15): Terminated
in expression starting at none:0
unknown function (ip: 0xfffff4d81b24)
unknown function (ip: 0xfffff4ed10bb)
unknown function (ip: 0xfffff4eda16f)
_ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE at /gnu/store/wa28l476c1bfshfiqcqbmk75zr8ml152-llvm-13.0.1/lib/libLLVM-13jl.so (unknown line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /gnu/store/wa28l476c1bfshfiqcqbmk75zr8ml152-llvm-13.0.1/lib/libLLVM-13jl.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /gnu/store/wa28l476c1bfshfiqcqbmk75zr8ml152-llvm-13.0.1/lib/libLLVM-13jl.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /gnu/store/wa28l476c1bfshfiqcqbmk75zr8ml152-llvm-13.0.1/lib/libLLVM-13jl.so (unknown line)
operator() at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/aotcompile.cpp:580 [inlined]
jl_dump_native_impl at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/aotcompile.cpp:592
jl_write_compiler_output at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/precompile.c:94
ijl_atexit_hook at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/init.c:207
jl_repl_entrypoint at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/jlapi.c:720
main at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/cli/loader_exe.c:59
__libc_start_call_main at /gnu/store/a19xbynxc3sg25xpkwmx7g0mdl7g31hx-glibc-2.35/lib/libc.so.6 (unknown line)
__libc_start_main at /gnu/store/a19xbynxc3sg25xpkwmx7g0mdl7g31hx-glibc-2.35/lib/libc.so.6 (unknown line)
_start at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/bin/julia (unknown line)
unknown function (ip: (nil))
Allocations: 145655328 (Pool: 145582643; Big: 72685); GC: 120
*** This error is usually fixed by running `make clean`. If the error persists, try `make cleanall`. ***
make[1]: *** [sysimage.mk:89: /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/lib/julia/sys-o.a] Error 1
make[1]: Leaving directory '/tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3'
make: *** [Makefile:88: julia-sysimg-release] Error 2
> Thumbs up for finding the solution!
>
> Ludo’.
--
Efraim Flashner <efraim <at> flashner.co.il> רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Fri, 23 Jun 2023 13:09:02 GMT)
Full text and
rfc822 format available.
Message #53 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hi,
On Fri, 23 Jun 2023 at 12:16, Efraim Flashner <efraim <at> flashner.co.il> wrote:
>> Should it be ‘target-64bit?’ instead?
>
> It should be, but julia fails the precompile stage with openblas-ilp64
Well, I picked target-x86-64? because the recipe contains,
,@(if (target-aarch64?)
`("USE_BLAS64=0")
'())
so using target-64bit? would apply USE_BLAS64=1 to aarch64 when it was
not. Well, since I know few about this architecture, I have tried to
keep the way it was. Somehow, I have no opinion about what is best. :-)
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Fri, 23 Jun 2023 13:09:02 GMT)
Full text and
rfc822 format available.
Message #56 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hi Efraim,
On Thu, 22 Jun 2023 at 23:13, Efraim Flashner <efraim <at> flashner.co.il> wrote:
> - ,@(if (target-aarch64?)
> - `("USE_BLAS64=0")
> - '())
[...]
> + ,@(if (target-64bit?)
> + `("USE_BLAS64=1"
Well, I do not know but using this change USE_BLAS64=1 will be applied
to aarch64 when it was specifically not previously. Is it expected?
> diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
> index b650b71f3b..deeffb67ae 100644
> --- a/gnu/packages/python-xyz.scm
> +++ b/gnu/packages/python-xyz.scm
> @@ -6953,19 +6953,36 @@ (define-public python-numpy
This change leads to a large rebuild. Therefore, it should go first to
a dedicated branch, no?
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Fri, 23 Jun 2023 14:05:02 GMT)
Full text and
rfc822 format available.
Message #59 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hi Efraim,
On Fri, 23 Jun 2023 at 12:16, Efraim Flashner <efraim <at> flashner.co.il> wrote:
> It should be, but julia fails the precompile stage with openblas-ilp64
>
> Precompilation complete. Summary:
> Total ─────── 1006.632176 seconds
> Generation ── 690.274180 seconds 68.5726%
> Execution ─── 316.357997 seconds 31.4274%
>
> signal (15): Terminated
> in expression starting at none:0
> unknown function (ip: 0xfffff4d81b24)
> unknown function (ip: 0xfffff4ed10bb)
> unknown function (ip: 0xfffff4eda16f)
> _ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE at /gnu/store/wa28l476c1bfshfiqcqbmk75zr8ml152-llvm-13.0.1/lib/libLLVM-13jl.so (unknown line)
> _ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /gnu/store/wa28l476c1bfshfiqcqbmk75zr8ml152-llvm-13.0.1/lib/libLLVM-13jl.so (unknown line)
> _ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /gnu/store/wa28l476c1bfshfiqcqbmk75zr8ml152-llvm-13.0.1/lib/libLLVM-13jl.so (unknown line)
> _ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /gnu/store/wa28l476c1bfshfiqcqbmk75zr8ml152-llvm-13.0.1/lib/libLLVM-13jl.so (unknown line)
> operator() at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/aotcompile.cpp:580 [inlined]
> jl_dump_native_impl at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/aotcompile.cpp:592
> jl_write_compiler_output at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/precompile.c:94
> ijl_atexit_hook at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/init.c:207
> jl_repl_entrypoint at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/jlapi.c:720
> main at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/cli/loader_exe.c:59
> __libc_start_call_main at /gnu/store/a19xbynxc3sg25xpkwmx7g0mdl7g31hx-glibc-2.35/lib/libc.so.6 (unknown line)
> __libc_start_main at /gnu/store/a19xbynxc3sg25xpkwmx7g0mdl7g31hx-glibc-2.35/lib/libc.so.6 (unknown line)
> _start at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/bin/julia (unknown line)
> unknown function (ip: (nil))
> Allocations: 145655328 (Pool: 145582643; Big: 72685); GC: 120
> *** This error is usually fixed by running `make clean`. If the error persists, try `make cleanall`. ***
> make[1]: *** [sysimage.mk:89: /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/lib/julia/sys-o.a] Error 1
> make[1]: Leaving directory '/tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3'
> make: *** [Makefile:88: julia-sysimg-release] Error 2
Using my patch, it fails with:
--8<---------------cut here---------------start------------->8---
Error in testset OpenBLAS_jll:
Test Failed at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/share/julia/stdlib/v1.8/OpenBLAS_jll/test/runtests.jl:16
Expression: dlsym(OpenBLAS_jll.libopenblas_handle, #= /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/share/julia/stdlib/v1.8/OpenBLAS_jll/test/runtests.jl:16 =# @blasfunc(openblas_set_num_threads); throw_error = false) != nothing
Evaluated: nothing != nothing
ERROR: LoadError: Test run finished with errors
in expression starting at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/test/runtests.jl:93
--8<---------------cut here---------------end--------------->8---
Well, I am still missing why the expression
dlsym(OpenBLAS_jll.libopenblas_handle,
#= /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/share/julia/stdlib/v1.8/OpenBLAS_jll/test/runtests.jl:16 =#
@blasfunc(openblas_set_num_threads); throw_error = false)
is evaluated to nothing. Any idea?
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Fri, 23 Jun 2023 14:17:02 GMT)
Full text and
rfc822 format available.
Message #62 received at 63986 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Fri, Jun 23, 2023 at 03:23:17PM +0200, Simon Tournier wrote:
> Hi Efraim,
>
> On Fri, 23 Jun 2023 at 12:16, Efraim Flashner <efraim <at> flashner.co.il> wrote:
>
> > It should be, but julia fails the precompile stage with openblas-ilp64
> >
> > Precompilation complete. Summary:
> > Total ─────── 1006.632176 seconds
> > Generation ── 690.274180 seconds 68.5726%
> > Execution ─── 316.357997 seconds 31.4274%
> >
> > signal (15): Terminated
> > in expression starting at none:0
> > unknown function (ip: 0xfffff4d81b24)
> > unknown function (ip: 0xfffff4ed10bb)
> > unknown function (ip: 0xfffff4eda16f)
> > _ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE at /gnu/store/wa28l476c1bfshfiqcqbmk75zr8ml152-llvm-13.0.1/lib/libLLVM-13jl.so (unknown line)
> > _ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /gnu/store/wa28l476c1bfshfiqcqbmk75zr8ml152-llvm-13.0.1/lib/libLLVM-13jl.so (unknown line)
> > _ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /gnu/store/wa28l476c1bfshfiqcqbmk75zr8ml152-llvm-13.0.1/lib/libLLVM-13jl.so (unknown line)
> > _ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /gnu/store/wa28l476c1bfshfiqcqbmk75zr8ml152-llvm-13.0.1/lib/libLLVM-13jl.so (unknown line)
> > operator() at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/aotcompile.cpp:580 [inlined]
> > jl_dump_native_impl at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/aotcompile.cpp:592
> > jl_write_compiler_output at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/precompile.c:94
> > ijl_atexit_hook at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/init.c:207
> > jl_repl_entrypoint at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/src/jlapi.c:720
> > main at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/cli/loader_exe.c:59
> > __libc_start_call_main at /gnu/store/a19xbynxc3sg25xpkwmx7g0mdl7g31hx-glibc-2.35/lib/libc.so.6 (unknown line)
> > __libc_start_main at /gnu/store/a19xbynxc3sg25xpkwmx7g0mdl7g31hx-glibc-2.35/lib/libc.so.6 (unknown line)
> > _start at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/bin/julia (unknown line)
> > unknown function (ip: (nil))
> > Allocations: 145655328 (Pool: 145582643; Big: 72685); GC: 120
> > *** This error is usually fixed by running `make clean`. If the error persists, try `make cleanall`. ***
> > make[1]: *** [sysimage.mk:89: /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/lib/julia/sys-o.a] Error 1
> > make[1]: Leaving directory '/tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3'
> > make: *** [Makefile:88: julia-sysimg-release] Error 2
>
> Using my patch, it fails with:
>
> --8<---------------cut here---------------start------------->8---
> Error in testset OpenBLAS_jll:
> Test Failed at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/share/julia/stdlib/v1.8/OpenBLAS_jll/test/runtests.jl:16
> Expression: dlsym(OpenBLAS_jll.libopenblas_handle, #= /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/share/julia/stdlib/v1.8/OpenBLAS_jll/test/runtests.jl:16 =# @blasfunc(openblas_set_num_threads); throw_error = false) != nothing
> Evaluated: nothing != nothing
> ERROR: LoadError: Test run finished with errors
> in expression starting at /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/test/runtests.jl:93
> --8<---------------cut here---------------end--------------->8---
Mine was on aarch64
> Well, I am still missing why the expression
>
> dlsym(OpenBLAS_jll.libopenblas_handle,
> #= /tmp/guix-build-julia-1.8.3.drv-0/julia-1.8.3/usr/share/julia/stdlib/v1.8/OpenBLAS_jll/test/runtests.jl:16 =#
> @blasfunc(openblas_set_num_threads); throw_error = false)
>
> is evaluated to nothing. Any idea?
It is something about the wrong internal interface in openblas, I don't
remember exactly what that error came from.
Try with the attached diff.
--
Efraim Flashner <efraim <at> flashner.co.il> רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[julia-openblas-ilp64.patch (text/plain, attachment)]
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Thu, 06 Jul 2023 14:14:02 GMT)
Full text and
rfc822 format available.
Message #65 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hi there!
What’s the status? Sounds like we have a couple of fixes already.
Maybe you can submit one of them to guix-patches <at> gnu.org so qa.guix can
pick it up. And if one of them is more intrusive (more rebuilds), then
submit it separately so it gets merged later? How does that sound?
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Sun, 20 Aug 2023 20:54:02 GMT)
Full text and
rfc822 format available.
Message #68 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hi!
Friendly ping. :-)
https://issues.guix.gnu.org/63986
Ludo’.
Ludovic Courtès <ludo <at> gnu.org> skribis:
> Hi there!
>
> What’s the status? Sounds like we have a couple of fixes already.
>
> Maybe you can submit one of them to guix-patches <at> gnu.org so qa.guix can
> pick it up. And if one of them is more intrusive (more rebuilds), then
> submit it separately so it gets merged later? How does that sound?
>
> Ludo’.
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Thu, 14 Sep 2023 10:34:02 GMT)
Full text and
rfc822 format available.
Message #71 received at 63986 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Sun, Aug 20, 2023 at 10:53:44PM +0200, Ludovic Courtès wrote:
> Hi!
>
> Friendly ping. :-)
>
> https://issues.guix.gnu.org/63986
>
> Ludo’.
>
> Ludovic Courtès <ludo <at> gnu.org> skribis:
>
> > Hi there!
> >
> > What’s the status? Sounds like we have a couple of fixes already.
> >
> > Maybe you can submit one of them to guix-patches <at> gnu.org so qa.guix can
> > pick it up. And if one of them is more intrusive (more rebuilds), then
> > submit it separately so it gets merged later? How does that sound?
> >
> > Ludo’.
I've attached a diff to adjust openblas64 and to use it for x86_64 in
julia. I don't know if it's faster than the current openblas.
--
Efraim Flashner <efraim <at> flashner.co.il> רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[julia-openblas-ilp64.patch (text/plain, attachment)]
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Fri, 15 Sep 2023 19:49:01 GMT)
Full text and
rfc822 format available.
Message #74 received at 63986 <at> debbugs.gnu.org (full text, mbox):
>jeu. 14 sept. 2023 at 13:33, Efraim Flashner <efraim <at> flashner.co.il> wrote:
> [[PGP Signed Part:Undecided]]
> On Sun, Aug 20, 2023 at 10:53:44PM +0200, Ludovic Courtès wrote:
>> Hi!
>>
>> Friendly ping. :-)
>>
>> https://issues.guix.gnu.org/63986
>>
>> Ludo’.
>>
>> Ludovic Courtès <ludo <at> gnu.org> skribis:
>>
>> > Hi there!
>> >
>> > What’s the status? Sounds like we have a couple of fixes already.
>> >
>> > Maybe you can submit one of them to guix-patches <at> gnu.org so qa.guix can
>> > pick it up. And if one of them is more intrusive (more rebuilds), then
>> > submit it separately so it gets merged later? How does that sound?
>> >
>> > Ludo’.
>
> I've attached a diff to adjust openblas64 and to use it for x86_64 in
> julia. I don't know if it's faster than the current openblas.
I have applied the patch in a freshly cloned guix repo, and build julia
within a shell as for the instructions under ’Contributing’.
I get the 13 ms when running the original test, so I guess the issue is
solved (other than thinking about the feasibility of performance tests
to avoid this kind of situations).
Thanks a lot !
C.
Reply sent
to
csantosb <at> inventati.org:
You have taken responsibility.
(Fri, 15 Sep 2023 19:51:02 GMT)
Full text and
rfc822 format available.
Notification sent
to
csantosb <at> inventati.org:
bug acknowledged by developer.
(Fri, 15 Sep 2023 19:51:02 GMT)
Full text and
rfc822 format available.
Message #79 received at 63986-done <at> debbugs.gnu.org (full text, mbox):
Information forwarded
to
efraim <at> flashner.co.il, csantosb <at> inventati.org, bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Sat, 16 Sep 2023 10:24:01 GMT)
Full text and
rfc822 format available.
Message #82 received at submit <at> debbugs.gnu.org (full text, mbox):
Hi,
This series is for triggering the rebuild of the Julia world. Some Julia
packages are long to build so let minimize the annoyance and make some
substitutes available before pushing to master. :-)
bug#63986: Julia is very slow
Cayetano Santos via Bug reports for GNU Guix <bug-guix <at> gnu.org>
Fri, 09 Jun 2023 23:42:00 +0200
id:87legsl3cm.fsf <at> inventati.org
https://yhetil.org/guix/87legsl3cm.fsf <at> inventati.org
https://issues.guix.gnu.org/msgid/87legsl3cm.fsf <at> inventati.org
Cheers,
simon
Efraim Flashner (2):
gnu: openblas-ilp64: Add symbol suffix.
gnu: julia: Conditionally use openblas with ILP64 support.
gnu/packages/julia.scm | 22 ++++++++++++++--------
gnu/packages/maths.scm | 4 +++-
2 files changed, 17 insertions(+), 9 deletions(-)
base-commit: eeec8371890fc95c60112da299dde78b21c948c2
--
2.38.1
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Sat, 16 Sep 2023 10:48:01 GMT)
Full text and
rfc822 format available.
Message #85 received at 63986 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi Efraim,
On Thu, 14 Sep 2023 at 13:33, Efraim Flashner <efraim <at> flashner.co.il> wrote:
> On Sun, Aug 20, 2023 at 10:53:44PM +0200, Ludovic Courtès wrote:
> I've attached a diff to adjust openblas64 and to use it for x86_64 in
> julia. I don't know if it's faster than the current openblas.
I have applied the diff and turned into two patches. Attached below.
Then, I have sent the cover-letter to guix-patches where you are CC.
The idea was to trigger the Julia world rebuild. However, I do not get
back any Debbugs ID for sending the two patches.
I think that before pushing the change to master, we should wait that QA
builds some substitutes. It will reduce the annoyance for user since
some Julia paackages are long to build.
I have no time today for waiting after some Debbugs troubles.
Cheers,
simon
[0000-cover-letter.patch (text/x-diff, inline)]
From 7f2347d8e64562e1de563cf1e4cdfd192e81a9df Mon Sep 17 00:00:00 2001
Message-Id: <cover.1694859477.git.zimon.toutoune <at> gmail.com>
From: Simon Tournier <zimon.toutoune <at> gmail.com>
X-Debbugs-CC: Efraim Flashner <efraim <at> flashner.co.il>, Cayetano Santos <csantosb <at> inventati.org>,
Date: Sat, 16 Sep 2023 12:17:57 +0200
Subject: [PATCH 0/2] Fix bug#63986: Julia is very slow
Hi,
This series is for triggering the rebuild of the Julia world. Some Julia
packages are long to build so let minimize the annoyance and make some
substitutes available before pushing to master. :-)
bug#63986: Julia is very slow
Cayetano Santos via Bug reports for GNU Guix <bug-guix <at> gnu.org>
Fri, 09 Jun 2023 23:42:00 +0200
id:87legsl3cm.fsf <at> inventati.org
https://yhetil.org/guix/87legsl3cm.fsf <at> inventati.org
https://issues.guix.gnu.org/msgid/87legsl3cm.fsf <at> inventati.org
Cheers,
simon
Efraim Flashner (2):
gnu: openblas-ilp64: Add symbol suffix.
gnu: julia: Conditionally use openblas with ILP64 support.
gnu/packages/julia.scm | 22 ++++++++++++++--------
gnu/packages/maths.scm | 4 +++-
2 files changed, 17 insertions(+), 9 deletions(-)
base-commit: eeec8371890fc95c60112da299dde78b21c948c2
--
2.38.1
[0001-gnu-openblas-ilp64-Add-symbol-suffix.patch (text/x-diff, inline)]
From 2f044415efb5dc5703bf2b09439c6f5e2fd1fd72 Mon Sep 17 00:00:00 2001
Message-Id: <2f044415efb5dc5703bf2b09439c6f5e2fd1fd72.1694859477.git.zimon.toutoune <at> gmail.com>
In-Reply-To: <cover.1694859477.git.zimon.toutoune <at> gmail.com>
References: <cover.1694859477.git.zimon.toutoune <at> gmail.com>
From: Efraim Flashner <efraim <at> flashner.co.il>
Date: Sat, 16 Sep 2023 11:59:28 +0200
Subject: [PATCH 1/2] gnu: openblas-ilp64: Add symbol suffix.
* gnu/packages/maths.scm (openblas-ilp64)[arguments]<#:make-flags>: Add the
symbol '64_' as suffix.
---
gnu/packages/maths.scm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index 6141c0988612..c051e60d5e40 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -4638,7 +4638,9 @@ (define-public openblas-ilp64
(arguments
(substitute-keyword-arguments (package-arguments openblas)
((#:make-flags flags #~'())
- #~(append (list "INTERFACE64=1" "LIBNAMESUFFIX=ilp64")
+ #~(append (list "INTERFACE64=1"
+ "SYMBOLSUFFIX=64_"
+ "LIBPREFIX=libopenblas64_")
#$flags))))
(synopsis "Optimized BLAS library based on GotoBLAS (ILP64 version)")
(license license:bsd-3)))
--
2.38.1
[0002-gnu-julia-Conditionally-use-openblas-with-ILP64-supp.patch (text/x-diff, inline)]
From 7f2347d8e64562e1de563cf1e4cdfd192e81a9df Mon Sep 17 00:00:00 2001
Message-Id: <7f2347d8e64562e1de563cf1e4cdfd192e81a9df.1694859477.git.zimon.toutoune <at> gmail.com>
In-Reply-To: <cover.1694859477.git.zimon.toutoune <at> gmail.com>
References: <cover.1694859477.git.zimon.toutoune <at> gmail.com>
From: Efraim Flashner <efraim <at> flashner.co.il>
Date: Sat, 16 Sep 2023 12:05:41 +0200
Subject: [PATCH 2/2] gnu: julia: Conditionally use openblas with ILP64
support.
Fixes <https://bugs.gnu.org/63986>.
Reported by Cayetano Santos <csantosb <at> inventati.org>.
* gnu/packages/julia.scm (julia)[arguments]<#:phases>: Conditionally use
OpenBLAS with ILP64 support for x86-64 target.
<#:make-flags>: Likewise.
[inputs]: Conditionally replace openblas by openblas-ilp64 for x86-64 target.
Co-authored-by: Simon Tournier <zimon.toutoune <at> gmail.com>
---
gnu/packages/julia.scm | 22 ++++++++++++++--------
1 file changed, 14 insertions(+), 8 deletions(-)
diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm
index ba541758223e..8ba2b480ce55 100644
--- a/gnu/packages/julia.scm
+++ b/gnu/packages/julia.scm
@@ -269,7 +269,10 @@ (define-public julia
(substitute* (jlpath "nghttp2")
(((from "libnghttp2")) (to "libnghttp2" "libnghttp2")))
(substitute* (jlpath "OpenBLAS")
- (((from "libopenblas")) (to "openblas" "libopenblas")))
+ (((from "libopenblas"))
+ ,@(if (target-x86-64?)
+ `((to "openblas" "libopenblas64_" "libopenblas"))
+ `((to "openblas" "libopenblas")))))
(substitute* (jlpath "OpenLibm")
(((from "libopenlibm")) (to "openlibm" "libopenlibm")))
(substitute* (jlpath "PCRE2")
@@ -479,12 +482,13 @@ (define-public julia
"NO_GIT=1" ; build from release tarball.
"USE_GPL_LIBS=1" ; proudly
- ,@(if (target-aarch64?)
- `("USE_BLAS64=0")
- '())
-
- "LIBBLAS=-lopenblas"
- "LIBBLASNAME=libopenblas"
+ ,@(if (target-x86-64?)
+ `("USE_BLAS64=1"
+ "LIBBLAS=-lopenblas64_"
+ "LIBBLASNAME=libopenblas64_")
+ `("USE_BLAS64=0"
+ "LIBBLAS=-lopenblas"
+ "LIBBLASNAME=libopenblas"))
(string-append "UTF8PROC_INC="
(assoc-ref %build-inputs "utf8proc")
@@ -513,7 +517,9 @@ (define-public julia
("llvm" ,llvm-julia)
("mbedtls-apache" ,mbedtls-apache)
("mpfr" ,mpfr)
- ("openblas" ,openblas)
+ ,@(if (target-x86-64?)
+ `(("openblas" ,openblas-ilp64))
+ `(("openblas" ,openblas)))
("openlibm" ,openlibm)
("p7zip" ,p7zip)
("pcre2" ,pcre2)
--
2.38.1
Did not alter fixed versions and reopened.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Sat, 16 Sep 2023 11:03:02 GMT)
Full text and
rfc822 format available.
Added tag(s) patch.
Request was from
Tobias Geerinckx-Rice <me <at> tobias.gr>
to
control <at> debbugs.gnu.org.
(Sat, 16 Sep 2023 11:04:02 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Sat, 16 Sep 2023 11:05:02 GMT)
Full text and
rfc822 format available.
Message #92 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hi Cayetano,
On Fri, 15 Sep 2023 at 19:51, help-debbugs <at> gnu.org (GNU bug Tracking System) wrote:
> #63986: Julia is very slow
>
> which was filed against the guix package, has been closed.
>
> The explanation is attached below, along with your original report.
> If you require more details, please reply to 63986 <at> debbugs.gnu.org.
Since nothing had been applied for fixing the bug, could you please keep
it open. Let close it once all is fine in master. WDYT?
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Sat, 16 Sep 2023 11:20:01 GMT)
Full text and
rfc822 format available.
Message #95 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hi,
On Sat, 16 Sep 2023 at 12:47, Simon Tournier <zimon.toutoune <at> gmail.com> wrote:
> I have applied the diff and turned into two patches. Attached below.
> Then, I have sent the cover-letter to guix-patches where you are CC.
> The idea was to trigger the Julia world rebuild. However, I do not get
> back any Debbugs ID for sending the two patches.
Now tacked by #66030. And progress here:
https://qa.guix.gnu.org/issue/66030
Cheers,
simon
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Wed, 20 Sep 2023 15:59:02 GMT)
Full text and
rfc822 format available.
Message #98 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hi Efraim,
Applying the patch is done in v3 of #66030.
https://issues.guix.gnu.org/issue/66030
and QA processed all.
https://qa.guix.gnu.org/issue/66030/
It is almost good except one strong annoyance [1]@
--8<---------------cut here---------------start------------->8---
Singular value decomposition | 57 57 5.0s
Hermitian: Error During Test at /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/rectfullpacked.jl:12
Got exception outside of a @test
could not load symbol "dsfrk_64_":
/gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3/bin/../lib/julia/liblapack.so: undefined symbol: dsfrk_64_
Stacktrace:
[1] sfrk!(transr::Char, uplo::Char, trans::Char, alpha::Float64, A::Matrix{Float64}, beta::Float64, C::Vector{Float64})
@ GenericLinearAlgebra.LAPACK2 /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/src/lapack.jl:523
[2] Ac_mul_A_RFP(A::Matrix{Float64}, uplo::Symbol)
@ GenericLinearAlgebra /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/src/rectfullpacked.jl:77
[3] macro expansion
@ /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/rectfullpacked.jl:13 [inlined]
[4] macro expansion
@ /gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3/share/julia/stdlib/v1.8/Test/src/Test.jl:1360 [inlined]
[5] macro expansion
@ /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/rectfullpacked.jl:13 [inlined]
[6] macro expansion
@ /gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3/share/julia/stdlib/v1.8/Test/src/Test.jl:1436 [inlined]
[7] macro expansion
@ /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/rectfullpacked.jl:7 [inlined]
[8] macro expansion
@ /gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3/share/julia/stdlib/v1.8/Test/src/Test.jl:1360 [inlined]
[9] top-level scope
@ /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/rectfullpacked.jl:7
[10] include(fname::String)
@ Base.MainInclude ./client.jl:476
[11] top-level scope
@ /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/runtests.jl:10
[12] include(mod::Module, _path::String)
@ Base ./Base.jl:419
[13] exec_options(opts::Base.JLOptions)
@ Base ./client.jl:303
[14] _start()
@ Base ./client.jl:522
--8<---------------cut here---------------end--------------->8---
Any idea? I have no idea and it is blocking the merge because ~20
packages are then broken. Well, if we have no idea, I will push the fix
for “Julia is slow” and we will fix later these ~20 failures. WDYT?
Cheers,
simon
1: https://bordeaux.guix.gnu.org/build/30e3aaec-dbde-423b-9554-67a64881e0cf/log
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Sun, 24 Sep 2023 08:39:02 GMT)
Full text and
rfc822 format available.
Message #101 received at 63986 <at> debbugs.gnu.org (full text, mbox):
>mer. 20 sept. 2023 at 17:57, Simon Tournier <zimon.toutoune <at> gmail.com> wrote:
> Hi Efraim,
> Any idea? I have no idea and it is blocking the merge because ~20
> packages are then broken. Well, if we have no idea, I will push the fix
> for “Julia is slow” and we will fix later these ~20 failures. WDYT?
Sounds good to me.
C.
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Mon, 25 Sep 2023 14:48:01 GMT)
Full text and
rfc822 format available.
Message #104 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hi,
Simon Tournier <zimon.toutoune <at> gmail.com> skribis:
> It is almost good except one strong annoyance [1]@
>
> Singular value decomposition | 57 57 5.0s
> Hermitian: Error During Test at /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/rectfullpacked.jl:12
> Got exception outside of a @test
> could not load symbol "dsfrk_64_":
> /gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3/bin/../lib/julia/liblapack.so: undefined symbol: dsfrk_64_
The ‘_64’ suffix is added by a patch in this very series, that adds
‘SYMBOLSUFFIX=64_’ to ‘openblas-ilp64’.
I don’t know what the rationale was for that configuration change, but
this is a good area of investigation.
Efraim, WDYT?
Thanks,
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Tue, 26 Sep 2023 09:29:02 GMT)
Full text and
rfc822 format available.
Message #107 received at 63986 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Mon, Sep 25, 2023 at 04:46:57PM +0200, Ludovic Courtès wrote:
> Hi,
>
> Simon Tournier <zimon.toutoune <at> gmail.com> skribis:
>
> > It is almost good except one strong annoyance [1]@
> >
> > Singular value decomposition | 57 57 5.0s
> > Hermitian: Error During Test at /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/rectfullpacked.jl:12
> > Got exception outside of a @test
> > could not load symbol "dsfrk_64_":
> > /gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3/bin/../lib/julia/liblapack.so: undefined symbol: dsfrk_64_
>
> The ‘_64’ suffix is added by a patch in this very series, that adds
> ‘SYMBOLSUFFIX=64_’ to ‘openblas-ilp64’.
>
> I don’t know what the rationale was for that configuration change, but
> this is a good area of investigation.
>
> Efraim, WDYT?
the SYMBOLSUFFIX change seems to be standard across other distributions,
and in fact there seem to be packages out there which rely on it
directly.
Looking at this error specifically and also the pcre2-8 errors we get
during the 'test phase for julia, it seems the suggested fix is to build
those libraries with julia. So we'd want to replace the pcre2 and the
lapack libraries in the julia sources (for pcre2 for all the
architectures, for lapack for x86_64 specifically) and let julia build
and link to them during the build phase.
--
Efraim Flashner <efraim <at> flashner.co.il> רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[signature.asc (application/pgp-signature, inline)]
Reply sent
to
Efraim Flashner <efraim <at> flashner.co.il>:
You have taken responsibility.
(Thu, 28 Sep 2023 11:43:02 GMT)
Full text and
rfc822 format available.
Notification sent
to
csantosb <at> inventati.org:
bug acknowledged by developer.
(Thu, 28 Sep 2023 11:43:02 GMT)
Full text and
rfc822 format available.
Message #112 received at 63986-done <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Wed, Sep 20, 2023 at 05:57:30PM +0200, Simon Tournier wrote:
> Hi Efraim,
>
> Applying the patch is done in v3 of #66030.
>
> https://issues.guix.gnu.org/issue/66030
>
> and QA processed all.
>
> https://qa.guix.gnu.org/issue/66030/
>
> It is almost good except one strong annoyance [1]@
>
> --8<---------------cut here---------------start------------->8---
> Singular value decomposition | 57 57 5.0s
> Hermitian: Error During Test at /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/rectfullpacked.jl:12
> Got exception outside of a @test
> could not load symbol "dsfrk_64_":
> /gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3/bin/../lib/julia/liblapack.so: undefined symbol: dsfrk_64_
> Stacktrace:
> [1] sfrk!(transr::Char, uplo::Char, trans::Char, alpha::Float64, A::Matrix{Float64}, beta::Float64, C::Vector{Float64})
> @ GenericLinearAlgebra.LAPACK2 /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/src/lapack.jl:523
> [2] Ac_mul_A_RFP(A::Matrix{Float64}, uplo::Symbol)
> @ GenericLinearAlgebra /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/src/rectfullpacked.jl:77
> [3] macro expansion
> @ /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/rectfullpacked.jl:13 [inlined]
> [4] macro expansion
> @ /gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3/share/julia/stdlib/v1.8/Test/src/Test.jl:1360 [inlined]
> [5] macro expansion
> @ /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/rectfullpacked.jl:13 [inlined]
> [6] macro expansion
> @ /gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3/share/julia/stdlib/v1.8/Test/src/Test.jl:1436 [inlined]
> [7] macro expansion
> @ /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/rectfullpacked.jl:7 [inlined]
> [8] macro expansion
> @ /gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3/share/julia/stdlib/v1.8/Test/src/Test.jl:1360 [inlined]
> [9] top-level scope
> @ /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/rectfullpacked.jl:7
> [10] include(fname::String)
> @ Base.MainInclude ./client.jl:476
> [11] top-level scope
> @ /gnu/store/zkx6p7kz3m5k5w5iy0l1d09b2n8b0ib3-julia-genericlinearalgebra-0.2.5/share/julia/loadpath/GenericLinearAlgebra/test/runtests.jl:10
> [12] include(mod::Module, _path::String)
> @ Base ./Base.jl:419
> [13] exec_options(opts::Base.JLOptions)
> @ Base ./client.jl:303
> [14] _start()
> @ Base ./client.jl:522
> --8<---------------cut here---------------end--------------->8---
>
> Any idea? I have no idea and it is blocking the merge because ~20
> packages are then broken. Well, if we have no idea, I will push the fix
> for “Julia is slow” and we will fix later these ~20 failures. WDYT?
This one was easy, I just upgraded it to 0.3.0. Then I had to adjust
julia-bandedmatrices and julia-arraylayouts and everything looks good.
Except for julia-optim.
I spent a few hours trying to get Julia to use accept '64' instead of
'64_' for SYMBOLSUFFIX and even packaged a newer version of lapack but I
wasn't able to force the issue. Ultimately any other julia packages
need to use libblastrampoline to determine whether to use openblas or
lapack.
Patches pushed, issue closed. I even tested the original reproducer from
the first email.
--
Efraim Flashner <efraim <at> flashner.co.il> רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Fri, 29 Sep 2023 08:47:02 GMT)
Full text and
rfc822 format available.
Message #115 received at 63986 <at> debbugs.gnu.org (full text, mbox):
Hi,
Efraim Flashner <efraim <at> flashner.co.il> skribis:
> On Mon, Sep 25, 2023 at 04:46:57PM +0200, Ludovic Courtès wrote:
[...]
>> > /gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3/bin/../lib/julia/liblapack.so: undefined symbol: dsfrk_64_
>>
>> The ‘_64’ suffix is added by a patch in this very series, that adds
>> ‘SYMBOLSUFFIX=64_’ to ‘openblas-ilp64’.
>>
>> I don’t know what the rationale was for that configuration change, but
>> this is a good area of investigation.
>>
>> Efraim, WDYT?
>
> the SYMBOLSUFFIX change seems to be standard across other distributions,
> and in fact there seem to be packages out there which rely on it
> directly.
>
> Looking at this error specifically and also the pcre2-8 errors we get
> during the 'test phase for julia, it seems the suggested fix is to build
> those libraries with julia. So we'd want to replace the pcre2 and the
> lapack libraries in the julia sources (for pcre2 for all the
> architectures, for lapack for x86_64 specifically) and let julia build
> and link to them during the build phase.
That would seem like a step backwards though, no? Usually we prefer to
unbundle things.
Actually, why does liblapack.so end up in Julia itself, as opposed to
just linking to libopenblas.so?
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org:
bug#63986; Package
guix.
(Fri, 29 Sep 2023 10:13:02 GMT)
Full text and
rfc822 format available.
Message #118 received at 63986 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Fri, Sep 29, 2023 at 10:46:07AM +0200, Ludovic Courtès wrote:
> Hi,
>
> Efraim Flashner <efraim <at> flashner.co.il> skribis:
>
> > On Mon, Sep 25, 2023 at 04:46:57PM +0200, Ludovic Courtès wrote:
>
> [...]
>
> >> > /gnu/store/h5mgc7ar7a05f9rwrd1makhzays5wd3s-julia-1.8.3/bin/../lib/julia/liblapack.so: undefined symbol: dsfrk_64_
> >>
> >> The ‘_64’ suffix is added by a patch in this very series, that adds
> >> ‘SYMBOLSUFFIX=64_’ to ‘openblas-ilp64’.
> >>
> >> I don’t know what the rationale was for that configuration change, but
> >> this is a good area of investigation.
> >>
> >> Efraim, WDYT?
> >
> > the SYMBOLSUFFIX change seems to be standard across other distributions,
> > and in fact there seem to be packages out there which rely on it
> > directly.
> >
> > Looking at this error specifically and also the pcre2-8 errors we get
> > during the 'test phase for julia, it seems the suggested fix is to build
> > those libraries with julia. So we'd want to replace the pcre2 and the
> > lapack libraries in the julia sources (for pcre2 for all the
> > architectures, for lapack for x86_64 specifically) and let julia build
> > and link to them during the build phase.
>
> That would seem like a step backwards though, no? Usually we prefer to
> unbundle things.
I tried it with inserting our source for pcre2 and adding a patch to fix
the configure phase of pcre2, we ended up with other failures, so I
think we're best off with what we have now.
> Actually, why does liblapack.so end up in Julia itself, as opposed to
> just linking to libopenblas.so?
Julia links to both (open)blas and to lapack. If we build openblas so
that it also provides lapack, or use (c)blas from lapack, then we can
use only one or the other. I suppose in theory it should be possible to
tell julia that libopenblas64_.so is really lapack and to use that for
both of them. I don't know how well that would work though.
--
Efraim Flashner <efraim <at> flashner.co.il> רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[signature.asc (application/pgp-signature, inline)]
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Fri, 27 Oct 2023 11:24:08 GMT)
Full text and
rfc822 format available.
This bug report was last modified 195 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.