GNU bug report logs - #30579
Go 1.10

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 30579 in the body.
You can then email your comments to 30579 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: guix-patches <at> gnu.org
Subject: Go 1.10
Date: Thu, 22 Feb 2018 13:25:03 -0500

[Message part 1 (text/plain, inline)]

This patch adds Go 1.10 but leaves Go 1.9 as the default Go.

Go 1.10 changes the way that Go decides whether to re-use built objects
or to recompile them [0]. Reading the release notes, it seems to
implement something like a memoized cache for built objects and test
results.

Go 1.10 works with our go-build-system, but the built objects are not
re-used. Instead, either most or all required objects are rebuilt, so
the build system will need some adjustment.

I think it's useful to have this new version of Go packaged, especially
if somebody wants to decipher how the new compiler decides when to
re-use the built objects :)

Also, there is a data race found in the test suite when bootstrapping
with GCC 5, so this patch changes the entire Go stack to GCC 6. I'll
send a report upstream.

[0] The "build cache" is mentioned in the link below. The mechanism is
called "content-based staleness" in some places.
https://golang.org/doc/go1.10#build

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 30579 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: 30579 <at> debbugs.gnu.org
Subject: [PATCH] gnu: Add go <at> 1.10.
Date: Thu, 22 Feb 2018 13:35:22 -0500

Building Go 1.10 with the Go 1.4 bootstrap, Thread Sanitizer from GCC 5
finds a data race during the the test suite of Go 1.10. With GCC 6, the
race doesn't seem to be present.

.##### ../misc/cgo/testsanitizers
--- FAIL: TestShared (0.03s)
    --- FAIL: TestShared/tsan_shared (2.54s)
        cshared_test.go:71: `/tmp/guix-build-go-1.10.drv-0/TestShared367937240/tsan_shared` exited with exit status 66
                ==================
                WARNING: ThreadSanitizer: data race (pid=28778)
                  Write of size 8 at 0x7ffff5495c30 by thread T1:
                    #0 pthread_attr_getstacksize <null> (tsan_shared+0x00000041be82)
                    #1 x_cgo_init <null> (libtsan_shared.so+0x00000008aeaf)
                    #2 runtime.rt0_go /tmp/guix-build-go-1.10.drv-0/go/src/runtime/asm_amd64.s:199 (libtsan_shared.so+0x000000081586)

                  Previous write of size 8 at 0x7ffff5495c30 by main thread:
                    #0 <null> <null> (0x000000000001)

                  Location is stack of thread T1.

                  Thread T1 (tid=28786, running) created by main thread at:
                    #0 pthread_create <null> (tsan_shared+0x00000040c794)
                    #1 _cgo_try_pthread_create <null> (libtsan_shared.so+0x00000008ad70)
                    #2 x_cgo_sys_thread_create <null> (libtsan_shared.so+0x00000008ab12)
                    #3 _rt0_amd64_lib /tmp/guix-build-go-1.10.drv-0/go/src/runtime/asm_amd64.s:56 (libtsan_shared.so+0x0000000813b6)

                SUMMARY: ThreadSanitizer: data race ??:0 __interceptor_pthread_attr_getstacksize

* gnu/packages/golang.scm (go-1.10): New variable.
(go-1.4)[native-inputs]: Use gcc:lib version 6.
---
 gnu/packages/golang.scm | 115 +++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 114 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index f11cc76d9..a542eaba8 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -187,7 +187,7 @@
     (inputs
      `(("tzdata" ,tzdata)
        ("pcre" ,pcre)
-       ("gcc:lib" ,gcc "lib")))
+       ("gcc:lib" ,gcc-6 "lib")))
     (native-inputs
      `(("pkg-config" ,%pkg-config)
        ("which" ,which)
@@ -372,6 +372,119 @@ in the style of communicating sequential processes (@dfn{CSP}).")
        ,@(package-native-inputs go-1.4)))
     (supported-systems %supported-systems)))
 
+(define-public go-1.10
+  (package
+    (inherit go-1.9)
+    (name "go")
+    (version "1.10")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append "https://storage.googleapis.com/golang/"
+                           name version ".src.tar.gz"))
+       (sha256
+        (base32
+         "093z1h0gmi458kv7smpx0ph6jm7ss9mzxa432kysbz85jhl4kppk"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments go-1.9)
+       ((#:phases phases)
+        `(modify-phases ,phases
+           (replace 'prebuild
+             (lambda* (#:key inputs outputs #:allow-other-keys)
+               (let* ((gcclib (string-append (assoc-ref inputs "gcc:lib") "/lib"))
+                      (ld (string-append (assoc-ref inputs "libc") "/lib"))
+                      (loader (car (find-files ld "^ld-linux.+")))
+                      (net-base (assoc-ref inputs "net-base"))
+                      (tzdata-path
+                       (string-append (assoc-ref inputs "tzdata") "/share/zoneinfo"))
+                      (output (assoc-ref outputs "out")))
+
+                 ;; Removing net/ tests, which fail when attempting to access
+                 ;; network resources not present in the build container.
+                 (for-each delete-file
+                           '("net/listen_test.go"
+                             "net/parse_test.go"
+                             "net/cgo_unix_test.go"))
+
+                 (substitute* "os/os_test.go"
+                   (("/usr/bin") (getcwd))
+                   (("/bin/pwd") (which "pwd"))
+                   (("/bin/sh") (which "sh")))
+
+                 ;; Add libgcc to runpath
+                 (substitute* "cmd/link/internal/ld/lib.go"
+                   (("!rpath.set") "true"))
+                 (substitute* "cmd/go/internal/work/gccgo.go"
+                   (("cgoldflags := \\[\\]string\\{\\}")
+                    (string-append "cgoldflags := []string{"
+                                   "\"-rpath=" gcclib "\""
+                                   "}"))
+                   (("\"-lgcc_s\", ")
+                    (string-append
+                     "\"-Wl,-rpath=" gcclib "\", \"-lgcc_s\", ")))
+                 (substitute* "cmd/go/internal/work/gc.go"
+                   (("ldflags = setextld\\(ldflags, compiler\\)")
+                    (string-append
+                     "ldflags = setextld(ldflags, compiler)\n"
+                     "ldflags = append(ldflags, \"-r\")\n"
+                     "ldflags = append(ldflags, \"" gcclib "\")\n")))
+
+                 ;; Disable failing tests: these tests attempt to access
+                 ;; commands or network resources which are neither available
+                 ;; nor necessary for the build to succeed.
+                 (for-each
+                  (match-lambda
+                    ((file regex)
+                     (substitute* file
+                       ((regex all before test_name)
+                        (string-append before "Disabled" test_name)))))
+                  '(("net/net_test.go" "(.+)(TestShutdownUnix.+)")
+                    ("net/dial_test.go" "(.+)(TestDialTimeout.+)")
+                    ("os/os_test.go" "(.+)(TestHostname.+)")
+                    ("time/format_test.go" "(.+)(TestParseInSydney.+)")
+                    ("time/format_test.go" "(.+)(TestParseInLocation.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestEcho.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestCommandRelativeName.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestCatStdin.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestCatGoodAndBadFile.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestExitStatus.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestPipes.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestStdinClose.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestIgnorePipeErrorOnSuccess.+)")
+                    ("syscall/syscall_unix_test.go" "(.+)(TestPassFD\\(.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestExtraFiles/areturn.+)")
+                    ("cmd/go/go_test.go" "(.+)(TestCoverageWithCgo.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestOutputStderrCapture.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestExtraFiles.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestExtraFilesRace.+)")
+                    ("net/lookup_test.go" "(.+)(TestLookupPort.+)")
+                    ("syscall/exec_linux_test.go"
+                     "(.+)(TestCloneNEWUSERAndRemapNoRootDisableSetgroups.+)")))
+
+                 ;; fix shebang for testar script
+                 ;; note the target script is generated at build time.
+                 (substitute* "../misc/cgo/testcarchive/carchive_test.go"
+                   (("#!/usr/bin/env") (string-append "#!" (which "env"))))
+
+                 (substitute* "net/lookup_unix.go"
+                   (("/etc/protocols") (string-append net-base "/etc/protocols")))
+                 (substitute* "net/port_unix.go"
+                   (("/etc/services") (string-append net-base "/etc/services")))
+                 (substitute* "time/zoneinfo_unix.go"
+                   (("/usr/share/zoneinfo/") tzdata-path))
+                 (substitute* (find-files "cmd" "\\.go")
+                   (("/lib(64)?/ld-linux.*\\.so\\.[0-9]") loader))
+                 #t)))
+           (replace 'set-bootstrap-variables
+             (lambda* (#:key outputs inputs #:allow-other-keys)
+               ;; Tell the build system where to find the bootstrap Go.
+               (let ((go  (assoc-ref inputs "go")))
+                 (setenv "GOROOT_BOOTSTRAP" go)
+                 (setenv "GOGC" "400")
+                 ;; Go 1.10 tries to write to $HOME in a test
+                 (setenv "HOME" "/tmp")
+                 #t)))))))))
+
 (define-public go go-1.9)
 
 (define-public go-github-com-alsm-ioprogress
-- 
2.16.2

Message #11 received at 30579 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: 30579 <at> debbugs.gnu.org
Subject: Re: Go 1.10
Date: Thu, 22 Feb 2018 14:00:47 -0500

[Message part 1 (text/plain, inline)]

On Thu, Feb 22, 2018 at 01:25:03PM -0500, Leo Famulari wrote:
> Also, there is a data race found in the test suite when bootstrapping
> with GCC 5, so this patch changes the entire Go stack to GCC 6. I'll
> send a report upstream.

Upstream bug report:

https://github.com/golang/go/issues/24046

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 30579 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Leo Famulari <leo <at> famulari.name>, 30579 <at> debbugs.gnu.org
Subject: Re: [bug#30579] [PATCH] gnu: Add go <at> 1.10.
Date: Mon, 26 Feb 2018 02:14:12 +0100

[Message part 1 (text/plain, inline)]

Leo Famulari <leo <at> famulari.name> writes:

> Building Go 1.10 with the Go 1.4 bootstrap, Thread Sanitizer from GCC 5
> finds a data race during the the test suite of Go 1.10. With GCC 6, the
> race doesn't seem to be present.

Can you add this comment next to the GCC input?  That way it's obvious
for people wondering about it, without having to grep the commit log.

Other than that LGTM.

>
> .##### ../misc/cgo/testsanitizers
> --- FAIL: TestShared (0.03s)
>     --- FAIL: TestShared/tsan_shared (2.54s)
>         cshared_test.go:71: `/tmp/guix-build-go-1.10.drv-0/TestShared367937240/tsan_shared` exited with exit status 66
>                 ==================
>                 WARNING: ThreadSanitizer: data race (pid=28778)
>                   Write of size 8 at 0x7ffff5495c30 by thread T1:
>                     #0 pthread_attr_getstacksize <null> (tsan_shared+0x00000041be82)
>                     #1 x_cgo_init <null> (libtsan_shared.so+0x00000008aeaf)
>                     #2 runtime.rt0_go /tmp/guix-build-go-1.10.drv-0/go/src/runtime/asm_amd64.s:199 (libtsan_shared.so+0x000000081586)
>
>                   Previous write of size 8 at 0x7ffff5495c30 by main thread:
>                     #0 <null> <null> (0x000000000001)
>
>                   Location is stack of thread T1.
>
>                   Thread T1 (tid=28786, running) created by main thread at:
>                     #0 pthread_create <null> (tsan_shared+0x00000040c794)
>                     #1 _cgo_try_pthread_create <null> (libtsan_shared.so+0x00000008ad70)
>                     #2 x_cgo_sys_thread_create <null> (libtsan_shared.so+0x00000008ab12)
>                     #3 _rt0_amd64_lib /tmp/guix-build-go-1.10.drv-0/go/src/runtime/asm_amd64.s:56 (libtsan_shared.so+0x0000000813b6)
>
>                 SUMMARY: ThreadSanitizer: data race ??:0 __interceptor_pthread_attr_getstacksize
>
> * gnu/packages/golang.scm (go-1.10): New variable.
> (go-1.4)[native-inputs]: Use gcc:lib version 6.
> ---
>  gnu/packages/golang.scm | 115 +++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 114 insertions(+), 1 deletion(-)
>
> diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
> index f11cc76d9..a542eaba8 100644
> --- a/gnu/packages/golang.scm
> +++ b/gnu/packages/golang.scm
> @@ -187,7 +187,7 @@
>      (inputs
>       `(("tzdata" ,tzdata)
>         ("pcre" ,pcre)
> -       ("gcc:lib" ,gcc "lib")))
> +       ("gcc:lib" ,gcc-6 "lib")))
>      (native-inputs
>       `(("pkg-config" ,%pkg-config)
>         ("which" ,which)
> @@ -372,6 +372,119 @@ in the style of communicating sequential processes (@dfn{CSP}).")
>         ,@(package-native-inputs go-1.4)))
>      (supported-systems %supported-systems)))
>  
> +(define-public go-1.10
> +  (package
> +    (inherit go-1.9)
> +    (name "go")
> +    (version "1.10")
> +    (source
> +     (origin
> +       (method url-fetch)
> +       (uri (string-append "https://storage.googleapis.com/golang/"
> +                           name version ".src.tar.gz"))
> +       (sha256
> +        (base32
> +         "093z1h0gmi458kv7smpx0ph6jm7ss9mzxa432kysbz85jhl4kppk"))))
> +    (arguments
> +     (substitute-keyword-arguments (package-arguments go-1.9)
> +       ((#:phases phases)
> +        `(modify-phases ,phases
> +           (replace 'prebuild
> +             (lambda* (#:key inputs outputs #:allow-other-keys)
> +               (let* ((gcclib (string-append (assoc-ref inputs "gcc:lib") "/lib"))
> +                      (ld (string-append (assoc-ref inputs "libc") "/lib"))
> +                      (loader (car (find-files ld "^ld-linux.+")))
> +                      (net-base (assoc-ref inputs "net-base"))
> +                      (tzdata-path
> +                       (string-append (assoc-ref inputs "tzdata") "/share/zoneinfo"))
> +                      (output (assoc-ref outputs "out")))
> +
> +                 ;; Removing net/ tests, which fail when attempting to access
> +                 ;; network resources not present in the build container.
> +                 (for-each delete-file
> +                           '("net/listen_test.go"
> +                             "net/parse_test.go"
> +                             "net/cgo_unix_test.go"))
> +
> +                 (substitute* "os/os_test.go"
> +                   (("/usr/bin") (getcwd))
> +                   (("/bin/pwd") (which "pwd"))
> +                   (("/bin/sh") (which "sh")))
> +
> +                 ;; Add libgcc to runpath
> +                 (substitute* "cmd/link/internal/ld/lib.go"
> +                   (("!rpath.set") "true"))
> +                 (substitute* "cmd/go/internal/work/gccgo.go"
> +                   (("cgoldflags := \\[\\]string\\{\\}")
> +                    (string-append "cgoldflags := []string{"
> +                                   "\"-rpath=" gcclib "\""
> +                                   "}"))
> +                   (("\"-lgcc_s\", ")
> +                    (string-append
> +                     "\"-Wl,-rpath=" gcclib "\", \"-lgcc_s\", ")))
> +                 (substitute* "cmd/go/internal/work/gc.go"
> +                   (("ldflags = setextld\\(ldflags, compiler\\)")
> +                    (string-append
> +                     "ldflags = setextld(ldflags, compiler)\n"
> +                     "ldflags = append(ldflags, \"-r\")\n"
> +                     "ldflags = append(ldflags, \"" gcclib "\")\n")))
> +
> +                 ;; Disable failing tests: these tests attempt to access
> +                 ;; commands or network resources which are neither available
> +                 ;; nor necessary for the build to succeed.
> +                 (for-each
> +                  (match-lambda
> +                    ((file regex)
> +                     (substitute* file
> +                       ((regex all before test_name)
> +                        (string-append before "Disabled" test_name)))))
> +                  '(("net/net_test.go" "(.+)(TestShutdownUnix.+)")
> +                    ("net/dial_test.go" "(.+)(TestDialTimeout.+)")
> +                    ("os/os_test.go" "(.+)(TestHostname.+)")
> +                    ("time/format_test.go" "(.+)(TestParseInSydney.+)")
> +                    ("time/format_test.go" "(.+)(TestParseInLocation.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestEcho.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestCommandRelativeName.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestCatStdin.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestCatGoodAndBadFile.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestExitStatus.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestPipes.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestStdinClose.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestIgnorePipeErrorOnSuccess.+)")
> +                    ("syscall/syscall_unix_test.go" "(.+)(TestPassFD\\(.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestExtraFiles/areturn.+)")
> +                    ("cmd/go/go_test.go" "(.+)(TestCoverageWithCgo.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestOutputStderrCapture.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestExtraFiles.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestExtraFilesRace.+)")
> +                    ("net/lookup_test.go" "(.+)(TestLookupPort.+)")
> +                    ("syscall/exec_linux_test.go"
> +                     "(.+)(TestCloneNEWUSERAndRemapNoRootDisableSetgroups.+)")))
> +
> +                 ;; fix shebang for testar script
> +                 ;; note the target script is generated at build time.
> +                 (substitute* "../misc/cgo/testcarchive/carchive_test.go"
> +                   (("#!/usr/bin/env") (string-append "#!" (which "env"))))
> +
> +                 (substitute* "net/lookup_unix.go"
> +                   (("/etc/protocols") (string-append net-base "/etc/protocols")))
> +                 (substitute* "net/port_unix.go"
> +                   (("/etc/services") (string-append net-base "/etc/services")))
> +                 (substitute* "time/zoneinfo_unix.go"
> +                   (("/usr/share/zoneinfo/") tzdata-path))
> +                 (substitute* (find-files "cmd" "\\.go")
> +                   (("/lib(64)?/ld-linux.*\\.so\\.[0-9]") loader))
> +                 #t)))
> +           (replace 'set-bootstrap-variables
> +             (lambda* (#:key outputs inputs #:allow-other-keys)
> +               ;; Tell the build system where to find the bootstrap Go.
> +               (let ((go  (assoc-ref inputs "go")))
> +                 (setenv "GOROOT_BOOTSTRAP" go)
> +                 (setenv "GOGC" "400")
> +                 ;; Go 1.10 tries to write to $HOME in a test
> +                 (setenv "HOME" "/tmp")
> +                 #t)))))))))
> +
>  (define-public go go-1.9)
>  
>  (define-public go-github-com-alsm-ioprogress
> -- 
> 2.16.2

[signature.asc (application/pgp-signature, inline)]

Message #17 received at 30579 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Leo Famulari <leo <at> famulari.name>, 30579 <at> debbugs.gnu.org
Subject: Re: [bug#30579] Go 1.10
Date: Mon, 26 Feb 2018 02:15:19 +0100

[Message part 1 (text/plain, inline)]

Leo Famulari <leo <at> famulari.name> writes:

> This patch adds Go 1.10 but leaves Go 1.9 as the default Go.
>
> Go 1.10 changes the way that Go decides whether to re-use built objects
> or to recompile them [0]. Reading the release notes, it seems to
> implement something like a memoized cache for built objects and test
> results.
>
> Go 1.10 works with our go-build-system, but the built objects are not
> re-used. Instead, either most or all required objects are rebuilt, so
> the build system will need some adjustment.
>
> I think it's useful to have this new version of Go packaged, especially
> if somebody wants to decipher how the new compiler decides when to
> re-use the built objects :)

Definitely.  Thanks for working on it!  :-)

>
> Also, there is a data race found in the test suite when bootstrapping
> with GCC 5, so this patch changes the entire Go stack to GCC 6. I'll
> send a report upstream.
>
> [0] The "build cache" is mentioned in the link below. The mechanism is
> called "content-based staleness" in some places.
> https://golang.org/doc/go1.10#build

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.