Package: guix;
Reported by: Ruud van Asseldonk <dev+guix <at> veniogames.com>
Date: Sun, 10 Dec 2017 23:09:02 UTC
Severity: normal
Done: ludo <at> gnu.org (Ludovic Courtès)
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 29654 in the body.
You can then email your comments to 29654 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
View this report as an mbox folder, status mbox, maintainer mbox
bug-guix <at> gnu.org:bug#29654; Package guix.
(Sun, 10 Dec 2017 23:09:02 GMT) Full text and rfc822 format available.Ruud van Asseldonk <dev+guix <at> veniogames.com>:bug-guix <at> gnu.org.
(Sun, 10 Dec 2017 23:09:02 GMT) Full text and rfc822 format available.Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
From: Ruud van Asseldonk <dev+guix <at> veniogames.com> To: bug-guix <at> gnu.org Subject: Manual database index.db embeds timestamps Date: Sun, 10 Dec 2017 23:47:13 +0100
The manual database file index.db embeds mtimes of the files it refers to, making it not reproducible. This is an issue for building reproducible profiles (as produced by `guix pack` for example). The number that are not reproducible can be seen with `accessdb index.db`, which will output something like $version$ -> "2.5.0" acme-client -> "- 1 1 1512941854 498178709 B - - gz secure ACME client" And when built again on a clean installation: $version$ -> "2.5.0" acme-client -> "- 1 1 1512942998 296814690 B - - gz secure ACME client" I asked the man-db maintainer about this, who replied: > Those are timestamps, seconds and nanoseconds respectively. You should > get reproducible output if you make sure the mtimes of manual pages are > reproducible. (man-db needs to keep track of mtimes so that it knows > when a database is out of date.) If the manual pages come straight from > the source package, this should be straightforward; if they're > generated, you'll need some strategy to ensure that they're generated > with a reproducible mtime. Now the strange thing is, when I check the profile directory, both the man file symlinks, and the files they refer to, have the mtime set to epoch. I suspected that the mtime would be reset after the manual database had already been built, but inserting calls to utime after `(symlink manpage dest-file)` in the `manual-database` procedure, did not resolve the issue. The mtime that ends up in index.db does correspond to a recent time, and for larger databases the timestamps can differ among entries, but so far I have only seen them differ in the nanoseconds. The seconds timestamp looks like the timestamp at which the database was generated. So either the mtime of the input files is not epoch when man-db runs, or man-db is somehow setting the mtime of every entry to the current time as it adds them. I looked at the man-db source and I did not find anything that looked like it was storing current times as mtimes in the database, although I did not look very carefully. I suspect that the mtimes of the files to scan are actually wrong. It should be possible to confirm this by introducing a delay when creating the man files, and when making the symlinks, and seeing if the delay shows up in the database. Help with diagnosing this further, or ideas about what could be going on and how to resolve it, would be appreciated. I am willing to work on a fix, but I am not very familiar with Guix yet. Kind regards, Ruud van Asseldonk
bug-guix <at> gnu.org:bug#29654; Package guix.
(Fri, 15 Dec 2017 21:31:02 GMT) Full text and rfc822 format available.Message #8 received at 29654 <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Ruud van Asseldonk <dev+guix <at> veniogames.com> Cc: 29654 <at> debbugs.gnu.org Subject: Re: bug#29654: Manual database index.db embeds timestamps Date: Fri, 15 Dec 2017 22:30:13 +0100
[Message part 1 (text/plain, inline)]
Hi Ruud, Ruud van Asseldonk <dev+guix <at> veniogames.com> skribis: > The manual database file index.db embeds mtimes of the files it refers > to, making it not reproducible. This is an issue for building > reproducible profiles (as produced by `guix pack` for example). > > The number that are not reproducible can be seen with `accessdb > index.db`, which will output something like > > $version$ -> "2.5.0" > acme-client -> "- 1 1 1512941854 498178709 B - - gz secure ACME client" > > And when built again on a clean installation: > > $version$ -> "2.5.0" > acme-client -> "- 1 1 1512942998 296814690 B - - gz secure ACME client" Good catch. I was already motivated to create the database directly from Scheme (‘man-db’ is quite slow), so that gave me an additional excuse. ;-) The attached patch does that. The timestamps are always set to zero. You can check for reproducibility by doing: guix package -p foo -i bar baz guix build --check /gnu/store/…-manual-database.drv where the .drv is the one shown in the ‘guix package’ output. Unfortunately, this is not fully deterministic: when running --check several times in a row, I occasionally get different results. I suspect GDBM’s output is not fully deterministic. Thoughts? Ludo’.
[0001-gnu-guile-gdbm-ffi-Default-to-Guile-2.2.patch (text/x-patch, inline)]
From 9da89cbdda484c106ff2706f09089d87bdf88a45 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo <at> gnu.org>
Date: Fri, 15 Dec 2017 22:08:34 +0100
Subject: [PATCH 1/2] gnu: guile-gdbm-ffi: Default to Guile 2.2.
* gnu/packages/guile.scm (guile-gdbm-ffi)[inputs]: Switch to GUILE-2.2.
(guile2.0-gdbm-ffi, guile2.2-gdbm-ffi): New variables.
---
gnu/packages/guile.scm | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
index c0fda71ea..887e360a3 100644
--- a/gnu/packages/guile.scm
+++ b/gnu/packages/guile.scm
@@ -1097,7 +1097,7 @@ inspired by the SCSH regular expression system.")
;; compile to the destination
(compile-file gdbm.scm-dest gdbm.go-dest)))))
(inputs
- `(("guile" ,guile-2.0)))
+ `(("guile" ,guile-2.2)))
(propagated-inputs
`(("gdbm" ,gdbm)))
(home-page "https://github.com/ijp/guile-gdbm")
@@ -1107,8 +1107,11 @@ inspired by the SCSH regular expression system.")
Guile's foreign function interface.")
(license license:gpl3+)))
+(define-public guile2.0-gdbm-ffi
+ (package-for-guile-2.0 guile-gdbm-ffi))
+
(define-public guile2.2-gdbm-ffi
- (package-for-guile-2.2 guile-gdbm-ffi))
+ (deprecated-package "guile2.2-gdbm-ffi" guile-gdbm-ffi))
(define-public guile-sqlite3
(let ((commit "607721fe1174a299e45d457acacf94eefb964071"))
--
2.15.1
[0002-DRAFT-profiles-Use-guix-man-db-to-create-the-manual-.patch (text/x-patch, inline)]
From d37d0640e073ea18a07254e93f75940fc0f74f74 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo <at> gnu.org>
Date: Fri, 15 Dec 2017 22:16:18 +0100
Subject: [PATCH 2/2] DRAFT profiles: Use (guix man-db) to create the manual
database.
DRAFT: Database is not entirely bit-reproducible.
* guix/man-db.scm: New file.
* Makefile.am (MODULES_NOT_COMPILED): Add it.
* guix/profiles.scm (manual-database): Rewrite to use (guix man-db).
---
Makefile.am | 3 +-
guix/man-db.scm | 184 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
guix/profiles.scm | 104 +++++++++++++-----------------
3 files changed, 230 insertions(+), 61 deletions(-)
create mode 100644 guix/man-db.scm
diff --git a/Makefile.am b/Makefile.am
index 85b9ab36d..fe1e685f3 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -34,7 +34,8 @@ nodist_noinst_SCRIPTS = \
# Modules that are not compiled but are installed nonetheless, such as
# build-side modules with unusual dependencies.
-MODULES_NOT_COMPILED =
+MODULES_NOT_COMPILED = \
+ guix/man-db.scm
include gnu/local.mk
diff --git a/guix/man-db.scm b/guix/man-db.scm
new file mode 100644
index 000000000..b42558b06
--- /dev/null
+++ b/guix/man-db.scm
@@ -0,0 +1,184 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2017 Ludovic Courtès <ludo <at> gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix man-db)
+ #:use-module (guix zlib)
+ #:use-module ((guix build utils) #:select (find-files))
+ #:use-module (srfi srfi-9)
+ #:use-module (srfi srfi-26)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 rdelim)
+ #:use-module (ice-9 regex)
+ #:export (mandb-entry?
+ mandb-entry-file-name
+ mandb-entry-name
+ mandb-entry-section
+ mandb-entry-synopsis
+
+ mandb-entries
+ write-mandb-database))
+
+;;; Comment:
+;;;
+;;; Scan gzipped man pages and create a man-db database. The database is
+;;; meant to be used by 'man -k KEYWORD'.
+;;;
+;;; The implementation here aims to be simpler than that of 'man-db', and to
+;;; produce deterministic output. See <https://bugs.gnu.org/29654>.
+;;;
+;;; Code:
+
+;; Load 'gdbm-ffi' at run time to simplify the job of 'imported-modules' & co.
+(module-use! (current-module) (resolve-interface '(gdbm)))
+
+(define-record-type <mandb-entry>
+ (mandb-entry file-name name section synopsis)
+ mandb-entry?
+ (file-name mandb-entry-file-name) ;e.g., "../abiword.1.gz"
+ (name mandb-entry-name) ;e.g., "ABIWORD"
+ (section mandb-entry-section) ;number
+ (synopsis mandb-entry-synopsis)) ;string
+
+(define (mandb-entry<? entry1 entry2)
+ (match entry1
+ (($ <mandb-entry> file1 name1 section1)
+ (match entry2
+ (($ <mandb-entry> file2 name2 section2)
+ (or (< section1 section2)
+ (string<? (basename file1) (basename file2))))))))
+
+(define abbreviate-file-name
+ (let ((man-file-rx (make-regexp "(.+)\\.[0-9][a-z]?(\\.gz)?$")))
+ (lambda (file)
+ (match (regexp-exec man-file-rx (basename file))
+ (#f
+ (basename file))
+ (matches
+ (match:substring matches 1))))))
+
+(define (entry->string entry)
+ "Return the wire format for ENTRY as a string."
+ (match entry
+ (($ <mandb-entry> file name section synopsis)
+ (string-append (abbreviate-file-name file) "\t"
+ (number->string section) "\t"
+ (number->string section)
+
+ ;; Timestamps, that we always set to the epoch.
+ "\t0\t0"
+
+ ;; XXX: Weird things.
+ "\tB\t-\t-\tgz\t"
+
+ synopsis "\x00"))))
+
+;; The man-db schema version we're compatible with.
+(define %version-key "$version$\x00")
+(define %version-value "2.5.0\x00")
+
+(define (write-mandb-database file entries)
+ "Write ENTRIES to FILE as a man-db database. FILE is usually
+\".../index.db\", and is a GDBM database."
+ (let ((db (gdbm-open file GDBM_WRCREAT)))
+ (gdbm-set! db %version-key %version-value)
+
+ ;; Write ENTRIES in sorted order so we get deterministic output.
+ (for-each (lambda (entry)
+ (gdbm-set! db
+ (string-append (mandb-entry-file-name entry)
+ "\x00")
+ (entry->string entry)))
+ (sort entries mandb-entry<?))
+ (gdbm-close db)))
+
+(define (read-synopsis port)
+ "Read from PORT a man page synopsis."
+ (define (section? line)
+ ;; True if LINE starts with ".SH", ".PP", or so.
+ (string-prefix? "." (string-trim line)))
+
+ (define (extract-synopsis str)
+ (match (string-contains str "\\-")
+ (#f "")
+ (index
+ (string-map (match-lambda
+ (#\newline #\space)
+ (chr chr))
+ (string-trim-both (string-drop str (+ 2 index)))))))
+
+ ;; Synopses look like "Command \- Do something.", possibly spanning several
+ ;; lines.
+ (let loop ((lines '()))
+ (match (read-line port 'concat)
+ ((? eof-object?)
+ (extract-synopsis (string-concatenate-reverse lines)))
+ ((? section?)
+ (extract-synopsis (string-concatenate-reverse lines)))
+ (line
+ (loop (cons line lines))))))
+
+(define* (man-page->entry file #:optional (resolve identity))
+ "Parse FILE, a gzipped man page, and return a <mandb-entry> for it."
+ (define (string->number* str)
+ (if (and (string-prefix? "\"" str)
+ (> (string-length str) 1)
+ (string-suffix? "\"" str))
+ (string->number (string-drop (string-drop-right str 1) 1))
+ (string->number str)))
+
+ (call-with-gzip-input-port (open-file file "r0")
+ (lambda (port)
+ (let loop ((name #f)
+ (section #f)
+ (synopsis #f))
+ (if (and name section synopsis)
+ (mandb-entry file name section synopsis)
+ (let ((line (read-line port)))
+ (if (eof-object? line)
+ (mandb-entry file name (or section 0) (or synopsis ""))
+ (match (string-tokenize line)
+ ((".TH" name (= string->number* section) _ ...)
+ (loop name section synopsis))
+ ((".SH" (or "NAME" "\"NAME\""))
+ (loop name section (read-synopsis port)))
+ ((".so" link)
+ (match (and=> (resolve link)
+ (cut man-page->entry <> resolve))
+ (#f
+ (loop name section synopsis))
+ (alias
+ (mandb-entry file
+ (mandb-entry-name alias)
+ (mandb-entry-section alias)
+ (mandb-entry-synopsis alias)))))
+ (_
+ (loop name section synopsis))))))))))
+
+(define (man-files directory)
+ "Return the list of man pages found under DIRECTORY, recursively."
+ (find-files directory "\\.[0-9][a-z]?(\\.gz)?$"))
+
+(define (mandb-entries directory)
+ "Return mandb entries for the man pages found under DIRECTORY, recursively."
+ (map (lambda (file)
+ (man-page->entry file
+ (lambda (link)
+ (let ((file (string-append directory "/" link
+ ".gz")))
+ (and (file-exists? file) file)))))
+ (man-files directory)))
diff --git a/guix/profiles.scm b/guix/profiles.scm
index cedf9faa8..962b9074b 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -33,6 +33,7 @@
#:use-module (guix derivations)
#:use-module (guix search-paths)
#:use-module (guix gexp)
+ #:use-module (guix modules)
#:use-module (guix monads)
#:use-module (guix store)
#:use-module (guix sets)
@@ -1113,82 +1114,65 @@ files for the fonts of the @var{manifest} entries."
(define (manual-database manifest)
"Return a derivation that builds the manual page database (\"mandb\") for
the entries in MANIFEST."
- (define man-db ;lazy reference
- (module-ref (resolve-interface '(gnu packages man)) 'man-db))
+ (define gdbm-ffi
+ (module-ref (resolve-interface '(gnu packages guile))
+ 'guile-gdbm-ffi))
+
+ (define zlib
+ (module-ref (resolve-interface '(gnu packages compression)) 'zlib))
+
+ (define config.scm
+ (scheme-file "config.scm"
+ #~(begin
+ (define-module (guix config)
+ #:export (%libz))
+
+ (define %libz
+ #+(file-append zlib "/lib/libz")))))
+
+ (define modules
+ (cons `((guix config) => ,config.scm)
+ (delete '(guix config)
+ (source-module-closure `((guix build utils)
+ (guix man-db))))))
(define build
- (with-imported-modules '((guix build utils))
+ (with-imported-modules modules
#~(begin
- (use-modules (guix build utils)
+ (add-to-load-path (string-append #$gdbm-ffi "/share/guile/site/"
+ (effective-version)))
+
+ (use-modules (guix man-db)
+ (guix build utils)
(srfi srfi-1)
- (srfi srfi-19)
- (srfi srfi-26))
+ (srfi srfi-19))
- (define entries
- (filter-map (lambda (directory)
+ (define (compute-entries)
+ (append-map (lambda (directory)
(let ((man (string-append directory "/share/man")))
- (and (directory-exists? man)
- man)))
+ (if (directory-exists? man)
+ (mandb-entries man)
+ '())))
'#$(manifest-inputs manifest)))
- (define manpages-collection-dir
- (string-append (getenv "PWD") "/manpages-collection"))
-
(define man-directory
(string-append #$output "/share/man"))
- (define (get-manpage-tail-path manpage-path)
- (let ((index (string-contains manpage-path "/share/man/")))
- (unless index
- (error "Manual path doesn't contain \"/share/man/\":"
- manpage-path))
- (string-drop manpage-path (+ index (string-length "/share/man/")))))
-
- (define (populate-manpages-collection-dir entries)
- (let ((manpages (append-map (cut find-files <> #:stat stat) entries)))
- (for-each (lambda (manpage)
- (let* ((dest-file (string-append
- manpages-collection-dir "/"
- (get-manpage-tail-path manpage))))
- (mkdir-p (dirname dest-file))
- (catch 'system-error
- (lambda ()
- (symlink manpage dest-file))
- (lambda args
- ;; Different packages may contain the same
- ;; manpage. Simply ignore the symlink error.
- #t))))
- manpages)))
-
- (mkdir-p manpages-collection-dir)
- (populate-manpages-collection-dir entries)
-
- ;; Create a mandb config file which contains a custom made
- ;; manpath. The associated catpath is the location where the database
- ;; gets generated.
- (copy-file #+(file-append man-db "/etc/man_db.conf")
- "man_db.conf")
- (substitute* "man_db.conf"
- (("MANDB_MAP /usr/man /var/cache/man/fsstnd")
- (string-append "MANDB_MAP " manpages-collection-dir " "
- man-directory)))
-
(mkdir-p man-directory)
- (setenv "MANPATH" (string-join entries ":"))
- (format #t "Creating manual page database for ~a packages... "
- (length entries))
+ (format #t "Creating manual page database...~%")
(force-output)
- (let* ((start-time (current-time))
- (exit-status (system* #+(file-append man-db "/bin/mandb")
- "--quiet" "--create"
- "-C" "man_db.conf"))
- (duration (time-difference (current-time) start-time)))
- (format #t "done in ~,3f s~%"
+ (let* ((start (current-time))
+ (entries (compute-entries))
+ (_ (write-mandb-database (string-append man-directory
+ "/index.db")
+ entries))
+ (duration (time-difference (current-time) start)))
+ (format #t "~a entries processed in ~,1f s~%"
+ (length entries)
(+ (time-second duration)
(* (time-nanosecond duration) (expt 10 -9))))
- (force-output)
- (zero? exit-status)))))
+ (force-output)))))
(gexp->derivation "manual-database" build
#:local-build? #t))
--
2.15.1
bug-guix <at> gnu.org:bug#29654; Package guix.
(Fri, 15 Dec 2017 21:55:02 GMT) Full text and rfc822 format available.Message #11 received at 29654 <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Ruud van Asseldonk <dev+guix <at> veniogames.com> Cc: 29654 <at> debbugs.gnu.org Subject: Re: bug#29654: Manual database index.db embeds timestamps Date: Fri, 15 Dec 2017 22:54:06 +0100
ludo <at> gnu.org (Ludovic Courtès) skribis: > I was already motivated to create the database directly from Scheme > (‘man-db’ is quite slow), so that gave me an additional excuse. ;-) FWIW, on my profile with 237 packages corresponding to 19,122 man pages, on a warm cache, (guix man-db) takes ~8 seconds whereas the previous method (invoking ‘man-db’) takes ~26 seconds. My guess is that the main difference is due to not forking a gzip process for each man page. Ludo’.
bug-guix <at> gnu.org:bug#29654; Package guix.
(Sat, 16 Dec 2017 00:37:01 GMT) Full text and rfc822 format available.Message #14 received at 29654 <at> debbugs.gnu.org (full text, mbox):
From: Ricardo Wurmus <rekado <at> elephly.net> To: Ludovic Courtès <ludo <at> gnu.org> Cc: Ruud van Asseldonk <dev+guix <at> veniogames.com>, 29654 <at> debbugs.gnu.org Subject: Re: bug#29654: Manual database index.db embeds timestamps Date: Sat, 16 Dec 2017 01:35:12 +0100
Hi Ludo,
> I was already motivated to create the database directly from Scheme
> (‘man-db’ is quite slow), so that gave me an additional excuse. ;-)
>
> The attached patch does that. The timestamps are always set to zero.
That’s great!
> Unfortunately, this is not fully deterministic: when running --check
> several times in a row, I occasionally get different results. I suspect
> GDBM’s output is not fully deterministic.
Hmm, I dumped the contents of the generated databases with gdbm_dump and
couldn’t find any difference aside from the header (which is produced by
gdbm_dump itself). Diffoscope shows a lot of differences, though.
I thought that maybe the difference comes from the fact that upon adding
new entries gdbm grows the hash table. After setting the initial size
to a multiple of the number of entries I haven’t been able to generate a
non-reproducible database.
My only change is in “write-mandb-database”:
(gdbm-open file GDBM_WRCREAT #:block-size (* 512 (length entries)))
I tried this:
./pre-inst-env guix package -p foo -i coreutils guile
for i in `seq 30`; do ./pre-inst-env guix build --check -K /gnu/store/pg3684khpj69py40v7p76b90r9q4j2lv-manual-database.drv; done
Seems fine. Coincidence or did I get lucky?
> +(define (entry->string entry)
> + "Return the wire format for ENTRY as a string."
> + (match entry
> + (($ <mandb-entry> file name section synopsis)
> + (string-append (abbreviate-file-name file) "\t"
> + (number->string section) "\t"
> + (number->string section)
> +
> + ;; Timestamps, that we always set to the epoch.
> + "\t0\t0"
> +
> + ;; XXX: Weird things.
> + "\tB\t-\t-\tgz\t"
What’s that?
--
Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
https://elephly.net
bug-guix <at> gnu.org:bug#29654; Package guix.
(Sat, 16 Dec 2017 23:52:02 GMT) Full text and rfc822 format available.Message #17 received at 29654 <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Ricardo Wurmus <rekado <at> elephly.net> Cc: Ruud van Asseldonk <dev+guix <at> veniogames.com>, 29654 <at> debbugs.gnu.org Subject: Re: bug#29654: Manual database index.db embeds timestamps Date: Sun, 17 Dec 2017 00:51:53 +0100
[Message part 1 (text/plain, inline)]
Howdy Ricardo, Ricardo Wurmus <rekado <at> elephly.net> skribis: >> Unfortunately, this is not fully deterministic: when running --check >> several times in a row, I occasionally get different results. I suspect >> GDBM’s output is not fully deterministic. > > Hmm, I dumped the contents of the generated databases with gdbm_dump and > couldn’t find any difference aside from the header (which is produced by > gdbm_dump itself). Diffoscope shows a lot of differences, though. > > I thought that maybe the difference comes from the fact that upon adding > new entries gdbm grows the hash table. After setting the initial size > to a multiple of the number of entries I haven’t been able to generate a > non-reproducible database. > > My only change is in “write-mandb-database”: > > (gdbm-open file GDBM_WRCREAT #:block-size (* 512 (length entries))) > > I tried this: > > ./pre-inst-env guix package -p foo -i coreutils guile > for i in `seq 30`; do ./pre-inst-env guix build --check -K /gnu/store/pg3684khpj69py40v7p76b90r9q4j2lv-manual-database.drv; done > > Seems fine. Coincidence or did I get lucky? I checked with the program below. It helps, but does not entirely fix it:
[t.scm (text/plain, inline)]
(use-modules (guix man-db)
(guix hash)
(guix base32))
(define %database "/tmp/index.db")
(let loop ()
(false-if-exception (delete-file %database))
(write-mandb-database %database
(mandb-entries "/home/ludo/.guix-profile/share/man"))
(pk (stat:size (stat %database))
(bytevector->nix-base32-string (file-sha256 %database)))
(loop))
[Message part 3 (text/plain, inline)]
Valgrind reports this:
--8<---------------cut here---------------start------------->8---
==8395== Syscall param write(buf) points to uninitialised byte(s)
==8395== at 0x53E4A8D: ??? (in /gnu/store/3h31zsqxjjg52da5gp3qmhkh4x8klhah-glibc-2.25/lib/libpthread-2.25.so)
==8395== by 0xACAF44D: _gdbm_full_write (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)
==8395== by 0xACAC6AD: gdbm_fd_open (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)
==8395== by 0x55FA0BF: ffi_call_unix64 (in /gnu/store/kvi64k387hqdrn59gsgd09brxh65jxjj-libffi-3.2.1/lib/libffi.so.6.0.4)
==8395== by 0x55F8EE0: ffi_call (in /gnu/store/kvi64k387hqdrn59gsgd09brxh65jxjj-libffi-3.2.1/lib/libffi.so.6.0.4)
==8395== by 0x4E8C23C: scm_i_foreign_call (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== by 0x4EF9243: vm_regular_engine (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== by 0x4EFC7B9: scm_call_n (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== by 0x4E80A06: scm_primitive_eval (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== by 0x4E80A62: scm_eval (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== by 0x4ECBA6F: scm_shell (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== by 0x4E974AC: invoke_main_func (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
==8395== Address 0xced0044 is 4 bytes inside a block of size 8,388,608 alloc'd
==8395== at 0x4C2AAD6: malloc (in /gnu/store/p2b1rzqlpdqbhn42g76xzgykbivwc063-valgrind-3.12.0/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8395== by 0xACAC5E6: gdbm_fd_open (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)
==8395== by 0x55FA0BF: ffi_call_unix64 (in /gnu/store/kvi64k387hqdrn59gsgd09brxh65jxjj-libffi-3.2.1/lib/libffi.so.6.0.4)
==8395== by 0x55F8EE0: ffi_call (in /gnu/store/kvi64k387hqdrn59gsgd09brxh65jxjj-libffi-3.2.1/lib/libffi.so.6.0.4)
==8395== by 0x4E8C23C: scm_i_foreign_call (in /gnu/store/gwspk20b7fbrs4l5rzgaadf8896h12bq-guile-2.2.3/lib/libguile-2.2.so.1.3.0)
--8<---------------cut here---------------end--------------->8---
>> +(define (entry->string entry)
>> + "Return the wire format for ENTRY as a string."
>> + (match entry
>> + (($ <mandb-entry> file name section synopsis)
>> + (string-append (abbreviate-file-name file) "\t"
>> + (number->string section) "\t"
>> + (number->string section)
>> +
>> + ;; Timestamps, that we always set to the epoch.
>> + "\t0\t0"
>> +
>> + ;; XXX: Weird things.
>> + "\tB\t-\t-\tgz\t"
>
> What’s that?
In db_store.c it’s done like this:
--8<---------------cut here---------------start------------->8---
MYDBM_SET (cont, xasprintf (
"%s\t%s\t%s\t%ld\t%ld\t%c\t%s\t%s\t%s\t%s",
dash_if_unset (in->name),
in->ext,
in->sec,
(long) in->mtime.tv_sec,
in->mtime.tv_nsec,
in->id,
in->pointer,
in->filter,
in->comp,
in->whatis));
--8<---------------cut here---------------end--------------->8---
and db_storage.h says:
--8<---------------cut here---------------start------------->8---
struct mandata {
struct mandata *next; /* ptr to next structure, if any */
char *addr; /* ptr to memory containing the fields */
char *name; /* Name of page, if != key */
/* The following are all const because they should be pointers to
* parts of strings allocated elsewhere (often the addr field above)
* and should not be written through or freed themselves.
*/
const char *ext; /* Filename ext w/o comp ext */
const char *sec; /* Section name/number */
char id; /* id for this entry */
const char *pointer; /* id related file pointer */
const char *comp; /* Compression extension */
const char *filter; /* filters needed for the page */
const char *whatis; /* whatis description for page */
struct timespec mtime; /* mod time for file */
};
--8<---------------cut here---------------end--------------->8---
The ‘B’ part gives the kind of manual page:
--8<---------------cut here---------------start------------->8---
/* These definitions give an inherent precedence to each particular type
of manual page:
ULT_MAN: ultimate manual page, the full source nroff file.
SO_MAN: source nroff file containing .so request to an ULT_MAN.
WHATIS_MAN: virtual `whatis referenced' page pointing to an ULT_MAN.
STRAY_CAT: pre-formatted manual page with no source.
WHATIS_CAT: virtual `whatis referenced' page pointing to a STRAY_CAT. */
--8<---------------cut here---------------end--------------->8---
I’ve updated man-db.scm to handle that better.
Thanks,
Ludo’.
[Message part 4 (text/x-patch, inline)]
diff --git a/guix/man-db.scm b/guix/man-db.scm
index b42558b06..3ce268547 100644
--- a/guix/man-db.scm
+++ b/guix/man-db.scm
@@ -29,6 +29,7 @@
mandb-entry-name
mandb-entry-section
mandb-entry-synopsis
+ mandb-entry-kind
mandb-entries
write-mandb-database))
@@ -47,12 +48,13 @@
(module-use! (current-module) (resolve-interface '(gdbm)))
(define-record-type <mandb-entry>
- (mandb-entry file-name name section synopsis)
+ (mandb-entry file-name name section synopsis kind)
mandb-entry?
(file-name mandb-entry-file-name) ;e.g., "../abiword.1.gz"
(name mandb-entry-name) ;e.g., "ABIWORD"
(section mandb-entry-section) ;number
- (synopsis mandb-entry-synopsis)) ;string
+ (synopsis mandb-entry-synopsis) ;string
+ (kind mandb-entry-kind)) ;'ultimate | 'link
(define (mandb-entry<? entry1 entry2)
(match entry1
@@ -74,16 +76,26 @@
(define (entry->string entry)
"Return the wire format for ENTRY as a string."
(match entry
- (($ <mandb-entry> file name section synopsis)
+ (($ <mandb-entry> file name section synopsis kind)
+ ;; See db_store.c:make_content in man-db for the format.
(string-append (abbreviate-file-name file) "\t"
(number->string section) "\t"
(number->string section)
- ;; Timestamps, that we always set to the epoch.
+ ;; Timestamp that we always set to the epoch.
"\t0\t0"
- ;; XXX: Weird things.
- "\tB\t-\t-\tgz\t"
+ ;; See "db_storage.h" in man-db for the different kinds.
+ "\t"
+ (case kind
+ ((ultimate) "A") ;ultimate man page
+ ((link) "B") ;".so" link to other man page
+ (else "A")) ;something that doesn't matter much
+
+ "\t-\t-\t"
+
+ (if (string-suffix? ".gz" file) "gz" "")
+ "\t"
synopsis "\x00"))))
@@ -94,7 +106,8 @@
(define (write-mandb-database file entries)
"Write ENTRIES to FILE as a man-db database. FILE is usually
\".../index.db\", and is a GDBM database."
- (let ((db (gdbm-open file GDBM_WRCREAT)))
+ (let ((db (gdbm-open file GDBM_WRCREAT
+ #:block-size (* 512 (length entries)))))
(gdbm-set! db %version-key %version-value)
;; Write ENTRIES in sorted order so we get deterministic output.
@@ -141,33 +154,37 @@
(string->number (string-drop (string-drop-right str 1) 1))
(string->number str)))
+ ;; Note: This works for both gzipped and uncompressed files.
(call-with-gzip-input-port (open-file file "r0")
(lambda (port)
(let loop ((name #f)
(section #f)
- (synopsis #f))
+ (synopsis #f)
+ (kind 'ultimate))
(if (and name section synopsis)
- (mandb-entry file name section synopsis)
+ (mandb-entry file name section synopsis kind)
(let ((line (read-line port)))
(if (eof-object? line)
- (mandb-entry file name (or section 0) (or synopsis ""))
+ (mandb-entry file name (or section 0) (or synopsis "")
+ kind)
(match (string-tokenize line)
((".TH" name (= string->number* section) _ ...)
- (loop name section synopsis))
+ (loop name section synopsis kind))
((".SH" (or "NAME" "\"NAME\""))
- (loop name section (read-synopsis port)))
+ (loop name section (read-synopsis port) kind))
((".so" link)
(match (and=> (resolve link)
(cut man-page->entry <> resolve))
(#f
- (loop name section synopsis))
+ (loop name section synopsis 'link))
(alias
(mandb-entry file
(mandb-entry-name alias)
(mandb-entry-section alias)
- (mandb-entry-synopsis alias)))))
+ (mandb-entry-synopsis alias)
+ 'link))))
(_
- (loop name section synopsis))))))))))
+ (loop name section synopsis kind))))))))))
(define (man-files directory)
"Return the list of man pages found under DIRECTORY, recursively."
bug-guix <at> gnu.org:bug#29654; Package guix.
(Sun, 17 Dec 2017 00:05:02 GMT) Full text and rfc822 format available.Message #20 received at 29654 <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Ricardo Wurmus <rekado <at> elephly.net> Cc: Ruud van Asseldonk <dev+guix <at> veniogames.com>, 29654 <at> debbugs.gnu.org Subject: Re: bug#29654: Manual database index.db embeds timestamps Date: Sun, 17 Dec 2017 01:04:38 +0100
ludo <at> gnu.org (Ludovic Courtès) skribis: > ==8395== Syscall param write(buf) points to uninitialised byte(s) > ==8395== at 0x53E4A8D: ??? (in /gnu/store/3h31zsqxjjg52da5gp3qmhkh4x8klhah-glibc-2.25/lib/libpthread-2.25.so) > ==8395== by 0xACAF44D: _gdbm_full_write (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0) > ==8395== by 0xACAC6AD: gdbm_fd_open (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0) This is almost certainly an uninitialized malloc’d area that goes straight to disk as can be seen by running the script I gave before with ‘MALLOC_PERTURB_’ set: --8<---------------cut here---------------start------------->8--- ludo <at> ribbon ~/src/guix$ MALLOC_PERTURB_=22 ./pre-inst-env guile t.scm ;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na") ;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na") ;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na") ;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na") C-c C-c ludo <at> ribbon ~/src/guix$ MALLOC_PERTURB_=44 ./pre-inst-env guile t.scm ;;; (33554432 "03kl992ypwxxp4cplbhz05b04ihjh96d3pldwgz7qaj4ls0qssr3") ;;; (33554432 "03kl992ypwxxp4cplbhz05b04ihjh96d3pldwgz7qaj4ls0qssr3") ;;; (33554432 "03kl992ypwxxp4cplbhz05b04ihjh96d3pldwgz7qaj4ls0qssr3") ;;; (33554432 "03kl992ypwxxp4cplbhz05b04ihjh96d3pldwgz7qaj4ls0qssr3") C-c C-c ludo <at> ribbon ~/src/guix$ MALLOC_PERTURB_=22 ./pre-inst-env guile t.scm ;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na") ;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na") ;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na") ;;; (33554432 "18rzgjskkgllwfkdyx7gbcc3z6aqsqm9lm6dzs9yrw9z7ppqx0na") --8<---------------cut here---------------end--------------->8--- So for now, until gdbm is fixed, we can always work around the issue by setting MALLOC_PERTURB_. How does that sound? Ludo’.
bug-guix <at> gnu.org:bug#29654; Package guix.
(Sun, 17 Dec 2017 00:27:02 GMT) Full text and rfc822 format available.Message #23 received at 29654 <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: bug-gdbm <at> gnu.org Cc: Ricardo Wurmus <rekado <at> elephly.net>, Ruud van Asseldonk <dev+guix <at> veniogames.com>, 29654 <at> debbugs.gnu.org Subject: GDBM output is not deterministic Date: Sun, 17 Dec 2017 01:26:17 +0100
Hello Sergey & co.,
While investigating <https://bugs.gnu.org/29654>, we realized that
databases produced by GDBM are not bit-for-bit reproducible. This can
be seen with this program:
--8<---------------cut here---------------start------------->8---
#include <unistd.h>
#include <gdbm.h>
int
main ()
{
unlink ("/tmp/t.db");
gdbm_open ("/tmp/t.db", 0, GDBM_WRCREAT, 0644, NULL);
}
--8<---------------cut here---------------end--------------->8---
Under Valgrind, we get:
--8<---------------cut here---------------start------------->8---
==15981== Syscall param write(buf) points to uninitialised byte(s)
==15981== at 0x5335080: __write_nocancel (in /gnu/store/3h31zsqxjjg52da5gp3qmhkh4x8klhah-glibc-2.25/lib/libc-2.25.so)
==15981== by 0x4E3E44D: _gdbm_full_write (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)
==15981== by 0x4E3B6AD: gdbm_fd_open (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)
==15981== by 0x400743: main (in /home/ludo/src/guix/a.out)
==15981== Address 0x55fb204 is 4 bytes inside a block of size 4,096 alloc'd
==15981== at 0x4C2AAD6: malloc (in /gnu/store/p2b1rzqlpdqbhn42g76xzgykbivwc063-valgrind-3.12.0/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15981== by 0x4E3B5E6: gdbm_fd_open (in /gnu/store/kg8ffb14msfnc9aivxj6djrl51g9b3zz-gdbm-1.13/lib/libgdbm.so.4.0.0)
==15981== by 0x400743: main (in /home/ludo/src/guix/a.out)
--8<---------------cut here---------------end--------------->8---
… which suggests that an uninitialized block is being written straightly
to disk.
If we tell libc to initialize ‘malloc’ areas that were otherwise left
uninitialized, we can confirm the problem (again with the program
above):
--8<---------------cut here---------------start------------->8---
$ for i in `seq 0 20` ; do MALLOC_PERTURB_=$i ./a.out ; sha256sum /tmp/t.db ; done
0483a04a43d0a08a5a0aa71d09b262b62eb2504e85c5fe7104b783a5f0f7ae15 /tmp/t.db
29122b2994face5b5a716c8cd0e6beda7f05c3441441447872c2f15e7e2db308 /tmp/t.db
34a130820ab6d5e638adae55e83cb784fec036c2da4727a113d3d1191331bd7c /tmp/t.db
9604ce76ea1f1b37968bd037798c0eb212a1e654f060b7c3d66011620b249212 /tmp/t.db
a1c275c1bf1b5980166431bd7245d2d649499a8cad233059abf2a6cbf2dec19b /tmp/t.db
1cbe43f146a158ac4c21064feac1371e7a6bb4ac9d0013b15acfe7fec754486f /tmp/t.db
83778759475790f1377adadd77b9d2eee7db7e5e43e006ef04e805f4b4309590 /tmp/t.db
6b80d00119fd806d05f8863056919ea0a6b66a0771d16439b072e2cc2eac6b58 /tmp/t.db
f1f65b95e39f74cff12c3e830eea77a9841c8793f4441c6a8bfa9ad92d765d8d /tmp/t.db
3dd2479f44c0be15e02f0040fcef3d765a306dbd2a47b3557cb19fb1282b0b05 /tmp/t.db
fed4cc15d84f676737954e80ab594034e47d2f2b7cbcd12dad722dddff6c5afb /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
3d9bccf127fd76b56617435c412aeec0c8fc95ab6b9d5f9e3f24060398899c26 /tmp/t.db
743808bd91a42e2c149a30bbdfacadbaa4124f82e01ea43323d5c1fd10de5499 /tmp/t.db
86a92606713ca03cab09f1cc97463184ed0c4d2051d7512dd98b136bc7ca1465 /tmp/t.db
93f5428a4fde073cab1737c1efd64e2554731db1ab51d77aec95c81592b014a0 /tmp/t.db
b93fd6692eeebfc5b49b64857de7647a88e541be7d4fd4e62fb6771cb4b00618 /tmp/t.db
ba08b6acee5812c3b3770f842aca7435729acc1082cddf8bfe26024f9ea8f54e /tmp/t.db
30dc2e3d1ea0150e4a200435b1f609de726680ebfc8661a9d121afbd4cf0b0a8 /tmp/t.db
0de2cc18e445ff0a2697cd2783f379fa202d57020db9a73c55c4f7b8fd5cac74 /tmp/t.db
dd938784d3799c5a575abc3fe24191f3e1ac593643719a9694f9fed32ed272da /tmp/t.db
$ for i in `seq 0 20` ; do MALLOC_PERTURB_=11 ./a.out ; sha256sum /tmp/t.db ; done
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
a43fb60e3d94093c11bd7bb2cf5253ccb5c3e772eca82ba9c9aac64204e15bfc /tmp/t.db
--8<---------------cut here---------------end--------------->8---
I guess the fix is to use ‘calloc’ instead of ‘malloc’ for this specific
allocation.
WDYT?
Ludo’.
bug-guix <at> gnu.org:bug#29654; Package guix.
(Sun, 17 Dec 2017 16:28:01 GMT) Full text and rfc822 format available.Message #26 received at 29654 <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Ricardo Wurmus <rekado <at> elephly.net> Cc: Ruud van Asseldonk <dev+guix <at> veniogames.com>, 29654 <at> debbugs.gnu.org Subject: Re: bug#29654: Manual database index.db embeds timestamps Date: Sun, 17 Dec 2017 17:27:21 +0100
Hello, I went ahead and pushed (guix man-db) with MALLOC_PERTURB_ set as a workaround: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=b8396f96bfeadfa63e7ad2afc2ab5a37f37f5f81 AFAICS this fixes the problem. Thanks, Ludo’.
bug-guix <at> gnu.org:bug#29654; Package guix.
(Sun, 17 Dec 2017 18:12:02 GMT) Full text and rfc822 format available.Message #29 received at 29654 <at> debbugs.gnu.org (full text, mbox):
From: Sergey Poznyakoff <gray <at> gnu.org.ua> To: "Ludovic Courtès" <ludo <at> gnu.org> Cc: Ricardo Wurmus <rekado <at> elephly.net>, bug-gdbm <at> gnu.org, Ruud van Asseldonk <dev+guix <at> veniogames.com>, 29654 <at> debbugs.gnu.org Subject: Re: GDBM output is not deterministic Date: Sun, 17 Dec 2017 11:19:25 +0200
Hi Ludovic, > I guess the fix is to use calloc Yes, that's quite reasonable. I'll fix that. Thanks for reporting! Regards, Sergey
bug-guix <at> gnu.org:bug#29654; Package guix.
(Sun, 17 Dec 2017 19:53:02 GMT) Full text and rfc822 format available.Message #32 received at 29654 <at> debbugs.gnu.org (full text, mbox):
From: Ruud van Asseldonk <dev+guix <at> veniogames.com> To: Ludovic Courtès <ludo <at> gnu.org>, Ricardo Wurmus <rekado <at> elephly.net> Cc: 29654 <at> debbugs.gnu.org Subject: Re: bug#29654: Manual database index.db embeds timestamps Date: Sun, 17 Dec 2017 20:51:58 +0100
Thanks so much for looking into this!
Unfortunately, I am unable to pull that commit. Guix pull output:
Updating from Git repository at
'https://git.savannah.gnu.org/git/guix.git'...
Building from Git commit b8396f96bfeadfa63e7ad2afc2ab5a37f37f5f81...
The following derivation will be built:
/gnu/store/m8wcw35f4d33rzf54zzcrg0g80m0zi4r-guix-latest.drv
copying and compiling to
'/gnu/store/2g14x91l9wh4wv8zsvnkfblrrdfcysg4-guix-latest' with Guile
2.2.2...
loading... 17.1% of 659 filesBacktrace:
12 (primitive-load "/gnu/store/9pgqmhjclkayr5bdhqfk69sbn84?")
In ./guix/build/pull.scm:
127:8 11 (build-guix _ _ #:system _ #:storedir _ #:localstatedir ?)
In ./guix/build/compile.scm:
158:6 10 (compile-files _ _ ("/gnu/store/2g14x91l9wh4wv8zsvn?" ?) ?)
107:11 9 (load-files "/gnu/store/2g14x91l9wh4wv8zsvnkfblrrdfcys?" ?)
In ice-9/boot-9.scm:
2792:17 8 (resolve-interface (guix man-db) #:select _ #:hide _ # _ ?)
2718:10 7 (_ (guix man-db) _ _ #:ensure _)
2986:16 6 (try-module-autoload _ _)
2316:4 5 (save-module-excursion #<procedure 44847e0 at ice-9/boo?>)
3006:22 4 (_)
In unknown file:
3 (primitive-load-path "guix/man-db" #<procedure 40e90a0 ?>)
In ice-9/eval.scm:
196:43 2 (_ #f)
In ice-9/boot-9.scm:
2795:6 1 (resolve-interface _ #:select _ #:hide _ #:prefix _ # _ ?)
In unknown file:
0 (scm-error misc-error #f "~A ~S" ("no code for modu?" ?) ?)
ERROR: In procedure scm-error:
ERROR: no code for module (gdbm)
This happens even in an environment with guile-gdbm-ffi and gdbm
packages. Am I missing something?
Best,
Ruud
Ludovic Courtès wrote:
> Hello,
>
> I went ahead and pushed (guix man-db) with MALLOC_PERTURB_ set as a
> workaround:
>
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=b8396f96bfeadfa63e7ad2afc2ab5a37f37f5f81
>
> AFAICS this fixes the problem.
>
> Thanks,
> Ludo’.
>
ludo <at> gnu.org (Ludovic Courtès):Ruud van Asseldonk <dev+guix <at> veniogames.com>:Message #37 received at 29654-done <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Ruud van Asseldonk <dev+guix <at> veniogames.com> Cc: Ricardo Wurmus <rekado <at> elephly.net>, 29654-done <at> debbugs.gnu.org Subject: Re: bug#29654: Manual database index.db embeds timestamps Date: Sun, 17 Dec 2017 22:32:27 +0100
Ruud van Asseldonk <dev+guix <at> veniogames.com> skribis:
> Unfortunately, I am unable to pull that commit. Guix pull output:
>
> Updating from Git repository at
> 'https://git.savannah.gnu.org/git/guix.git'...
> Building from Git commit b8396f96bfeadfa63e7ad2afc2ab5a37f37f5f81...
> The following derivation will be built:
> /gnu/store/m8wcw35f4d33rzf54zzcrg0g80m0zi4r-guix-latest.drv
> copying and compiling to
> '/gnu/store/2g14x91l9wh4wv8zsvnkfblrrdfcysg4-guix-latest' with Guile
> 2.2.2...
> loading... 17.1% of 659 filesBacktrace:
> 12 (primitive-load "/gnu/store/9pgqmhjclkayr5bdhqfk69sbn84?")
> In ./guix/build/pull.scm:
> 127:8 11 (build-guix _ _ #:system _ #:storedir _ #:localstatedir ?)
> In ./guix/build/compile.scm:
> 158:6 10 (compile-files _ _ ("/gnu/store/2g14x91l9wh4wv8zsvn?" ?) ?)
> 107:11 9 (load-files "/gnu/store/2g14x91l9wh4wv8zsvnkfblrrdfcys?" ?)
> In ice-9/boot-9.scm:
> 2792:17 8 (resolve-interface (guix man-db) #:select _ #:hide _ # _ ?)
> 2718:10 7 (_ (guix man-db) _ _ #:ensure _)
> 2986:16 6 (try-module-autoload _ _)
> 2316:4 5 (save-module-excursion #<procedure 44847e0 at ice-9/boo?>)
> 3006:22 4 (_)
> In unknown file:
> 3 (primitive-load-path "guix/man-db" #<procedure 40e90a0 ?>)
> In ice-9/eval.scm:
> 196:43 2 (_ #f)
> In ice-9/boot-9.scm:
> 2795:6 1 (resolve-interface _ #:select _ #:hide _ #:prefix _ # _ ?)
> In unknown file:
> 0 (scm-error misc-error #f "~A ~S" ("no code for modu?" ?) ?)
>
> ERROR: In procedure scm-error:
> ERROR: no code for module (gdbm)
My bad, fixed in 16613d230b3d9a9cf307c5c5d3899eb0a0c93b0e, which makes
GDBM is “soft dependency” (we only need it when building the manual
database, not when building Guix itself, after all.)
Thanks!
Ludo’.
Debbugs Internal Request <help-debbugs <at> gnu.org>
to internal_control <at> debbugs.gnu.org.
(Mon, 15 Jan 2018 12:24:07 GMT) Full text and rfc822 format available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.