GNU bug report logs - #28745
tarballs generated on github are generated on demand (leading to different hash sums)

Previous Next

Package: guix;

Reported by: ng0 <ng0 <at> infotropique.org>

Date: Sun, 8 Oct 2017 11:41:01 UTC

Severity: important

Done: ludo <at> gnu.org (Ludovic Courtès)

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 28745 in the body.
You can then email your comments to 28745 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-guix <at> gnu.org:
bug#28745; Package guix. (Sun, 08 Oct 2017 11:41:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to ng0 <ng0 <at> infotropique.org>:
New bug report received and forwarded. Copy sent to bug-guix <at> gnu.org. (Sun, 08 Oct 2017 11:41:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: ng0 <ng0 <at> infotropique.org>
To: bug-guix <at> gnu.org
Subject: tarballs generated on github are generated on demand (leading to
 different hash sums)
Date: Sun, 8 Oct 2017 11:40:09 +0000
[Message part 1 (text/plain, inline)]
Past and recent discussion in our IRC channel and on the mailing list
show that we can not rely on tarballs on github keeping the same
hash forever.
According to github they are "generated on demand", leading to
regular hash mismatches.

Since some of our own dependencies are on github (at the very least
guile-git), we need to come up with a solution.

Right now we have around 449 packages with tarball sources from
github in our gnu/packages.
We could:

- Move them all to use git-download and just use
  the commit that has been tagged in the versions that produce
  the tarballs on github.

- Mirror the content somewhere reliable in snapshots for
  some time. Problem here: we start to rely on this "somewhere"
  to be trustworthy and introduce one more point to trust
  (however due to pre-recorded hash sum this is just an annoyance,
  not a grave issue).

- Your idea here.

-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://dist.krosos.org/dist/keys/
https://www.infotropique.org https://krosos.org
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-guix <at> gnu.org:
bug#28745; Package guix. (Sun, 08 Oct 2017 11:45:02 GMT) Full text and rfc822 format available.

Message #8 received at 28745 <at> debbugs.gnu.org (full text, mbox):

From: ng0 <ng0 <at> infotropique.org>
To: 28745 <at> debbugs.gnu.org
Subject: Re: bug#28745: tarballs generated on github are generated on demand
 (leading to different hash sums)
Date: Sun, 8 Oct 2017 11:44:02 +0000
[Message part 1 (text/plain, inline)]
ng0 transcribed 2.1K bytes:
…
> Since some of our own dependencies are on github (at the very least
> guile-git), we need to come up with a solution.
…

Correction: libgit2 is on github, a dependency of guile-git (which is on gitlab).

-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://dist.krosos.org/dist/keys/
https://www.infotropique.org https://krosos.org
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-guix <at> gnu.org:
bug#28745; Package guix. (Sun, 08 Oct 2017 21:20:02 GMT) Full text and rfc822 format available.

Message #11 received at 28745 <at> debbugs.gnu.org (full text, mbox):

From: Jan Nieuwenhuizen <janneke <at> gnu.org>
To: ng0 <ng0 <at> infotropique.org>
Cc: 28745 <at> debbugs.gnu.org
Subject: Re: bug#28745: tarballs generated on github are generated on demand
 (leading to different hash sums)
Date: Sun, 08 Oct 2017 23:18:52 +0200
ng0 writes:

> ng0 transcribed 2.1K bytes:
> …
>> Since some of our own dependencies are on github (at the very least
>> guile-git), we need to come up with a solution.
> …
>
> Correction: libgit2 is on github, a dependency of guile-git (which is on gitlab).

Sure, see bug#28659 ...possbily this needs to be merged that bug.
janneke


-- 
Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com




Information forwarded to bug-guix <at> gnu.org:
bug#28745; Package guix. (Sun, 08 Oct 2017 22:00:02 GMT) Full text and rfc822 format available.

Message #14 received at 28745 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: ng0 <ng0 <at> infotropique.org>
Cc: 28745 <at> debbugs.gnu.org
Subject: Re: bug#28745: tarballs generated on github are generated on demand
 (leading to different hash sums)
Date: Sun, 08 Oct 2017 23:58:48 +0200
ng0 <ng0 <at> infotropique.org> writes:

> Right now we have around 449 packages with tarball sources from
> github in our gnu/packages.

I assume that this problem does not exist for tarballs that have been
signed and uploaded by the maintainer.  This is only a problem for
auto-generated tarballs for tags, so it’s probably less than 449
packages.

> - Move them all to use git-download and just use
>   the commit that has been tagged in the versions that produce
>   the tarballs on github.

This doesn’t seem like a bad idea.  It’s not great that we’ll have to
bootstrap the build systems for all these packages.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net





Severity set to 'important' from 'normal' Request was from ludo <at> gnu.org (Ludovic Courtès) to control <at> debbugs.gnu.org. (Wed, 11 Oct 2017 13:30:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-guix <at> gnu.org:
bug#28745; Package guix. (Mon, 16 Oct 2017 03:11:02 GMT) Full text and rfc822 format available.

Message #19 received at 28745 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: bug#28745 <28745 <at> debbugs.gnu.org>
Subject: [PATCH] tarballs generated on github are generated on demand (leading
 to different hash sums)
Date: Sun, 15 Oct 2017 23:10:43 -0400
[Message part 1 (text/plain, inline)]
Hello,

I could finish a script that helped me finding all of our affected
packages, verify that only the hash but not the content of the archives
had changed, as well as automate the hash update for those safe to
update.

Attached is the patch and the scripts I used. I think we might
want to reuse some of it to extend guix lint to warn packagers that
archives coming from .*github.*archives URL are not guaranteed to be
stable and that it would be better, if available, to use manually
uploaded releases archives.

Thanks!

Maxim

PS: I've also uploaded the scripts here:
https://notabug.org/apteryx/fiasco for ease of cloning. Any comments
about my nascent (ab)use of Scheme are welcome!

[0001-gnu-packages-Fix-the-hashes-of-mutated-GitHub-archiv.patch (text/x-patch, attachment)]
[Message part 3 (text/plain, attachment)]
[Message part 4 (text/plain, attachment)]
[Message part 5 (text/plain, attachment)]

Reply sent to ludo <at> gnu.org (Ludovic Courtès):
You have taken responsibility. (Fri, 20 Oct 2017 21:05:01 GMT) Full text and rfc822 format available.

Notification sent to ng0 <ng0 <at> infotropique.org>:
bug acknowledged by developer. (Fri, 20 Oct 2017 21:05:02 GMT) Full text and rfc822 format available.

Message #24 received at 28745-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: bug#28745 <28745-done <at> debbugs.gnu.org>
Subject: Re: bug#28745: [PATCH] tarballs generated on github are generated on
 demand (leading to different hash sums)
Date: Fri, 20 Oct 2017 23:04:43 +0200
Hi,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> I could finish a script that helped me finding all of our affected
> packages, verify that only the hash but not the content of the archives
> had changed, as well as automate the hash update for those safe to
> update.

Great job!

> Attached is the patch and the scripts I used. I think we might
> want to reuse some of it to extend guix lint to warn packagers that
> archives coming from .*github.*archives URL are not guaranteed to be
> stable and that it would be better, if available, to use manually
> uploaded releases archives.

Unfortunately, it’s become commonplace to publish nothing else than a
Git tag.  Now, in those cases, we could also use ‘git-fetch’, which
wouldn’t be affected by problems with generated tarballs.

Thoughts?

> PS: I've also uploaded the scripts here:
> https://notabug.org/apteryx/fiasco for ease of cloning. Any comments
> about my nascent (ab)use of Scheme are welcome!

The code looks nice!

> From 774a764149ecb0e234ae09c9a0a273af671c3c86 Mon Sep 17 00:00:00 2001
> From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
> Date: Sun, 15 Oct 2017 22:17:12 -0400
> Subject: [PATCH] gnu: packages: Fix the hashes of mutated GitHub archives.
>
> Fixes bug https://bugs.gnu.org/28745.
>
> * gnu/packages/audio.scm (csound): Fix hash.
> * gnu/packages/engineering.scm (fritzing): Likewise.
> * gnu/packages/erlang.scm (erlang): Likewise.
> * gnu/packages/fonts.scm (font-google-material-design-icons): Likewise.
> * gnu/packages/graphics.scm (ogre): Likewise.
> * gnu/packages/java.scm (java-plexus-interpolation, antlr3): Likewise.
> * gnu/packages/serialization.scm (yaml-cpp): Likewise.
> * gnu/packages/version-control.scm (libgit2): Likewise.

I’ve checked the hashes by running:

  ./pre-inst-env guix build -S --no-substitutes csound fritzing erlang \
     font-google-material-design-icons ogre java-plexus-interpolation \
     antlr3 yaml-cpp libgit2  --max-jobs=2

and everything went well.

Pushed as fd75eb6cd4e5c689f9e6ce7dd8d87f423778d308, thanks!

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#28745; Package guix. (Sun, 22 Oct 2017 03:14:01 GMT) Full text and rfc822 format available.

Message #27 received at 28745-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: bug#28745 <28745-done <at> debbugs.gnu.org>
Subject: Re: bug#28745: [PATCH] tarballs generated on github are generated on
 demand (leading to different hash sums)
Date: Sat, 21 Oct 2017 23:13:36 -0400
ludo <at> gnu.org (Ludovic Courtès) writes:

> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>
[...]
>
>> Attached is the patch and the scripts I used. I think we might
>> want to reuse some of it to extend guix lint to warn packagers that
>> archives coming from .*github.*archives URL are not guaranteed to be
>> stable and that it would be better, if available, to use manually
>> uploaded releases archives.
>
> Unfortunately, it’s become commonplace to publish nothing else than a
> Git tag.  Now, in those cases, we could also use ‘git-fetch’, which
> wouldn’t be affected by problems with generated tarballs.
>
> Thoughts?

I think the status quo is reasonable for now; if this becomes a recurring
problem we can reopen the issue and do something more about it.

>> PS: I've also uploaded the scripts here:
>> https://notabug.org/apteryx/fiasco for ease of cloning. Any comments
>> about my nascent (ab)use of Scheme are welcome!
>
> The code looks nice!

OK, that's reassuring! :)

>
>> From 774a764149ecb0e234ae09c9a0a273af671c3c86 Mon Sep 17 00:00:00 2001
>> From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
>> Date: Sun, 15 Oct 2017 22:17:12 -0400
>> Subject: [PATCH] gnu: packages: Fix the hashes of mutated GitHub archives.
>>
>> Fixes bug https://bugs.gnu.org/28745.
>>
>> * gnu/packages/audio.scm (csound): Fix hash.
>> * gnu/packages/engineering.scm (fritzing): Likewise.
>> * gnu/packages/erlang.scm (erlang): Likewise.
>> * gnu/packages/fonts.scm (font-google-material-design-icons): Likewise.
>> * gnu/packages/graphics.scm (ogre): Likewise.
>> * gnu/packages/java.scm (java-plexus-interpolation, antlr3): Likewise.
>> * gnu/packages/serialization.scm (yaml-cpp): Likewise.
>> * gnu/packages/version-control.scm (libgit2): Likewise.
>
> I’ve checked the hashes by running:
>
>   ./pre-inst-env guix build -S --no-substitutes csound fritzing erlang \
>      font-google-material-design-icons ogre java-plexus-interpolation \
>      antlr3 yaml-cpp libgit2  --max-jobs=2
>
> and everything went well.
>
> Pushed as fd75eb6cd4e5c689f9e6ce7dd8d87f423778d308, thanks!
>
> Ludo’.

Thanks!

Maxim




Information forwarded to bug-guix <at> gnu.org:
bug#28745; Package guix. (Sun, 22 Oct 2017 10:49:01 GMT) Full text and rfc822 format available.

Message #30 received at 28745-done <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: bug#28745 <28745-done <at> debbugs.gnu.org>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#28745: [PATCH] tarballs generated on github are generated on
 demand (leading to different hash sums)
Date: Sun, 22 Oct 2017 12:48:04 +0200
Ludovic Courtès <ludo <at> gnu.org> writes:

> Unfortunately, it’s become commonplace to publish nothing else than a
> Git tag.  Now, in those cases, we could also use ‘git-fetch’, which
> wouldn’t be affected by problems with generated tarballs.
>
> Thoughts?

For a couple of packages I’ve already started using git-fetch with the
tag (instead of the commit hash).  I think that’s preferable over using
auto-generated tarballs.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net






bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Sun, 19 Nov 2017 12:24:04 GMT) Full text and rfc822 format available.

This bug report was last modified 6 years and 131 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.