GNU bug report logs - #22086
25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf systems

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 22086 in the body.
You can then email your comments to 22086 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Wolfgang Jenkner <wjenkner <at> inode.at>
To: bug-gnu-emacs <at> gnu.org
Subject: 25.1.50;
 [PATCH] Integrate the musl hybrid malloc patch for elf systems
Date: Thu, 03 Dec 2015 18:57:47 +0100

[Message part 1 (text/plain, inline)]

In the thread around

http://permalink.gmane.org/gmane.emacs.devel/182469

Rich Felker announced that he had found that a small addition to
unexelf.c allowed emacs to extend its hybrid malloc implementation
(previously only used for cygwin) to musl libc, but without using
features specific to that malloc implementation.

A corresponding patch was tested in the "Alpine Linux" distribution, see

http://git.alpinelinux.org/cgit/aports/plain/testing/emacs/musl.patch?id=d6f211ec868df4657c745b8ba2bae77b2a7fb7f6

This patch series proposes the relevant part of that change and adds
some (rather trivial) groundwork to make it reliably work on all systems
which use unexelf.c for dumping.

Personally, I've been using hybrid malloc on top of the libc malloc
(jemalloc) on FreeBSD 10 for several months now without problems.

I also tested it with other malloc libraries, in particular some of
those derived from Doug Lea's malloc.  It also works on GNU/Linux, on
top of glibc.  This might be of some interest as, IIRC, some glibc
people were not too happy about keeping malloc_get_state and stuff.

If you get an error about exceeding "static heap size" during dumping
just bump STATIC_HEAP_SIZE in src/sheap.c a bit.

If the commit messages in the following patches don't explain enough,
please ask.  Also, please apply the patch from

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22085

beforehand.

[0002-For-HYBRID_MALLOC-give-most-gmalloc-symbols-internal.patch (text/x-diff, attachment)]

[0003-Link-temacs-with-gnulib-objects-compiled-with-Demacs.patch (text/x-diff, attachment)]

[0004-Add-musl-patch-to-support-HYBRID_MALLOC-on-elf-syste.patch (text/x-diff, attachment)]

[0005-configure.ac-Stop-using-mmap-for-buffers-for-FreeBSD.patch (text/x-diff, attachment)]

Message #8 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Wolfgang Jenkner <wjenkner <at> inode.at>
Cc: Rich Felker <dalias <at> aerifal.cx>, 22086 <at> debbugs.gnu.org
Subject: Re: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf
 systems
Date: Wed, 16 Dec 2015 00:28:48 -0800

Those patches look reasonable, except that one is listed as being by Rich 
Felker, who has not signed copyright papers for GNU Emacs as far as I know. I'll 
CC: this to him to give him a heads-up. Rich, would you mind transferring 
copyright to that patch to the FSF? Here's a link:

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22086;msg=5;att=3;filename=0004-Add-musl-patch-to-support-HYBRID_MALLOC-on-elf-syste.patch

Message #11 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Rich Felker <dalias <at> aerifal.cx>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Wolfgang Jenkner <wjenkner <at> inode.at>, 22086 <at> debbugs.gnu.org
Subject: Re: 25.1.50; [PATCH] Integrate the musl hybrid malloc patch for elf
 systems
Date: Wed, 16 Dec 2015 12:15:42 -0500

On Wed, Dec 16, 2015 at 12:28:48AM -0800, Paul Eggert wrote:
> Those patches look reasonable, except that one is listed as being by
> Rich Felker, who has not signed copyright papers for GNU Emacs as
> far as I know. I'll CC: this to him to give him a heads-up. Rich,
> would you mind transferring copyright to that patch to the FSF?
> Here's a link:
> 
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22086;msg=5;att=3;filename=0004-Add-musl-patch-to-support-HYBRID_MALLOC-on-elf-syste.patch

I have no objection, but I figured it was trivial enough not to need
any explicit assignment. Who do I need to contact about doing the
assignment papers?

Rich

Message #14 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Rich Felker <dalias <at> aerifal.cx>
Cc: wjenkner <at> inode.at, eggert <at> cs.ucla.edu, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50;
 [PATCH] Integrate the musl hybrid malloc patch for elf systems
Date: Wed, 16 Dec 2015 19:47:40 +0200

> Date: Wed, 16 Dec 2015 12:15:42 -0500
> From: Rich Felker <dalias <at> aerifal.cx>
> Cc: Wolfgang Jenkner <wjenkner <at> inode.at>, 22086 <at> debbugs.gnu.org
> 
> Who do I need to contact about doing the assignment papers?

Us.  I can send you the form with instructions (off-list) if you are
willing to do this.

Thanks.

Message #17 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Wolfgang Jenkner <wjenkner <at> inode.at>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Rich Felker <dalias <at> aerifal.cx>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50;
 [PATCH] Integrate the musl hybrid malloc patch for elf systems
Date: Thu, 17 Dec 2015 14:16:41 +0100

On Wed, Dec 16 2015, Paul Eggert wrote:

> Those patches look reasonable, except that one is listed as being by
> Rich Felker, who has not signed copyright papers for GNU Emacs as far
> as I know. I'll CC: this to him to give him a heads-up. Rich, would
> you mind transferring copyright to that patch to the FSF? Here's
> a link:
>
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22086;msg=5;att=3;filename=0004-Add-musl-patch-to-support-HYBRID_MALLOC-on-elf-syste.patch


Rich Felker has asked me in private mail (please, please CC your replies
to 22086 <at> debbugs.gnu.org) to clarify the authorship of that patch.
I actually stated in the commit message

    Except for build system fixes this is essentially the same as

    http://git.alpinelinux.org/cgit/aports/tree/testing/emacs/musl.patch?id=d6f211ec868df4657c745b8ba2bae77b2a7fb7f6

So, I wrote the build system fixes, and, strictly speaking, only the two
lines in src/unexelf.c which declare and initialize bss_sbrk_did_unexec
are directly from the original "Alpine Linux" patch above.

I can split the patch accordingly and ascribe only the latter part to
Rich Felker if this is preferred.  The original patch still helped me to
quickly find where I had to change something; this is why a attributed
the whole patch to Rich Felker as it is just an expansion, as it were,
of his version.

Wolfgang

Message #20 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Wolfgang Jenkner <wjenkner <at> inode.at>
Cc: eggert <at> cs.ucla.edu, dalias <at> aerifal.cx, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50;
 [PATCH] Integrate the musl hybrid malloc patch for elf systems
Date: Thu, 17 Dec 2015 18:17:25 +0200

> From: Wolfgang Jenkner <wjenkner <at> inode.at>
> Date: Thu, 17 Dec 2015 14:16:41 +0100
> Cc: Rich Felker <dalias <at> aerifal.cx>, 22086 <at> debbugs.gnu.org
> 
> I can split the patch accordingly and ascribe only the latter part to
> Rich Felker if this is preferred.

If this is not too much trouble, yes, please split it.  It is better
to have each part attributed to the person who wrote it, instead of
both of them as a whole attributed to both of you.

Thanks.

Message #23 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Rich Felker <dalias <at> aerifal.cx>
To: Wolfgang Jenkner <wjenkner <at> inode.at>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Thu, 17 Dec 2015 11:26:07 -0500

On Thu, Dec 17, 2015 at 02:16:41PM +0100, Wolfgang Jenkner wrote:
> On Wed, Dec 16 2015, Paul Eggert wrote:
> 
> > Those patches look reasonable, except that one is listed as being by
> > Rich Felker, who has not signed copyright papers for GNU Emacs as far
> > as I know. I'll CC: this to him to give him a heads-up. Rich, would
> > you mind transferring copyright to that patch to the FSF? Here's
> > a link:
> >
> > http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22086;msg=5;att=3;filename=0004-Add-musl-patch-to-support-HYBRID_MALLOC-on-elf-syste.patch
> 
> 
> Rich Felker has asked me in private mail (please, please CC your replies
> to 22086 <at> debbugs.gnu.org) to clarify the authorship of that patch.
> I actually stated in the commit message
> 
>     Except for build system fixes this is essentially the same as
> 
>     http://git.alpinelinux.org/cgit/aports/tree/testing/emacs/musl.patch?id=d6f211ec868df4657c745b8ba2bae77b2a7fb7f6
> 
> So, I wrote the build system fixes, and, strictly speaking, only the two
> lines in src/unexelf.c which declare and initialize bss_sbrk_did_unexec
> are directly from the original "Alpine Linux" patch above.
> 
> I can split the patch accordingly and ascribe only the latter part to
> Rich Felker if this is preferred.  The original patch still helped me to
> quickly find where I had to change something; this is why a attributed
> the whole patch to Rich Felker as it is just an expansion, as it were,
> of his version.

I'm not concerned with how it's split or attributed as long as the FSF
is fine with it. I just wanted to avoid misrepresenting parts that I
did not actually write when describing my changes for copyright
assignment. But this does solve my confusion about why I thought the
patch was trivial -- my original version was hardly anything but some
//'s and #if 0's.

Assuming you already have the proper assignment on file, does it
suffice for me to just mention that parts of the patch are your work?

Rich

Message #26 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Wolfgang Jenkner <wjenkner <at> inode.at>
To: Rich Felker <dalias <at> aerifal.cx>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50;
 [PATCH] Integrate the musl hybrid malloc patch for elf systems
Date: Fri, 18 Dec 2015 01:06:48 +0100

On Thu, Dec 17 2015, Rich Felker wrote:

> I'm not concerned with how it's split or attributed as long as the FSF
> is fine with it. I just wanted to avoid misrepresenting parts that I
> did not actually write when describing my changes for copyright
> assignment.

Let's just split the patch.  If you intend to sign the assignment for
"past and future changes" the usual procedure is to wait until this is
on file and then to simply commit the changes under your name.  I don't
think that

    [Which files have you changed so far, and which new files have you written
    so far?]

means that you have to describe those changes beforehand (but I could be
wrong).

Wolfgang

Message #29 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Wolfgang Jenkner <wjenkner <at> inode.at>
Cc: eggert <at> cs.ucla.edu, dalias <at> aerifal.cx, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50;
 [PATCH] Integrate the musl hybrid malloc patch for elf systems
Date: Fri, 18 Dec 2015 08:57:43 +0200

> From: Wolfgang Jenkner <wjenkner <at> inode.at>
> Date: Fri, 18 Dec 2015 01:06:48 +0100
> Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 22086 <at> debbugs.gnu.org
> 
> I don't think that
> 
>     [Which files have you changed so far, and which new files have you written
>     so far?]
> 
> means that you have to describe those changes beforehand (but I could be
> wrong).

You need only to mention the file names where the changes were made,
that's all.

Message #32 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Wolfgang Jenkner <wjenkner <at> inode.at>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: eggert <at> cs.ucla.edu, dalias <at> aerifal.cx, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50;
 [PATCH] Integrate the musl hybrid malloc patch for elf systems
Date: Fri, 18 Dec 2015 14:15:07 +0100

On Fri, Dec 18 2015, Eli Zaretskii wrote:

>> From: Wolfgang Jenkner <wjenkner <at> inode.at>
>> Date: Fri, 18 Dec 2015 01:06:48 +0100
>> Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 22086 <at> debbugs.gnu.org
>> 
>> I don't think that
>> 
>>     [Which files have you changed so far, and which new files have you written
>>     so far?]
>> 
>> means that you have to describe those changes beforehand (but I could be
>> wrong).
>
> You need only to mention the file names where the changes were made,
> that's all.

Yes, but do you have to mention file names for changes which are not yet
in emacs?

Message #35 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Wolfgang Jenkner <wjenkner <at> inode.at>
Cc: eggert <at> cs.ucla.edu, dalias <at> aerifal.cx, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50;
 [PATCH] Integrate the musl hybrid malloc patch for elf systems
Date: Fri, 18 Dec 2015 16:38:31 +0200

> From: Wolfgang Jenkner <wjenkner <at> inode.at>
> Cc: eggert <at> cs.ucla.edu,  dalias <at> aerifal.cx,  22086 <at> debbugs.gnu.org
> Date: Fri, 18 Dec 2015 14:15:07 +0100
> 
> > You need only to mention the file names where the changes were made,
> > that's all.
> 
> Yes, but do you have to mention file names for changes which are not yet
> in emacs?

No.

Message #38 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Wolfgang Jenkner <wjenkner <at> inode.at>
Cc: Rich Felker <dalias <at> aerifal.cx>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Sun, 20 Dec 2015 14:33:22 -0800

While thinking over this patch I'd like to propose what should be a simpler 
approach. This new proposal is more radical, and so should not be applied to the 
emacs-25 branch, but it should make the port to musl etc. automatic.

The simpler approach is to remove gmalloc.c, and to use the system memory 
allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all platforms.

We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't work on 
Cygwin for some reason; and we can support the similar hybrid on Darwin, if it's 
still needed. But in neither approach should we override the system malloc; any 
Emacs-specific allocation function we define should be called (say) emalloc 
instead of malloc, so that it does not conflict with the system malloc. That 
way, we don't have to worry about name-space collisions, either at compile-time 
or at link-time.

If I'm wrong about gmalloc.c and it is still needed on some platforms for some 
reason, we can continue to use it, but it should define emalloc etc., and not 
attempt to override the C standard library.

Long ago as I recall, we really needed to override the C standard library on 
some platforms, due to the funny way in which undumped storage was made 
read-only. That need is obsolete, though, which should let us simplify things now.

Message #41 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Rich Felker <dalias <at> aerifal.cx>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Wolfgang Jenkner <wjenkner <at> inode.at>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Sun, 20 Dec 2015 20:59:52 -0500

On Sun, Dec 20, 2015 at 02:33:22PM -0800, Paul Eggert wrote:
> While thinking over this patch I'd like to propose what should be a
> simpler approach. This new proposal is more radical, and so should
> not be applied to the emacs-25 branch, but it should make the port
> to musl etc. automatic.
> 
> The simpler approach is to remove gmalloc.c, and to use the system
> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on
> all platforms.
> 
> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC
> wouldn't work on Cygwin for some reason; and we can support the
> similar hybrid on Darwin, if it's still needed. But in neither
> approach should we override the system malloc; any Emacs-specific
> allocation function we define should be called (say) emalloc instead
> of malloc, so that it does not conflict with the system malloc. That
> way, we don't have to worry about name-space collisions, either at
> compile-time or at link-time.
> 
> If I'm wrong about gmalloc.c and it is still needed on some
> platforms for some reason, we can continue to use it, but it should
> define emalloc etc., and not attempt to override the C standard
> library.
> 
> Long ago as I recall, we really needed to override the C standard
> library on some platforms, due to the funny way in which undumped
> storage was made read-only. That need is obsolete, though, which
> should let us simplify things now.

I don't object to this change if you can reliably ensure that nothing
in the pre-dump emacs will call malloc. When I looked, though, this
seemed to be very difficult; I had been looking (albeit casually) for
a clean solution to this problem for nearly a decade before hybrid
malloc showed up. Unless there's a quick solution I think switching to
hybrid malloc would be a first good step, but I like eliminating
gmalloc.c better.

Rich

Message #44 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Rich Felker <dalias <at> aerifal.cx>
Cc: Wolfgang Jenkner <wjenkner <at> inode.at>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Sun, 20 Dec 2015 18:37:35 -0800

Rich Felker wrote:
> I don't object to this change if you can reliably ensure that nothing
> in the pre-dump emacs will call malloc.

We can't reliably ensure that, I'm afraid, as Emacs might call some library 
function that in turn calls malloc. It might be a 3rd-party library that Emacs 
has no control over.

Why can't pre-dump emacs call malloc? Is it a performance thing or a correctness 
thing? If a performance thing it shouldn't be much of a problem, as we can 
arrange for Emacs to never directly call malloc, and that should be the vast 
majority of the calls.

Message #47 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Rich Felker <dalias <at> aerifal.cx>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Wolfgang Jenkner <wjenkner <at> inode.at>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Sun, 20 Dec 2015 21:51:17 -0500

On Sun, Dec 20, 2015 at 06:37:35PM -0800, Paul Eggert wrote:
> Rich Felker wrote:
> >I don't object to this change if you can reliably ensure that nothing
> >in the pre-dump emacs will call malloc.
> 
> We can't reliably ensure that, I'm afraid, as Emacs might call some
> library function that in turn calls malloc. It might be a 3rd-party
> library that Emacs has no control over.
> 
> Why can't pre-dump emacs call malloc? Is it a performance thing or a
> correctness thing? If a performance thing it shouldn't be much of a
> problem, as we can arrange for Emacs to never directly call malloc,
> and that should be the vast majority of the calls.

It's a correctness thing. The dumper can only dump the brk heap that's
contiguous with .data/.bss (and even that's not contiguous on modern
hardened kernels). This is why fake brk in static storage has to be
used by hybrid malloc. But even if the system malloc did guarantee
contiguity, it's not valid to keep using heap data structures from one
instance of the allocator with a new instance of the allocator, and
there are different failure modes for static and dynamic linking, but
both are bad.

In practice it _might_ work to varying degrees, but it's fundamentally
fragile and wrong.

Rich

Message #50 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: Paul Eggert <eggert <at> cs.ucla.edu>, Wolfgang Jenkner <wjenkner <at> inode.at>
Cc: Rich Felker <dalias <at> aerifal.cx>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Sun, 20 Dec 2015 22:37:24 -0500

On 12/20/2015 5:33 PM, Paul Eggert wrote:
> While thinking over this patch I'd like to propose what should be a
> simpler approach. This new proposal is more radical, and so should not
> be applied to the emacs-25 branch, but it should make the port to musl
> etc. automatic.
>
> The simpler approach is to remove gmalloc.c, and to use the system
> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all
> platforms.
>
> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't
> work on Cygwin for some reason; and we can support the similar hybrid on
> Darwin, if it's still needed.

SYSTEM_MALLOC doesn't work on Cygwin, largely because Cygwin's malloc 
doesn't support malloc_set_state and malloc_get_state.  There may be 
other problems too.  (It's been a while since I tried it.)

Ken

Message #53 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: wjenkner <at> inode.at, dalias <at> aerifal.cx, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50;
 [PATCH] Integrate the musl hybrid malloc patch for elf systems
Date: Mon, 21 Dec 2015 05:44:01 +0200

> From: Paul Eggert <eggert <at> cs.ucla.edu>
> Date: Sun, 20 Dec 2015 14:33:22 -0800
> Cc: Rich Felker <dalias <at> aerifal.cx>, 22086 <at> debbugs.gnu.org
> 
> The simpler approach is to remove gmalloc.c, and to use the system memory 
> allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all platforms.
> 
> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't work on 
> Cygwin for some reason; and we can support the similar hybrid on Darwin, if it's 
> still needed. But in neither approach should we override the system malloc; any 
> Emacs-specific allocation function we define should be called (say) emalloc 
> instead of malloc, so that it does not conflict with the system malloc. That 
> way, we don't have to worry about name-space collisions, either at compile-time 
> or at link-time.
> 
> If I'm wrong about gmalloc.c and it is still needed on some platforms for some 
> reason, we can continue to use it, but it should define emalloc etc., and not 
> attempt to override the C standard library.

Wouldn't memory allocated with emallooc conflict with uses of malloc
in the startup code?

Other than that, I'm okay with leaving gmalloc.c's allocator under a
different name.  Removing it is something we should consider
separately.

> Long ago as I recall, we really needed to override the C standard library on 
> some platforms, due to the funny way in which undumped storage was made 
> read-only. That need is obsolete, though, which should let us simplify things now.

Why is it obsolete?

Message #56 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Rich Felker <dalias <at> aerifal.cx>
To: Ken Brown <kbrown <at> cornell.edu>
Cc: Wolfgang Jenkner <wjenkner <at> inode.at>, Paul Eggert <eggert <at> cs.ucla.edu>,
 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Sun, 20 Dec 2015 23:06:28 -0500

On Sun, Dec 20, 2015 at 10:37:24PM -0500, Ken Brown wrote:
> On 12/20/2015 5:33 PM, Paul Eggert wrote:
> >While thinking over this patch I'd like to propose what should be a
> >simpler approach. This new proposal is more radical, and so should not
> >be applied to the emacs-25 branch, but it should make the port to musl
> >etc. automatic.
> >
> >The simpler approach is to remove gmalloc.c, and to use the system
> >memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all
> >platforms.
> >
> >We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't
> >work on Cygwin for some reason; and we can support the similar hybrid on
> >Darwin, if it's still needed.
> 
> SYSTEM_MALLOC doesn't work on Cygwin, largely because Cygwin's
> malloc doesn't support malloc_set_state and malloc_get_state.  There
> may be other problems too.  (It's been a while since I tried it.)

I don't see how this is possible; malloc_[gs]et_state do not exist on
other systems either. Presumably this is some hack needed for the
dumper, which wouldn't be needed if malloc weren't used pre-dumping.

Rich

Message #59 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Rich Felker <dalias <at> aerifal.cx>
Cc: Wolfgang Jenkner <wjenkner <at> inode.at>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Mon, 21 Dec 2015 03:10:04 -0800

Rich Felker wrote:
> In practice it_might_  work to varying degrees, but it's fundamentally
> fragile and wrong.

Yes, of course. But I don't see how it's any more fundamentally fragile and 
wrong than what we're doing already, as we cannot prevent library functions from 
calling library allocators.

> The dumper can only dump the brk heap that's
> contiguous with .data/.bss (and even that's not contiguous on modern
> hardened kernels).

That should be fixed, then.

Message #62 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: wjenkner <at> inode.at, dalias <at> aerifal.cx, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Mon, 21 Dec 2015 03:18:03 -0800

Eli Zaretskii wrote:
> Wouldn't memory allocated with emallooc conflict with uses of malloc
> in the startup code?

It shouldn't conflict, as long as memory allocated by the system malloc is freed 
by the system free, and memory allocated with emalloc is freed by efree. If 
efree is the hybrid implementation, it'd even be OK to combine the system malloc 
with efree, or to combine emalloc with the system free in some cases. Unless I'm 
misunderstanding the question?

>> >Long ago as I recall, we really needed to override the C standard library on
>> >some platforms, due to the funny way in which undumped storage was made
>> >read-only. That need is obsolete, though, which should let us simplify things now.
> Why is it obsolete?

Formerly Emacs made both static storage and storage allocated via malloc 
read-only in the dumped Emacs, for efficiency reasons. As I recall, this 
required replacing the system malloc with gmalloc. This approach had a lot of 
problems though, and we stopped making that storage read-only a while ago.

Message #65 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: Rich Felker <dalias <at> aerifal.cx>
Cc: Wolfgang Jenkner <wjenkner <at> inode.at>, Paul Eggert <eggert <at> cs.ucla.edu>,
 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Mon, 21 Dec 2015 07:24:00 -0500

On 12/20/2015 11:06 PM, Rich Felker wrote:
> On Sun, Dec 20, 2015 at 10:37:24PM -0500, Ken Brown wrote:
>> On 12/20/2015 5:33 PM, Paul Eggert wrote:
>>> While thinking over this patch I'd like to propose what should be a
>>> simpler approach. This new proposal is more radical, and so should not
>>> be applied to the emacs-25 branch, but it should make the port to musl
>>> etc. automatic.
>>>
>>> The simpler approach is to remove gmalloc.c, and to use the system
>>> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all
>>> platforms.
>>>
>>> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't
>>> work on Cygwin for some reason; and we can support the similar hybrid on
>>> Darwin, if it's still needed.
>>
>> SYSTEM_MALLOC doesn't work on Cygwin, largely because Cygwin's
>> malloc doesn't support malloc_set_state and malloc_get_state.  There
>> may be other problems too.  (It's been a while since I tried it.)
>
> I don't see how this is possible; malloc_[gs]et_state do not exist on
> other systems either. Presumably this is some hack needed for the
> dumper, which wouldn't be needed if malloc weren't used pre-dumping.

You're right.  I wasn't thinking clearly.  But several years ago, before 
Cygwin started putting the heap in high memory, there were still issues 
that made it impossible to use the system malloc.  I've forgotten the 
details.

Ken

Message #68 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Wolfgang Jenkner <wjenkner <at> inode.at>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Rich Felker <dalias <at> aerifal.cx>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50;
 [PATCH] Integrate the musl hybrid malloc patch for elf systems
Date: Mon, 21 Dec 2015 16:14:47 +0100

On Sun, Dec 20 2015, Paul Eggert wrote:

> The simpler approach is to remove gmalloc.c, and to use the system
> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on
> all platforms.

The system malloc can't manage the "static heap" array implemented in
sheap.c (because usually there's no malloc_hook), so, presumably this
would be ditched as well.  So... I wonder what you are proposing here.

Message #71 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: wjenkner <at> inode.at, dalias <at> aerifal.cx, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Mon, 21 Dec 2015 17:37:28 +0200

> Cc: wjenkner <at> inode.at, dalias <at> aerifal.cx, 22086 <at> debbugs.gnu.org
> From: Paul Eggert <eggert <at> cs.ucla.edu>
> Date: Mon, 21 Dec 2015 03:18:03 -0800
> 
> Eli Zaretskii wrote:
> > Wouldn't memory allocated with emallooc conflict with uses of malloc
> > in the startup code?
> 
> It shouldn't conflict, as long as memory allocated by the system malloc is freed 
> by the system free, and memory allocated with emalloc is freed by
> efree.

I don't see how this could be arranged.  E.g., the environ array might
be created by the startup code, and then the application could
manipulate it with setenv and unsetenv.  The former will use the libc
malloc, the latter emalloc/efree.

Message #74 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Wolfgang Jenkner <wjenkner <at> inode.at>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Rich Felker <dalias <at> aerifal.cx>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50;
 [PATCH] Integrate the musl hybrid malloc patch for elf systems
Date: Mon, 21 Dec 2015 16:46:28 +0100

On Mon, Dec 21 2015, Wolfgang Jenkner wrote:

> sheap.c (because usually there's no malloc_hook), so, presumably this
--------------------------------------^
I meant __morecore, sorry.

Message #77 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Wolfgang Jenkner <wjenkner <at> inode.at>
Cc: Rich Felker <dalias <at> aerifal.cx>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Mon, 21 Dec 2015 09:06:20 -0800

Wolfgang Jenkner wrote:
>> The simpler approach is to remove gmalloc.c, and to use the system
>> >memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on
>> >all platforms.
> The system malloc can't manage the "static heap" array implemented in
> sheap.c (because usually there's no malloc_hook), so, presumably this
> would be ditched as well.  So... I wonder what you are proposing here.

If we don't need the static heap, then let's ditch it. If we do need it, then 
let's have our own allocator (named emalloc, say), which uses the static heap 
and/or the system malloc as needed. Regardless, we shouldn't be trying to 
redefine 'malloc', nor should we be supplying our own allocator merely because 
that was a good idea back in 1989.

Message #80 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: wjenkner <at> inode.at, dalias <at> aerifal.cx, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Mon, 21 Dec 2015 09:11:14 -0800

Eli Zaretskii wrote:
>> It shouldn't conflict, as long as memory allocated by the system malloc is freed
>> >by the system free, and memory allocated with emalloc is freed by
>> >efree.
> I don't see how this could be arranged.  E.g., the environ array might
> be created by the startup code, and then the application could
> manipulate it with setenv and unsetenv.  The former will use the libc
> malloc, the latter emalloc/efree.

If by "startup code" you mean the C library, then it's not a problem, as 
portable C code already cannot free environ or environ[0] or environ[1] ... (and 
Emacs respects this, both when run as temacs and when run as dumped). If by 
"startup code" you mean something else, then I'm afraid I'm not following the 
example.

Message #83 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Wolfgang Jenkner <wjenkner <at> inode.at>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Rich Felker <dalias <at> aerifal.cx>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50;
 [PATCH] Integrate the musl hybrid malloc patch for elf systems
Date: Mon, 21 Dec 2015 18:28:19 +0100

On Mon, Dec 21 2015, Paul Eggert wrote:

> Wolfgang Jenkner wrote:
>>> The simpler approach is to remove gmalloc.c, and to use the system
>>> >memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on
>>> >all platforms.
>> The system malloc can't manage the "static heap" array implemented in
>> sheap.c (because usually there's no malloc_hook), so, presumably this
>> would be ditched as well.  So... I wonder what you are proposing here.
>
> If we don't need the static heap, then let's ditch it.

Is it needed for what you have in mind?

> If we do need
> it, then let's have our own allocator (named emalloc, say), which uses
> the static heap and/or the system malloc as needed.

Well, that's exactly what hybrid malloc does.

> Regardless, we
> shouldn't be trying to redefine 'malloc', nor should we be supplying
> our own allocator merely because that was a good idea back in 1989.

Isn't this a minor problem since it concerns only those systems which
will still need the malloc from src/gmalloc.c after dumping (non-ELF
unix-like systems)?

Message #86 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Rich Felker <dalias <at> aerifal.cx>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Wolfgang Jenkner <wjenkner <at> inode.at>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Mon, 21 Dec 2015 13:01:01 -0500

On Mon, Dec 21, 2015 at 03:10:04AM -0800, Paul Eggert wrote:
> Rich Felker wrote:
> >In practice it_might_  work to varying degrees, but it's fundamentally
> >fragile and wrong.
> 
> Yes, of course. But I don't see how it's any more fundamentally
> fragile and wrong than what we're doing already, as we cannot
> prevent library functions from calling library allocators.

Unless the library code is static-linked to emacs, there's only an
issue if emacs actually saves a direct/indirect reference to the
allocated memory. If the library allocates it internally (e.g. in its
global ctors or init functions, with the pointer(s) saved to static
storage in the library) then all such references will be lost during
dumping and it doesn't matter whatsoever whether the memory is
properly dumped and available at runtime later.

Rich

Message #89 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Colascione <dancol <at> dancol.org>
To: Rich Felker <dalias <at> aerifal.cx>, Ken Brown <kbrown <at> cornell.edu>
Cc: Wolfgang Jenkner <wjenkner <at> inode.at>, 22086 <at> debbugs.gnu.org,
 Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Mon, 21 Dec 2015 12:08:46 -0800

[Message part 1 (text/plain, inline)]

On 12/20/2015 08:06 PM, Rich Felker wrote:
> On Sun, Dec 20, 2015 at 10:37:24PM -0500, Ken Brown wrote:
>> On 12/20/2015 5:33 PM, Paul Eggert wrote:
>>> While thinking over this patch I'd like to propose what should be a
>>> simpler approach. This new proposal is more radical, and so should not
>>> be applied to the emacs-25 branch, but it should make the port to musl
>>> etc. automatic.
>>>
>>> The simpler approach is to remove gmalloc.c, and to use the system
>>> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all
>>> platforms.
>>>
>>> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't
>>> work on Cygwin for some reason; and we can support the similar hybrid on
>>> Darwin, if it's still needed.
>>
>> SYSTEM_MALLOC doesn't work on Cygwin, largely because Cygwin's
>> malloc doesn't support malloc_set_state and malloc_get_state.  There
>> may be other problems too.  (It's been a while since I tried it.)
> 
> I don't see how this is possible; malloc_[gs]et_state do not exist on
> other systems either. Presumably this is some hack needed for the
> dumper, which wouldn't be needed if malloc weren't used pre-dumping.

We really shouldn't be dumping the native heap at all, really.
Eventually, Emacs should be a position-independent executable (as should
every other program), and to unexec a PIE, we need to emit relocations,
and to emit relocations, we need to know where all the pointers are. We
can't do that if the internal heap structure is opaque to us.

[signature.asc (application/pgp-signature, attachment)]

Message #92 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Rich Felker <dalias <at> aerifal.cx>
To: Daniel Colascione <dancol <at> dancol.org>
Cc: Wolfgang Jenkner <wjenkner <at> inode.at>, 22086 <at> debbugs.gnu.org,
 Ken Brown <kbrown <at> cornell.edu>, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Mon, 21 Dec 2015 15:49:13 -0500

On Mon, Dec 21, 2015 at 12:08:46PM -0800, Daniel Colascione wrote:
> On 12/20/2015 08:06 PM, Rich Felker wrote:
> > On Sun, Dec 20, 2015 at 10:37:24PM -0500, Ken Brown wrote:
> >> On 12/20/2015 5:33 PM, Paul Eggert wrote:
> >>> While thinking over this patch I'd like to propose what should be a
> >>> simpler approach. This new proposal is more radical, and so should not
> >>> be applied to the emacs-25 branch, but it should make the port to musl
> >>> etc. automatic.
> >>>
> >>> The simpler approach is to remove gmalloc.c, and to use the system
> >>> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined on all
> >>> platforms.
> >>>
> >>> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC wouldn't
> >>> work on Cygwin for some reason; and we can support the similar hybrid on
> >>> Darwin, if it's still needed.
> >>
> >> SYSTEM_MALLOC doesn't work on Cygwin, largely because Cygwin's
> >> malloc doesn't support malloc_set_state and malloc_get_state.  There
> >> may be other problems too.  (It's been a while since I tried it.)
> > 
> > I don't see how this is possible; malloc_[gs]et_state do not exist on
> > other systems either. Presumably this is some hack needed for the
> > dumper, which wouldn't be needed if malloc weren't used pre-dumping.
> 
> We really shouldn't be dumping the native heap at all, really.
> Eventually, Emacs should be a position-independent executable (as should
> every other program), and to unexec a PIE, we need to emit relocations,
> and to emit relocations, we need to know where all the pointers are. We
> can't do that if the internal heap structure is opaque to us.

Actually you can, because the internal heap structure is not what you
need to dump. The "lisp heap" is what you need to dump, and it's
walkable in the same way you walk it now for garbage collection
purposes. It should be possible to adapt the GC code into a dumper
that dumps only the lisp data (with relocations in a form emacs can
internally process, i.e. not relying on writing a new executable
binary with ELF-level or other system-specific relocs) and no unwanted
additional state; then, even static linking should work correctly.

Rich

Message #95 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Colascione <dancol <at> dancol.org>
To: Rich Felker <dalias <at> aerifal.cx>
Cc: Wolfgang Jenkner <wjenkner <at> inode.at>, 22086 <at> debbugs.gnu.org,
 Ken Brown <kbrown <at> cornell.edu>, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#22086: 25.1.50;
 [PATCH] Integrate the musl hybrid malloc patch for elf systems
Date: Mon, 21 Dec 2015 12:58:37 -0800

On December 21, 2015 12:49:13 PM PST, Rich Felker <dalias <at> aerifal.cx> wrote:
>On Mon, Dec 21, 2015 at 12:08:46PM -0800, Daniel Colascione wrote:
>> On 12/20/2015 08:06 PM, Rich Felker wrote:
>> > On Sun, Dec 20, 2015 at 10:37:24PM -0500, Ken Brown wrote:
>> >> On 12/20/2015 5:33 PM, Paul Eggert wrote:
>> >>> While thinking over this patch I'd like to propose what should be
>a
>> >>> simpler approach. This new proposal is more radical, and so
>should not
>> >>> be applied to the emacs-25 branch, but it should make the port to
>musl
>> >>> etc. automatic.
>> >>>
>> >>> The simpler approach is to remove gmalloc.c, and to use the
>system
>> >>> memory allocator, i.e., to behave as if SYSTEM_MALLOC is defined
>on all
>> >>> platforms.
>> >>>
>> >>> We can still support hybrid malloc for Cygwin, if SYSTEM_MALLOC
>wouldn't
>> >>> work on Cygwin for some reason; and we can support the similar
>hybrid on
>> >>> Darwin, if it's still needed.
>> >>
>> >> SYSTEM_MALLOC doesn't work on Cygwin, largely because Cygwin's
>> >> malloc doesn't support malloc_set_state and malloc_get_state. 
>There
>> >> may be other problems too.  (It's been a while since I tried it.)
>> > 
>> > I don't see how this is possible; malloc_[gs]et_state do not exist
>on
>> > other systems either. Presumably this is some hack needed for the
>> > dumper, which wouldn't be needed if malloc weren't used
>pre-dumping.
>> 
>> We really shouldn't be dumping the native heap at all, really.
>> Eventually, Emacs should be a position-independent executable (as
>should
>> every other program), and to unexec a PIE, we need to emit
>relocations,
>> and to emit relocations, we need to know where all the pointers are.
>We
>> can't do that if the internal heap structure is opaque to us.
>
>Actually you can, because the internal heap structure is not what you
>need to dump. The "lisp heap" is what you need to dump, and it's
>walkable in the same way you walk it now for garbage collection
>purposes. It should be possible to adapt the GC code into a dumper
>that dumps only the lisp data (with relocations in a form emacs can
>internally process, i.e. not relying on writing a new executable
>binary with ELF-level or other system-specific relocs) and no unwanted
>additional state; then, even static linking should work correctly.
>
>Rich

Sure. That's the XEmacs portable dumper approach, and it works all right. I'm just worried that we might have implicit dependencies on the C heap being preserved across unexec.

The same code we use to relocate when Emacs is a PIE would also help us do GC compaction. (We'd have to pin conservatively reachable roots.)

Message #98 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Rich Felker <dalias <at> aerifal.cx>
Cc: Wolfgang Jenkner <wjenkner <at> inode.at>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Wed, 23 Dec 2015 00:24:46 -0800

Rich Felker wrote:
> Unless the library code is static-linked to emacs, there's only an
> issue if emacs actually saves a direct/indirect reference to the
> allocated memory.

It shouldn't be a problem, then. I doubt whether there are many such instances; 
if there are any, we can arrange for them to be copied before dumping or 
recreated after.

Message #101 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Wolfgang Jenkner <wjenkner <at> inode.at>
Cc: Rich Felker <dalias <at> aerifal.cx>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid malloc
 patch for elf systems
Date: Wed, 23 Dec 2015 00:31:00 -0800

Wolfgang Jenkner wrote:

>> If we don't need the static heap, then let's ditch it.
>
> Is it needed for what you have in mind?

I hope not. Perhaps it is needed for Cygwin, for reasons I don't understand. If 
it is needed then obviously we should keep it, and use the hybrid allocator. If 
not, not.

>> Regardless, we
>> shouldn't be trying to redefine 'malloc', nor should we be supplying
>> our own allocator merely because that was a good idea back in 1989.
>
> Isn't this a minor problem since it concerns only those systems which
> will still need the malloc from src/gmalloc.c after dumping (non-ELF
> unix-like systems)?

Yes, it's a minor problem, in that none of these systems actually need gmalloc.c 
any more.  If I understand things correctly, the main "problem" here (and it is 
a good problem to have) is that we need to carefully excise gmalloc.c out of 
Emacs, without breaking things.

Message #104 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Wolfgang Jenkner <wjenkner <at> inode.at>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Daniel Colascione <dancol <at> dancol.org>,
 Rich Felker <dalias <at> aerifal.cx>, 22086 <at> debbugs.gnu.org,
 Ken Brown <kbrown <at> cornell.edu>
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Sat, 30 Jan 2016 01:17:21 -0800

[Message part 1 (text/plain, inline)]

The recent emacs-devel thread "Removal of unexec support" has raised the 
priority of this bug, so I redid the patches to separate out Rich Felker's 
contribution, which is so small as to not require copyright papers, and fixed 
several problems I found with the resulting approach. I came up with the 
attached set of patches relative to commit 
ef760b899ad89f941f552ed2d3ac9e45156f3e3c. I would like to commit this patch set 
to the emacs-25 branch soon, and am sending this email to give you (particularly 
Eli) a heads-up about this.

These patches attempt to be more conservative than the other alternatives 
discussed in Bug#22086. They don't try to build a better dumper or remove 
gmalloc.c or anything like that. All they try to do, is to disentangle Emacs 
from glibc malloc internals, by renaming functions whose APIs are no longer 
compatible with glibc, and by using glibc's <malloc.h> rather than guessing what 
it will say, and that sort of thing. The goal is for the resulting Emacs to not 
only port to musl, but also to port to future glibc with less likelihood of trouble.

[0001-Internal-linkage-for-gmalloc-etc.-if-HYBRID_MALLOC.patch (text/x-diff, attachment)]

[0002-Link-temacs-with-gnulib-compiled-with-Demacs.patch (text/x-diff, attachment)]

[0003-unexelf.c-hook-to-support-HYBRID_MALLOC-on-ELF.patch (text/x-diff, attachment)]

[0004-Add-musl-patch-to-support-HYBRID_MALLOC-on-elf-syste.patch (text/x-diff, attachment)]

[0005-Pacify-GCC-on-extern-decls.patch (text/x-diff, attachment)]

[0006-Report-static-heap-usage-on-non-Cygwin-too.patch (text/x-diff, attachment)]

[0007-Pacify-enable-gcc-warnings-when-HYBRID_MALLOC.patch (text/x-diff, attachment)]

[0008-src-alloc.c-Include-sheap.h.patch (text/x-diff, attachment)]

[0009-Include-malloc.h-when-advisable.patch (text/x-diff, attachment)]

[0010-Build-lib-e-.o-only-on-platforms-that-need-it.patch (text/x-diff, attachment)]

[0011-Fix-extern-symbols-defined-and-not-used.patch (text/x-diff, attachment)]

[0012-Shrink-static-heap-a-bit.patch (text/x-diff, attachment)]

Message #107 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: wjenkner <at> inode.at, dalias <at> aerifal.cx, 22086 <at> debbugs.gnu.org,
 kbrown <at> cornell.edu, dancol <at> dancol.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Sat, 30 Jan 2016 11:40:27 +0200

> Cc: 22086 <at> debbugs.gnu.org, Rich Felker <dalias <at> aerifal.cx>,
>  Daniel Colascione <dancol <at> dancol.org>, Ken Brown <kbrown <at> cornell.edu>,
>  Eli Zaretskii <eliz <at> gnu.org>
> From: Paul Eggert <eggert <at> cs.ucla.edu>
> Date: Sat, 30 Jan 2016 01:17:21 -0800
> 
> The recent emacs-devel thread "Removal of unexec support" has raised the 
> priority of this bug, so I redid the patches to separate out Rich Felker's 
> contribution, which is so small as to not require copyright papers, and fixed 
> several problems I found with the resulting approach. I came up with the 
> attached set of patches relative to commit 
> ef760b899ad89f941f552ed2d3ac9e45156f3e3c. I would like to commit this patch set 
> to the emacs-25 branch soon, and am sending this email to give you (particularly 
> Eli) a heads-up about this.

I'm sorry, I can't afford testing this large patchset at this time,
while preparations to the pretest are under way.  I also don't think
such pervasive changes should be done on the release branch.  I think
when the time comes you should commit this to the master branch, not
to emacs-25.  In the meantime, a public feature branch will allow
people to try it and see if it breaks anything.

Thanks.

> These patches attempt to be more conservative than the other alternatives 
> discussed in Bug#22086. They don't try to build a better dumper or remove 
> gmalloc.c or anything like that. All they try to do, is to disentangle Emacs 
> from glibc malloc internals, by renaming functions whose APIs are no longer 
> compatible with glibc, and by using glibc's <malloc.h> rather than guessing what 
> it will say, and that sort of thing. The goal is for the resulting Emacs to not 
> only port to musl, but also to port to future glibc with less likelihood of trouble.

As the issue with glibc is only relevant to GNU/Linux systems, I
wonder if a solution that is limited to some file(s) specific to that
target could be possible.  Even then I'd hesitate to do this on the
release branch, since making Emacs less stable even on that single
platform will most probably delay the v25.1 release too much -- we
cannot possibly release Emacs 25 that is not stable on GNU/Linux.  But
at least we won't need to review and debug the results of this on
platforms that don't need these changes at all.

The glibc guys said the change won't happen too soon, so I see no
reason to delay Emacs 25 on behalf of this issue.

Thanks again for doing this work.

Message #112 received at 22086-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: wjenkner <at> inode.at, dalias <at> aerifal.cx, dancol <at> dancol.org, kbrown <at> cornell.edu,
 22086-done <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Sat, 30 Jan 2016 16:43:28 -0800

Eli Zaretskii wrote:
> I think
> when the time comes you should commit this to the master branch, not
> to emacs-25.

I think it's ready enough for the master branch, so I committed it there and 
will mark this bug as done.

I had to do a merge first, which was pretty painful, but that should be a 
different thread.

Message #115 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Rich Felker <dalias <at> aerifal.cx>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Eli Zaretskii <eliz <at> gnu.org>, Wolfgang Jenkner <wjenkner <at> inode.at>,
 Daniel Colascione <dancol <at> dancol.org>, 22086 <at> debbugs.gnu.org,
 Ken Brown <kbrown <at> cornell.edu>
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Sat, 30 Jan 2016 19:53:10 -0500

On Sat, Jan 30, 2016 at 01:17:21AM -0800, Paul Eggert wrote:
> The recent emacs-devel thread "Removal of unexec support" has raised
> the priority of this bug, so I redid the patches to separate out
> Rich Felker's contribution, which is so small as to not require
> copyright papers, and fixed several problems I found with the
> resulting approach. I came up with the attached set of patches
> relative to commit ef760b899ad89f941f552ed2d3ac9e45156f3e3c. I would
> like to commit this patch set to the emacs-25 branch soon, and am
> sending this email to give you (particularly Eli) a heads-up about
> this.

Hey! I'm very sorry I dropped the ball on assignment papers. I've been
busy with a lot of other stuff and forgot that was still pending. Let
me know if you'd still prefer to have them, but I agree totally that
my part of the patch was trivial and it was my opinion from the
beginning that I had no copyright claim to it.

Rich

Message #118 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Wolfgang Jenkner <wjenkner <at> inode.at>
To: 22086 <at> debbugs.gnu.org
Cc: eggert <at> cs.ucla.edu
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Sun, 31 Jan 2016 17:51:44 +0100

On Sat, Jan 30 2016, Paul Eggert wrote:

> I think it's ready enough for the master branch, so I committed it
> there and will mark this bug as done.

Thanks, I think patch 5 of my OP (for turning off direct use of mmap for
buffer allocations on FreeBSD) should also be applied.

When I do this, the build is correctly configured for hybrid malloc on
such a system, but linking temacs now fails with

  CCLD     temacs
gmalloc.o: In function `align':
/usr/opt/src/emacs-paul-test/src/gmalloc.c:440: undefined reference to `__after_morecore_hook'
/usr/opt/src/emacs-paul-test/src/gmalloc.c:441: undefined reference to `__after_morecore_hook'
gmalloc.o: In function `malloc_initialize_1':
/usr/opt/src/emacs-paul-test/src/gmalloc.c:553: undefined reference to `__malloc_initialize_hook'
/usr/opt/src/emacs-paul-test/src/gmalloc.c:554: undefined reference to `__malloc_initialize_hook'
collect2: error: ld returned 1 exit status
Makefile:594: recipe for target 'temacs' failed

Message #121 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Wolfgang Jenkner <wjenkner <at> inode.at>, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Sun, 31 Jan 2016 09:54:26 -0800

[Message part 1 (text/plain, inline)]

Wolfgang Jenkner wrote:

> Thanks, I think patch 5 of my OP (for turning off direct use of mmap for
> buffer allocations on FreeBSD) should also be applied.

Sorry, I missed that one. I don't recall why it's needed but I'll take your word 
for it. Done.

> /usr/opt/src/emacs-paul-test/src/gmalloc.c:440: undefined reference to `__after_morecore_hook'
> ...
> /usr/opt/src/emacs-paul-test/src/gmalloc.c:553: undefined reference to `__malloc_initialize_hook'

I installed the attached patch to try to fix these; please give it a try.

[0001-Port-new-hybrid-malloc-to-FreeBSD.patch (text/x-diff, attachment)]

Message #124 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Wolfgang Jenkner <wjenkner <at> inode.at>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Mon, 01 Feb 2016 16:15:43 +0100

On Sun, Jan 31 2016, Paul Eggert wrote:

> Wolfgang Jenkner wrote:
>
>> Thanks, I think patch 5 of my OP (for turning off direct use of mmap for
>> buffer allocations on FreeBSD) should also be applied.
>
> Sorry, I missed that one. I don't recall why it's needed but I'll take
> your word for it. Done.

Thanks.  Actually, I haven't bothered to check if the value of
use_mmap_for_buffers has an effect when hybrid malloc is enabled, but
it's a clean-up at least and the configure summary would be misleading.

> I installed the attached patch to try to fix these; please give it a try.

Thanks, this almost worked, I had just to double the static heap size.

In the ensuing successful build, I got the following message after
dumping:

18788992 of 33554432 static heap bytes used

However, this is with my standard configuration, which doesn't include
everything:

ssystem-configuration-features is a variable defined in ‘C source code’.
Its value is
"XPM JPEG TIFF GIF PNG IMAGEMAGICK SOUND NOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB X11 MODULES"

Message #127 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Wolfgang Jenkner <wjenkner <at> inode.at>
Cc: 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Mon, 1 Feb 2016 08:58:58 -0800

On 02/01/2016 07:15 AM, Wolfgang Jenkner wrote:
> I had just to double the static heap size.

OK, I installed a patch doing that into Emacs master.

> In the ensuing successful build, I got the following message after
> dumping:
>
> 18788992 of 33554432 static heap bytes used
>
> However, this is with my standard configuration, which doesn't include
> everything:
>
> ssystem-configuration-features is a variable defined in ‘C source code’.
> Its value is
> "XPM JPEG TIFF GIF PNG IMAGEMAGICK SOUND NOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB X11 MODULES"

I wonder why you're using way more static heap bytes than I am? I get 
this on Fedora 23 x86-64 when configured via './configure 
--enable-gcc-warnings --with-modules emacs_cv_var_doug_lea_malloc=no':

11892256 of 33554432 static heap bytes used
92482 pure bytes used

10765856 of 33554432 static heap bytes used
2781901 pure bytes used

The first is when building bootstrap-emacs, the second is when building 
emacs. On my build, system-configuration-features is "XPM JPEG TIFF GIF 
PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS NOTIFY ACL 
LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB 
TOOLKIT_SCROLL_BARS GTK3 X11 MODULES", which is a superset of your features.

Is the FreeBSD runtime calling malloc a lot on its own behalf? It might 
be interesting to find out why.

Message #130 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Wolfgang Jenkner <wjenkner <at> inode.at>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Mon, 01 Feb 2016 19:34:20 +0100

On Mon, Feb 01 2016, Paul Eggert wrote:

> On 02/01/2016 07:15 AM, Wolfgang Jenkner wrote:

>> 18788992 of 33554432 static heap bytes used

> I wonder why you're using way more static heap bytes than I am? I get
> this on Fedora 23 x86-64 when configured via
> './configure --enable-gcc-warnings --with-modules
> emacs_cv_var_doug_lea_malloc=no':
>
> 11892256 of 33554432 static heap bytes used
> 92482 pure bytes used
>
> 10765856 of 33554432 static heap bytes used
> 2781901 pure bytes used
>
> The first is when building bootstrap-emacs, the second is when
> building emacs.

Ah, indeed, the static heap size is much smaller in the final dump:

Pure-hashed: 24083 strings, 3801 vectors, 38521 conses, 3706 bytecodes, 102 others
Dumping under the name emacs
9663104 of 33554432 static heap bytes used
2777644 pure bytes used
Adding name emacs-25.1.50.1
: paxctl -zex emacs

Also, I get an only slightly larger number when dumping bootstrap-emacs
with the pre-compiled elisp files.  Perhaps, you got the first number
with the *.elc files still being around from a previous test?

Message #133 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Wolfgang Jenkner <wjenkner <at> inode.at>
Cc: eggert <at> cs.ucla.edu, 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Mon, 01 Feb 2016 21:38:09 +0200

> From: Wolfgang Jenkner <wjenkner <at> inode.at>
> Date: Mon, 01 Feb 2016 19:34:20 +0100
> Cc: 22086 <at> debbugs.gnu.org
> 
> On Mon, Feb 01 2016, Paul Eggert wrote:
> 
> > On 02/01/2016 07:15 AM, Wolfgang Jenkner wrote:
> 
> >> 18788992 of 33554432 static heap bytes used
> 
> > I wonder why you're using way more static heap bytes than I am? I get
> > this on Fedora 23 x86-64 when configured via
> > './configure --enable-gcc-warnings --with-modules
> > emacs_cv_var_doug_lea_malloc=no':
> >
> > 11892256 of 33554432 static heap bytes used
> > 92482 pure bytes used
> >
> > 10765856 of 33554432 static heap bytes used
> > 2781901 pure bytes used
> >
> > The first is when building bootstrap-emacs, the second is when
> > building emacs.
> 
> Ah, indeed, the static heap size is much smaller in the final dump:
> 
> Pure-hashed: 24083 strings, 3801 vectors, 38521 conses, 3706 bytecodes, 102 others
> Dumping under the name emacs
> 9663104 of 33554432 static heap bytes used
> 2777644 pure bytes used
> Adding name emacs-25.1.50.1
> : paxctl -zex emacs
> 
> Also, I get an only slightly larger number when dumping bootstrap-emacs
> with the pre-compiled elisp files.  Perhaps, you got the first number
> with the *.elc files still being around from a previous test?

You guys are aware that the amount of memory used when dumping depends
on such factors as the length of the absolute file name of the
directory where you build Emacs, right?

Message #136 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>, Wolfgang Jenkner <wjenkner <at> inode.at>
Cc: 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Mon, 1 Feb 2016 15:08:19 -0800

On 02/01/2016 11:38 AM, Eli Zaretskii wrote:
>> From: Wolfgang Jenkner <wjenkner <at> inode.at>
>> Date: Mon, 01 Feb 2016 19:34:20 +0100
>>
>>
>> Also, I get an only slightly larger number when dumping bootstrap-emacs
>> with the pre-compiled elisp files.  Perhaps, you got the first number
>> with the *.elc files still being around from a previous test?

Yes, that's probably it. Though I'm still surprised by the discrepancy. 
The biggest .el file is not that big. Is Emacs reading all the .el files 
into memory, into a single Emacs instance that buffers all the .el files 
at once?

> You guys are aware that the amount of memory used when dumping depends
> on such factors as the length of the absolute file name of the
> directory where you build Emacs, right?

Oh yes. Even doing the exact same dump twice can yield different 
results, presumably due to ASLR.

Message #139 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Wolfgang Jenkner <wjenkner <at> inode.at>
To: 22086 <at> debbugs.gnu.org
Cc: eggert <at> cs.ucla.edu
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Tue, 09 Feb 2016 15:55:23 +0100

[Message part 1 (text/plain, inline)]

On Sat, Jan 30 2016, Paul Eggert wrote:

> I think it's ready enough for the master branch, so I committed it
> there and will mark this bug as done.

I think, (g)calloc and hybrid_calloc are still needed, though?

[0001-Restore-the-calloc-family.patch (text/x-diff, inline)]

From cead27eae68b5d8d7db36b1a2965c541c4a7eb54 Mon Sep 17 00:00:00 2001
From: Wolfgang Jenkner <wjenkner <at> inode.at>
Date: Mon, 8 Feb 2016 17:16:15 +0100
Subject: [PATCH] Restore the calloc family.

* src/gmalloc.c (calloc, gcalloc, hybrid_calloc): Restore definitions.
They were lost in a4817d8 but calloc is still (marginally) used in
code statically liked with emacs, so hybrid_calloc is needed.
Also, in the non-hybrid case, we can't get rid of calloc anyway as
other libraries liked with emacs may need it.
* src/conf_post.h: Restore redefinition of calloc to hybrid_calloc.
---
 src/conf_post.h |  1 +
 src/gmalloc.c   | 15 ++++++++++++---
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/src/conf_post.h b/src/conf_post.h
index c5eec5a..2788abf 100644
--- a/src/conf_post.h
+++ b/src/conf_post.h
@@ -100,6 +100,7 @@ typedef bool bool_bf;
 #define malloc hybrid_malloc
 #define realloc hybrid_realloc
 #define aligned_alloc hybrid_aligned_alloc
+#define calloc hybrid_calloc
 #define free hybrid_free
 #endif
 #endif	/* HYBRID_MALLOC */
diff --git a/src/gmalloc.c b/src/gmalloc.c
index 0b76aee..dd18293 100644
--- a/src/gmalloc.c
+++ b/src/gmalloc.c
@@ -72,7 +72,7 @@ extern void *(*__morecore) (ptrdiff_t);
 #undef free
 #define malloc gmalloc
 #define realloc grealloc
-#define calloc do_not_call_me /* Emacs never calls calloc.  */
+#define calloc gcalloc
 #define aligned_alloc galigned_alloc
 #define free gfree
 #define malloc_info gmalloc_info
@@ -101,6 +101,8 @@ extern void *malloc (size_t size) ATTRIBUTE_MALLOC_SIZE ((1));
 /* Re-allocate the previously allocated block
    in ptr, making the new block SIZE bytes long.  */
 extern void *realloc (void *ptr, size_t size) ATTRIBUTE_ALLOC_SIZE ((2));
+/* Allocate NMEMB elements of SIZE bytes each, all initialized to 0.  */
+extern void *calloc (size_t nmemb, size_t size) ATTRIBUTE_MALLOC_SIZE ((1,2));
 /* Free a block.  */
 extern void free (void *ptr);
 
@@ -1465,7 +1467,6 @@ License along with this library.  If not, see <http://www.gnu.org/licenses/>.
 
 /* Allocate an array of NMEMB elements each SIZE bytes long.
    The entire array is initialized to zeros.  */
-#ifndef calloc
 void *
 calloc (size_t nmemb, size_t size)
 {
@@ -1483,7 +1484,6 @@ calloc (size_t nmemb, size_t size)
     return memset (result, 0, bytes);
   return result;
 }
-#endif
 /* Copyright (C) 1991, 1992, 1993, 1994, 1995 Free Software Foundation, Inc.
 This file is part of the GNU C Library.
 
@@ -1714,6 +1714,7 @@ valloc (size_t size)
 /* Declare system malloc and friends.  */
 extern void *malloc (size_t size);
 extern void *realloc (void *ptr, size_t size);
+extern void *calloc (size_t nmemb, size_t size);
 extern void free (void *ptr);
 #ifdef HAVE_ALIGNED_ALLOC
 extern void *aligned_alloc (size_t alignment, size_t size);
@@ -1732,6 +1733,14 @@ hybrid_malloc (size_t size)
   return gmalloc (size);
 }
 
+void *
+hybrid_calloc (size_t nmemb, size_t size)
+{
+  if (DUMPED)
+    return calloc (nmemb, size);
+  return gcalloc (nmemb, size);
+}
+
 void
 hybrid_free (void *ptr)
 {
-- 
2.7.0

Message #142 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: 22086 <at> debbugs.gnu.org, Wolfgang Jenkner <wjenkner <at> inode.at>
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Tue, 9 Feb 2016 15:31:40 -0800

> I think, (g)calloc and hybrid_calloc are still needed, though?

I suppose you're right, so I installed your patch in master. Though this 
stuff is all pretty iffy. Why does Emacs redefine calloc but not 
aligned_alloc or posix_memalign, for example? Is it because we know no 
library uses these newer allocators?

Anyway, I suppose it's better to play it safe and continue to redefine 
calloc, the way Emacs has done for decades, until we have a better way 
to do dumping and restoring and memory allocation for such.

Message #145 received at 22086 <at> debbugs.gnu.org (full text, mbox):

From: Wolfgang Jenkner <wjenkner <at> inode.at>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 22086 <at> debbugs.gnu.org
Subject: Re: bug#22086: 25.1.50; [PATCH] Integrate the musl hybrid mallo
Date: Wed, 10 Feb 2016 13:27:37 +0100

On Tue, Feb 09 2016, Paul Eggert wrote:

>> I think, (g)calloc and hybrid_calloc are still needed, though?
>
> I suppose you're right, so I installed your patch in master.

Thanks, I could have pushed it myself, I just posted it here to avoid
interfering with your plans concerning this stuff.

> this stuff is all pretty iffy. Why does Emacs redefine calloc but not
> aligned_alloc or posix_memalign, for example?

Here on FreeBSD 10, in the non-hybrid case both aligned_alloc and
posix_memalign are redefined; in the hybrid case there is only
hybrid_aligned_alloc:

/opt/src/emacs-calloc-non-hybrid;!2;: nm -g src/emacs | grep align
00000000005e4210 R QCalign_to
0000000000b83df8 D _aligned_blocks
0000000000b83df0 D _aligned_blocks_mutex
00000000005c92b0 T aligned_alloc
00000000005c8e70 T galigned_alloc
00000000005c90e0 T memalign
00000000005c90f0 T posix_memalign
/opt/src/emacs-calloc-non-hybrid;!3;: cd ../emacs-calloc-hybrid/
/opt/src/emacs-calloc-hybrid;!4;: nm -g src/emacs | grep align
00000000005e3380 R QCalign_to
                 U aligned_alloc@@FBSD_1.3
00000000005c71a0 T hybrid_aligned_alloc
/opt/src/emacs-calloc-hybrid;!5;: 

> Is it because we know no library uses these newer allocators?

Just for the record, as you know this stuff way better than I do:

IIUC, we need hybrid_ versions for all allocation functions that are
actually used in code statically linked with emacs (at least those which
somewhere refer to memory allocated before dumping, but I don't think
there's much point in making a distinction here).

On the other hand, in the non-hybrid case, we have to override all of
them (if they exist in libc), so that references in shared libraries
linked with emacs to, say, posix_memalign don't resolve to the libc
version.

> Anyway, I suppose it's better to play it safe and continue to redefine
> calloc, the way Emacs has done for decades, until we have a better way
> to do dumping and restoring and memory allocation for such.

In the hybrid case we just do #define calloc hybrid_calloc etc. in the
emacs sources, so that's not a problem.  On the other hand, in the
non-hybrid case, what has been done for decades (i.e. overriding the
libc implementation of calloc etc.) has turned out to be a problem after
all (bug#22085).

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.