GNU bug report logs - #21093
Web server: response bodies systematically loaded in memory

Previous Next

To reply to this bug, email your comments to 21093 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: bug-guile <at> gnu.org
Subject: Web server: response bodies systematically loaded in memory
Date: Mon, 20 Jul 2015 00:10:28 +0200

The ‘sanitize-response’ procedure systematically loads the whole
response body in memory, which causes obvious scalability issues (this
is in 2.0.11.)

In particular, when a request handler returns a port-taking procedure as
its second return value, ‘sanitize-request’ will just call that
procedure passing it a string output port.

This procedure should instead be called from the server implementation’s
‘write’ hook, but that would necessitate an API change.

Thoughts?

Ludo’.

Message #8 received at 21093 <at> debbugs.gnu.org (full text, mbox):

From: Andy Wingo <wingo <at> pobox.com>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: guile-devel <at> gnu.org, 21093 <at> debbugs.gnu.org
Subject: Re: bug#21093: Web server: response bodies systematically loaded in
 memory
Date: Fri, 24 Jun 2016 10:15:40 +0200

On Mon 20 Jul 2015 00:10, ludo <at> gnu.org (Ludovic Courtès) writes:

> The ‘sanitize-response’ procedure systematically loads the whole
> response body in memory, which causes obvious scalability issues (this
> is in 2.0.11.)
>
> In particular, when a request handler returns a port-taking procedure as
> its second return value, ‘sanitize-request’ will just call that
> procedure passing it a string output port.
>
> This procedure should instead be called from the server implementation’s
> ‘write’ hook, but that would necessitate an API change.
>
> Thoughts?

How would you set the Content-Length?  Just leave it off and set
Connection: close ?  Set it in the headers perhaps?  Then you have to
verify later, which I dunno how nice that is.  Maybe it is OK.

I would be happy to review patches :)  Cc guile-devel.

Andy

Message #11 received at 21093 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Andy Wingo <wingo <at> pobox.com>
Cc: guile-devel <at> gnu.org, 21093 <at> debbugs.gnu.org
Subject: Re: bug#21093: Web server: response bodies systematically loaded in
 memory
Date: Fri, 24 Jun 2016 10:58:21 +0200

Andy Wingo <wingo <at> pobox.com> skribis:

> On Mon 20 Jul 2015 00:10, ludo <at> gnu.org (Ludovic Courtès) writes:
>
>> The ‘sanitize-response’ procedure systematically loads the whole
>> response body in memory, which causes obvious scalability issues (this
>> is in 2.0.11.)
>>
>> In particular, when a request handler returns a port-taking procedure as
>> its second return value, ‘sanitize-request’ will just call that
>> procedure passing it a string output port.
>>
>> This procedure should instead be called from the server implementation’s
>> ‘write’ hook, but that would necessitate an API change.
>>
>> Thoughts?
>
> How would you set the Content-Length?  Just leave it off and set
> Connection: close ?  Set it in the headers perhaps?  Then you have to
> verify later, which I dunno how nice that is.  Maybe it is OK.

I think it could work this way:

  1. By default, provide no ‘Content-Length’ and force chunked encoding
     (so that the recipient can make sure it received everything.)

     This is useful for data generated on the fly.

  2. Provide an optional mechanism allowing the user to specify the
     content length upfront.

     Useful for statically-generated data that cannot fit in memory.

I haven’t thought yet about the actual API (I’ll be happy to do so as
time permits; to be clear, I don’t consider it a 2.0.12 blocker.)

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.