GNU bug report logs - #22667
string->bytevector encoding error

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 22667 in the body.
You can then email your comments to 22667 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Josep Portella Florit <jpf <at> primfilat.com>
To: bug-guile <at> gnu.org
Subject: string->bytevector encoding error
Date: Sun, 14 Feb 2016 23:18:16 +0100

'string->bytevector' throws 'encoding-error' when the encoding is
"UTF-16" and the length of the string is greater than 128.  The same
for the encoding "UTF-32" and a string of length greater than 64.

Tested on Guile 2.0.11 and 2.1.2 with the same result.

How to reproduce:

(use-modules (ice-9 iconv))

;; These expressions evaluate as expected:
(string->bytevector (make-string 128 #\A) "UTF-16")
;; => #vu8(254 255 0 65 0 65 ...)
(string->bytevector (make-string 64 #\A) "UTF-32")
;; => #vu8(0 0 254 255 0 0 0 65 0 0 0 65 ...)

;; These expressions throw 'encoding-error':
(string->bytevector (make-string 129 #\A) "UTF-16")
(string->bytevector (make-string 65 #\A) "UTF-32")

Message #8 received at 22667 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: Josep Portella Florit <jpf <at> primfilat.com>
Cc: 22667 <at> debbugs.gnu.org, Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: bug#22667: string->bytevector encoding error
Date: Thu, 03 Mar 2016 12:55:55 -0500

Josep Portella Florit <jpf <at> primfilat.com> writes:

> 'string->bytevector' throws 'encoding-error' when the encoding is
> "UTF-16" and the length of the string is greater than 128.  The same
> for the encoding "UTF-32" and a string of length greater than 64.
>
> Tested on Guile 2.0.11 and 2.1.2 with the same result.
>
> How to reproduce:
>
> (use-modules (ice-9 iconv))
>
> ;; These expressions evaluate as expected:
> (string->bytevector (make-string 128 #\A) "UTF-16")
> ;; => #vu8(254 255 0 65 0 65 ...)
> (string->bytevector (make-string 64 #\A) "UTF-32")
> ;; => #vu8(0 0 254 255 0 0 0 65 0 0 0 65 ...)
>
> ;; These expressions throw 'encoding-error':
> (string->bytevector (make-string 129 #\A) "UTF-16")
> (string->bytevector (make-string 65 #\A) "UTF-32")

Thee bug is in 'display_string_using_iconv', introduced in commit
f4bc4e5934, which improperly assumes that a 256-byte buffer
(encoded_output) will be large enough to hold the results of converting
up to 256 bytes of UTF-8 (utf8_buf) to an arbitrary encoding using
'iconv'.  When this assumption fails, the call to 'iconv' on line 997

      done = iconv (id->output_cd, &input, &input_left,
		    &output, &output_left);

reports a failure due to the output buffer being too small, and the code
fails to handle this case.

To be continued...

     Mark

Message #11 received at 22667 <at> debbugs.gnu.org (full text, mbox):

From: Andy Wingo <wingo <at> pobox.com>
To: Josep Portella Florit <jpf <at> primfilat.com>
Cc: 22667 <at> debbugs.gnu.org
Subject: Re: bug#22667: string->bytevector encoding error
Date: Mon, 20 Jun 2016 18:07:21 +0200

On Sun 14 Feb 2016 23:18, Josep Portella Florit <jpf <at> primfilat.com> writes:

> 'string->bytevector' throws 'encoding-error' when the encoding is
> "UTF-16" and the length of the string is greater than 128.  The same
> for the encoding "UTF-32" and a string of length greater than 64.
>
> Tested on Guile 2.0.11 and 2.1.2 with the same result.
>
> How to reproduce:
>
> (use-modules (ice-9 iconv))
>
> ;; These expressions evaluate as expected:
> (string->bytevector (make-string 128 #\A) "UTF-16")
> ;; => #vu8(254 255 0 65 0 65 ...)
> (string->bytevector (make-string 64 #\A) "UTF-32")
> ;; => #vu8(0 0 254 255 0 0 0 65 0 0 0 65 ...)
>
> ;; These expressions throw 'encoding-error':
> (string->bytevector (make-string 129 #\A) "UTF-16")
> (string->bytevector (make-string 65 #\A) "UTF-32")

For what it's worth, this appears to be fixed in 2.1.3.  However we
should fix 2.0 as well.

Andy

Message #16 received at 22667-done <at> debbugs.gnu.org (full text, mbox):

From: Andy Wingo <wingo <at> pobox.com>
To: Josep Portella Florit <jpf <at> primfilat.com>
Cc: 22667-done <at> debbugs.gnu.org
Subject: Re: bug#22667: string->bytevector encoding error
Date: Sun, 07 Aug 2016 23:25:29 +0200

Hi,

On Mon 20 Jun 2016 18:07, Andy Wingo <wingo <at> pobox.com> writes:

> On Sun 14 Feb 2016 23:18, Josep Portella Florit <jpf <at> primfilat.com> writes:
>
>> (use-modules (ice-9 iconv))
>> ;; These expressions throw 'encoding-error':
>> (string->bytevector (make-string 129 #\A) "UTF-16")
>> (string->bytevector (make-string 65 #\A) "UTF-32")
>
> For what it's worth, this appears to be fixed in 2.1.3.  However we
> should fix 2.0 as well.

Fixed.  Thanks for the report, and thanks to Mark for diagnosing the
problem.

Andy.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.