GNU bug report logs - #13936
Default to UTF-8 for most Emacs source files

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 13936 in the body.
You can then email your comments to 13936 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: bug-gnu-emacs <at> gnu.org
Cc: Kenichi Handa <handa <at> gnu.org>
Subject: Default to UTF-8 for most Emacs source files
Date: Tue, 12 Mar 2013 14:16:47 -0700

[Message part 1 (text/plain, inline)]

Severity: wishlist
Tags: patch

Attached is a proposed patch to change the encoding to UTF-8 for most Emacs
source files that contain non-ASCII characters.  This shouldn't affect
the Emacs build or maintenance procedures, as Emacs itself can use
either UTF-8 or non-UTF-8 encodings.  Switching to UTF-8 does make it
easier to use non-Emacs tools, such as 'grep -r'.

This patch is a followon to Bug#13880 but is a bit more ambitious as
it affects more files.  Like the patch in Bug#13880, the patch is a
bit tricky to view, since it contains mixed encodings.

A few files remain in non-UTF-8 encodings, for technical reasons; see
admin/notes/unicode.  Perhaps some of these can be converted to UTF-8
some day, but they'll need more work.

[utf-8.txt (text/plain, attachment)]

Message #8 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>, Kenichi Handa <handa <at> gnu.org>
Cc: 13936 <at> debbugs.gnu.org
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Tue, 12 Mar 2013 23:54:05 +0200

> Date: Tue, 12 Mar 2013 14:16:47 -0700
> From: Paul Eggert <eggert <at> cs.ucla.edu>
> 
> Attached is a proposed patch to change the encoding to UTF-8 for most Emacs
> source files that contain non-ASCII characters.  This shouldn't affect
> the Emacs build or maintenance procedures, as Emacs itself can use
> either UTF-8 or non-UTF-8 encodings.  Switching to UTF-8 does make it
> easier to use non-Emacs tools, such as 'grep -r'.
> 
> This patch is a followon to Bug#13880 but is a bit more ambitious as
> it affects more files.  Like the patch in Bug#13880, the patch is a
> bit tricky to view, since it contains mixed encodings.
> 
> A few files remain in non-UTF-8 encodings, for technical reasons; see
> admin/notes/unicode.  Perhaps some of these can be converted to UTF-8
> some day, but they'll need more work.

I'm not sure converting etc/HELLO is a good idea.  Perhaps Handa-san
could chime in and comment on that matter.

Message #11 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 13936 <at> debbugs.gnu.org
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Tue, 12 Mar 2013 23:04:59 +0100

Paul Eggert <eggert <at> cs.ucla.edu> writes:

> -  CJK variety: GB($AT*Fx(B,$A?*7"(B), BIG5($(0&x86(B,$(0DeBv(B), JIS($B855$(B,$B3+H/(B), KSC($(Cj*Q((B,$(CKR[!(B)
> +  CJK variety: GB(元气,开发), BIG5(元氣,開發), JIS(元気,開発), KSC(元氣,開發)

You cannot do that.  You are throwing away the charset difference which
is the whole point of the exercise.

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Message #14 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: 13936 <at> debbugs.gnu.org
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Tue, 12 Mar 2013 17:01:05 -0700

[Message part 1 (text/plain, inline)]

On 03/12/13 15:04, Andreas Schwab wrote:
> You are throwing away the charset difference which
> is the whole point of the exercise.

Thanks to you and Eli for catching that.  Revised patch attached;
it's like the original except that it does not alter etc/HELLO.

[utf-8.txt (text/plain, attachment)]

Message #17 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: handa <handa <at> gnu.org>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: 13936 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Thu, 14 Mar 2013 12:48:51 +0900

In article <m24nggjmus.fsf <at> igel.home>, Andreas Schwab <schwab <at> linux-m68k.org> writes:

> Paul Eggert <eggert <at> cs.ucla.edu> writes:
> > -  CJK variety: GB($AT*Fx(B,$A?*7"(B), BIG5($(0&x86(B,$(0DeBv(B), JIS($B855$(B,$B3+H/(B), KSC($(Cj*Q((B,$(CKR[!(B)
> > +  CJK variety: GB($B85]c(B,$A?*7"(B), BIG5($B85]f(B,$B3+b$(B), JIS($B855$(B,$B3+H/(B), KSC($B85]f(B,$B3+b$(B)

> You cannot do that.  You are throwing away the charset difference which
> is the whole point of the exercise.

Right.  When a file is in some legacy encoding such as
iso-2022-7bit, Emacs attached charset properties to proper
ranges of text, which works as a hint for selecting a proper
font especially for CJK characters.  The same can be said
for etc/tutorials/TUTORIAL.ja.  This file should be shown in
a proper Japanese font even for a user not in Japanese
lang. env.

---
Kenichi Handa
handa <at> gnu.org

Message #20 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: handa <handa <at> gnu.org>
Cc: 13936 <at> debbugs.gnu.org, Andreas Schwab <schwab <at> linux-m68k.org>
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Mon, 18 Mar 2013 00:22:28 -0700

On 03/13/2013 08:48 PM, handa wrote:
> The same can be said for etc/tutorials/TUTORIAL.ja.
> This file should be shown in a proper Japanese font
> even for a user not in Japanese lang. env.

For a file like TUTORIAL.ja, the Local Variables: section
can specify a preferred charset.  Wouldn't this address the issue for
non-Japanese users?  Something like this, say:

;;; Local Variables:
;;; coding: utf-8
;;; eval: (set-charset-priority 'japanese-jisx0208)
;;; End:

Or perhaps we can come up with a better way to specify a file-local
charset priority.

Message #23 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: handa <handa <at> gnu.org>, 13936 <at> debbugs.gnu.org
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Mon, 18 Mar 2013 10:43:36 +0100

Paul Eggert <eggert <at> cs.ucla.edu> writes:

> For a file like TUTORIAL.ja, the Local Variables: section
> can specify a preferred charset.  Wouldn't this address the issue for
> non-Japanese users?  Something like this, say:
>
> ;;; Local Variables:
> ;;; coding: utf-8
> ;;; eval: (set-charset-priority 'japanese-jisx0208)

That relies on enable-local-eval.  The default is to query, which is not
a good thing for a tutorial.

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Message #26 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: 13936 <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Mon, 18 Mar 2013 09:27:44 -0400

>> For a file like TUTORIAL.ja, the Local Variables: section
>> can specify a preferred charset.  Wouldn't this address the issue for
>> non-Japanese users?  Something like this, say:
>> 
>> ;;; Local Variables:
>> ;;; coding: utf-8
>> ;;; eval: (set-charset-priority 'japanese-jisx0208)

We could indeed do something like that in those cases where the
preferred charset is the same for the whole file.
[ Tho with only 2 semi-colons on each line, please.  ]

> That relies on enable-local-eval.  The default is to query, which is not
> a good thing for a tutorial.

That can easily be changed if we decide it's a good solution.


        Stefan

Message #29 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: 13936 <at> debbugs.gnu.org, Kenichi Handa <handa <at> gnu.org>
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Mon, 18 Mar 2013 13:40:10 -0700

Another thought is to add support for a local variable 'lang'
to specify the language, HTML-style, e.g.:

;; Local Variables:
;; coding: utf-8
;; lang: ja
;; End:

where the "ja" comes from the usual standard list (see Internet BCP 47
and <http://www.iana.org/assignments/language-subtag-registry>).
Emacs could use the language to set charset-priority on a per-file basis.
This would be friendlier to users than the
"eval: (set-charset-priority 'japanese-jisx0208)" approach
that I mentioned earlier.  But it'd be more implementation work.

In the meantime I will propose a stripped-down patch that does not
alter TUTORIAL.ja and the other CJK files; this should avoid the
charset problems that Mr. Handa mentioned.

Message #32 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: 13936 <at> debbugs.gnu.org
Cc: Kenichi Handa <handa <at> gnu.org>
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Mon, 18 Mar 2013 13:44:52 -0700

[Message part 1 (text/plain, inline)]

Attached is a simplified version of the patch.  It leaves
the CJK files alone, and converts just the non-CJK files
(cyrillic, hebrew, tibetan, etc.).  As before, it's a bit
tricky to view the patch since it contains a mixture of
encodings.

[encoding.txt (text/plain, attachment)]

Message #35 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Kenichi Handa <handa <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 13936 <at> debbugs.gnu.org
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Wed, 20 Mar 2013 17:18:10 +0900

In article <51477BAA.6020601 <at> cs.ucla.edu>, Paul Eggert <eggert <at> cs.ucla.edu> writes:

> In the meantime I will propose a stripped-down patch that does not
> alter TUTORIAL.ja and the other CJK files; this should avoid the
> charset problems that Mr. Handa mentioned.

Among CJK files, I think K(orean) files can be in UTF-8
without problem.  Are there any people familiar with Korean
situation?

---
Kenichi Handa
handa <at> gnu.org

Message #38 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Kenichi Handa <handa <at> gnu.org>
Cc: 13936 <at> debbugs.gnu.org
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Wed, 20 Mar 2013 09:43:38 -0700

On 03/20/13 01:18, Kenichi Handa wrote:
> Among CJK files, I think K(orean) files can be in UTF-8
> without problem.

It's easy enough to convert the K files to UTF-8 too, and I'll propose
a patch to do that in followup email.

> Are there any people familiar with Korean situation?

Sorry, I don't know.

For what it's worth, when I use Emacs to convert TUTORIAL.ko to
UTF-8 and back, the result is identical to the original, so no
information is lost by making that change.  (This is not true
for TUTORIAL.ja.)

I have another question.  Shouldn't it be OK to convert Elisp source
files such as leim/quail/japanese.el to UTF-8 as well?  Emacs
internally converts their text to UTF-8 while compiling them, so the
corresponding .elc files are in UTF-8 already, and there should be no
functional difference if we convert the .el files to UTF-8.

Converting these files to UTF-8 would fix an inconsistency in Emacs
behavior.  For example, if I visit the file leim/quail/japanese.el I see
this definition:

  (defvar quail-japanese-use-double-n nil
    "If non-nil, use type \"nn\" to insert ん.")

where the character 'ん' is displayed using code point 0x2473 in
charset japanese-jisx0208.  But if I *use* the above definition string,
by typing "C-h v quail-japanese-use-double-n RET", the help string
that I see has been translated to UTF-8, so Emacs displays that
character using code point 0x3093 in charset unicode instead.  It
would be better if the runtime behavior matched the source code, and
an easy way to do that would be to convert the source code to UTF-8.

Here is the list of the remaining .el files that I'd like to convert
to UTF-8:

	leim/quail/cyril-jis.el
	leim/quail/hanja-jis.el
	leim/quail/japanese.el
	leim/quail/py-punct.el
	leim/quail/pypunct-b5.el
	lisp/international/ja-dic-cnv.el
	lisp/international/ja-dic-utl.el
	lisp/international/kinsoku.el
	lisp/international/kkc.el
	lisp/international/titdic-cnv.el
	lisp/language/japan-util.el
	lisp/language/japanese.el
	lisp/term/x-win.el

x-win.el is a special case, since it has two "Kana: Fixme:" lines
talking about problems when converting to UTF-8 -- evidently these are
issues in our current setup anyway since Emacs converts the text to UTF-8
before compiling it.

Message #41 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Kenichi Handa <handa <at> gnu.org>
Cc: 13936 <at> debbugs.gnu.org
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Wed, 20 Mar 2013 09:48:12 -0700

[Message part 1 (text/plain, inline)]

Attached is a revised patch, which differs from the previous
one in that it also converts the K files as suggested by Mr. Handa.
The newly converted files are:

 etc/tutorials/TUTORIAL.ko
 leim/quail/hanja.el
 leim/quail/hanja3.el
 leim/quail/symbol-ksc.el

I've compressed the patch to save space.

[encoding.txt.gz (application/x-gzip, attachment)]

Message #46 received at 13936-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: 13936-done <at> debbugs.gnu.org
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Mon, 01 Apr 2013 18:20:12 -0700

I installed the most-recently-proposed patch
as trunk bzr 112210 and am marking this as done.

Message #49 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: 13936 <at> debbugs.gnu.org
Cc: eggert <at> cs.ucla.edu
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Wed, 03 Apr 2013 14:31:33 -0400

> I installed the most-recently-proposed patch
> as trunk bzr 112210 and am marking this as done.

BTW, I'd like to make it so that Elisp uses utf-8 by default.
I welcome suggestions of patches that try to do that while minimizing
the breakage that it might introduce.


        Stefan

Message #52 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 13936 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Wed, 03 Apr 2013 21:39:02 +0300

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Date: Wed, 03 Apr 2013 14:31:33 -0400
> Cc: eggert <at> cs.ucla.edu
> 
> > I installed the most-recently-proposed patch
> > as trunk bzr 112210 and am marking this as done.
> 
> BTW, I'd like to make it so that Elisp uses utf-8 by default.
> I welcome suggestions of patches that try to do that while minimizing
> the breakage that it might introduce.

I thought Handa-san was working on that, and he bumped into some
problem he needs to fix, before the infrastructure for this is ready.

Message #55 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 13936 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Wed, 03 Apr 2013 20:35:52 -0400

>> > I installed the most-recently-proposed patch
>> > as trunk bzr 112210 and am marking this as done.
>> BTW, I'd like to make it so that Elisp uses utf-8 by default.
>> I welcome suggestions of patches that try to do that while minimizing
>> the breakage that it might introduce.
> I thought Handa-san was working on that, and he bumped into some
> problem he needs to fix, before the infrastructure for this is ready.

I don't think Handa's work is related.  He's trying to improve the
performance of Emacs when handling utf-8 files, whereas my request is
for a change in behavior, regardless of performance.


        Stefan

Message #58 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Kenichi Handa <handa <at> gnu.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 13936 <at> debbugs.gnu.org, eliz <at> gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Sat, 06 Apr 2013 00:03:22 +0900

In article <jwvfvz7rv51.fsf-monnier+emacs <at> gnu.org>, Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

> > I thought Handa-san was working on that, and he bumped into some
> > problem he needs to fix, before the infrastructure for this is ready.

> I don't think Handa's work is related.  He's trying to improve the
> performance of Emacs when handling utf-8 files, whereas my request is
> for a change in behavior, regardless of performance.

If the performance is not the problem,

(with-coding-priority '(utf-8)
  (load-file "_FILE_NAME_"))

will work.  And, as I've just fixed the problem and
installed the optimization for ASCII and UTF-8 file reading,
the above code should run faster than before.

---
Kenichi Handa
handa <at> gnu.org

Message #61 received at 13936 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: Kenichi Handa <handa <at> gnu.org>
Cc: 13936 <at> debbugs.gnu.org, eliz <at> gnu.org, eggert <at> cs.ucla.edu
Subject: Re: bug#13936: Default to UTF-8 for most Emacs source files
Date: Fri, 05 Apr 2013 14:58:54 -0400

>> I don't think Handa's work is related.  He's trying to improve the
>> performance of Emacs when handling utf-8 files, whereas my request is
>> for a change in behavior, regardless of performance.
> If the performance is not the problem,
> (with-coding-priority '(utf-8)
>   (load-file "_FILE_NAME_"))

That will work only for files using the utf-8 coding system.  My request
is for ways to make it so that more of the .el files are written using
the utf-8 coding system.  E.g. something like setting
buffer-coding-system in emacs-lisp-mode, for example.


        Stefan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.