GNU bug report logs - #24542
25.1.50; The symbol `@' and sexp scanning

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 24542 in the body.
You can then email your comments to 24542 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: bug-gnu-emacs <at> gnu.org
Subject: 25.1.50; The symbol `@' and sexp scanning
Date: Sun, 25 Sep 2016 19:42:17 +0200

Hi,

the following breaks my "el-search" package when searching in
"skeleton.el".  It assumes that calling `scan-sexps' works at the start
of any sexp.

Recipe: in emacs -Q,

insert

    (eq element '@)

in an elisp mode buffer (e.g. scratch).  Put point at the quote or the
"@".  Eval

    (goto-char (scan-sexps (point) 1))

and you get an error like:


Debugger entered--Lisp error: (scan-error "Containing expression ends prematurely" 15 16)
  scan-sexps(13 1)
  (goto-char (scan-sexps (point) 1))
  eval((goto-char (scan-sexps (point) 1)) nil)
  eval-expression((goto-char (scan-sexps (point) 1)) nil)
  funcall-interactively(eval-expression (goto-char (scan-sexps (point) 1)) nil)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)


I think `scan-sexps' should not error.  FWIW, if I add one more sexp to
the list after "'@" (different from "'@"), there is no error.


TIA,

Michael.




In GNU Emacs 25.1.50.2 (x86_64-pc-linux-gnu, GTK+ Version 3.21.5)
 of 2016-09-23 built on drachen
Repository revision: 14c36d76df035faa127580d706a0564f4e496991
Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
System Description:	Debian GNU/Linux testing (stretch)

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND DBUS GSETTINGS NOTIFY
LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LC_ALL: de_DE.utf8
  value of $LC_COLLATE: C
  value of $LC_TIME: C
  value of $LANG: de_DE.utf8
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Message #8 received at 24542 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: 24542 <at> debbugs.gnu.org
Subject: Re: bug#24542: 25.1.50; The symbol `@' and sexp scanning
Date: Mon, 02 Jan 2017 17:58:48 +0100

Michael Heerdegen <michael_heerdegen <at> web.de> writes:

> insert
>
>     (eq element '@)
>
> in an elisp mode buffer (e.g. scratch).  Put point at the quote or the
> "@".  Eval
>
>     (goto-char (scan-sexps (point) 1))
>
> and you get an error like:
>
>
> Debugger entered--Lisp error: (scan-error "Containing expression ends
> prematurely" 15 16)

It seems that you can replace the symbol `@' in this recipe with any
symbol whose name has the form "@+", i.e. `@@' or `@@@' or...  When the
symbol contains only one character different from "@", it doesn't seem
to happen.


Michael.

Message #11 received at 24542 <at> debbugs.gnu.org (full text, mbox):

From: npostavs <at> users.sourceforge.net
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 24542 <at> debbugs.gnu.org
Subject: Re: bug#24542: 25.1.50; The symbol `@' and sexp scanning
Date: Mon, 02 Jan 2017 15:37:28 -0500

Michael Heerdegen <michael_heerdegen <at> web.de> writes:

> Michael Heerdegen <michael_heerdegen <at> web.de> writes:
>
>> insert
>>
>>     (eq element '@)
>>
>> in an elisp mode buffer (e.g. scratch).  Put point at the quote or the
>> "@".  Eval
>>
>>     (goto-char (scan-sexps (point) 1))
>>
>> and you get an error like:
>>
>>
>> Debugger entered--Lisp error: (scan-error "Containing expression ends
>> prematurely" 15 16)
>
> It seems that you can replace the symbol `@' in this recipe with any
> symbol whose name has the form "@+", i.e. `@@' or `@@@' or...  When the
> symbol contains only one character different from "@", it doesn't seem
> to happen.

`@' has the prefix syntax flag, the elisp manual says `(elisp) Syntax Flags':

   * `p' identifies an additional prefix character for Lisp syntax.
     These characters are treated as whitespace when they appear between
                          ^^^^^^^^^^^^^^^^^^^^^
     expressions.  When they appear within an expression, they are
     handled according to their usual syntax classes.

So (eq element '@) acts the same as (eq element ' ).  This could be
changed with something the following patch, but I'm not sure if it's the
right thing for non-Lisp languages...

diff --git i/src/syntax.c w/src/syntax.c
index 0ee1c74..253d3fb 100644
--- i/src/syntax.c
+++ w/src/syntax.c
@@ -2681,7 +2681,16 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
 	    }
 
 	  if (prefix)
-	    continue;
+            {
+              int next_c = FETCH_CHAR_AS_MULTIBYTE (from_byte);
+              int next_syntax = SYNTAX_WITH_FLAGS (next_c);
+              bool next_prefix = SYNTAX_FLAGS_PREFIX (next_syntax);
+              enum syntaxcode next_code = syntax_multibyte (next_c, multibyte_symbol_p);
+              if (next_prefix
+                  || next_code == Ssymbol
+                  || next_code == Sword)
+                continue;
+            }
 
 	  switch (code)
 	    {

Message #14 received at 24542 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: npostavs <at> users.sourceforge.net
Cc: 24542 <at> debbugs.gnu.org
Subject: Re: bug#24542: 25.1.50; The symbol `@' and sexp scanning
Date: Tue, 03 Jan 2017 13:46:26 +0100

npostavs <at> users.sourceforge.net writes:

> >> insert
> >>
> >>     (eq element '@)
> >>
> >> in an elisp mode buffer (e.g. scratch).  Put point at the quote or the
> >> "@".  Eval
> >>
> >>     (goto-char (scan-sexps (point) 1))
> >>
> >> and you get an error like:
> >>
> >>
> >> Debugger entered--Lisp error: (scan-error "Containing expression ends
> >> prematurely" 15 16)
> >
> > It seems that you can replace the symbol `@' in this recipe with any
> > symbol whose name has the form "@+", i.e. `@@' or `@@@' or...  When the
> > symbol contains only one character different from "@", it doesn't seem
> > to happen.
>
> `@' has the prefix syntax flag, the elisp manual says `(elisp) Syntax Flags':
>
>    * `p' identifies an additional prefix character for Lisp syntax.
>      These characters are treated as whitespace when they appear between
>                           ^^^^^^^^^^^^^^^^^^^^^
>      expressions.  When they appear within an expression, they are
>      handled according to their usual syntax classes.
>
> So (eq element '@) acts the same as (eq element ' ).

I don't agree with your interpretation.  `@' is an (textual
representation of an) expression in the above example, so "these
characters" don't "appear between expressions" in this case.  Could be
that this wording led to a wrong implementation, however.

> This could be changed with something the following patch, but I'm not
> sure if it's the right thing for non-Lisp languages...

> diff --git i/src/syntax.c w/src/syntax.c
> index 0ee1c74..253d3fb 100644
> --- i/src/syntax.c
> +++ w/src/syntax.c
> @@ -2681,7 +2681,16 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
>  	    }
>  
>  	  if (prefix)
> -	    continue;
> +            {
> +              int next_c = FETCH_CHAR_AS_MULTIBYTE (from_byte);
> +              int next_syntax = SYNTAX_WITH_FLAGS (next_c);
> +              bool next_prefix = SYNTAX_FLAGS_PREFIX (next_syntax);
> +              enum syntaxcode next_code = syntax_multibyte (next_c, multibyte_symbol_p);
> +              if (next_prefix
> +                  || next_code == Ssymbol
> +                  || next_code == Sword)
> +                continue;
> +            }
>  
>  	  switch (code)
>  	    {

Thanks for the patch.  Can anyone judge whether we can do this?


Thanks,

Michael. 

Message #17 received at 24542 <at> debbugs.gnu.org (full text, mbox):

From: npostavs <at> users.sourceforge.net
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 24542 <at> debbugs.gnu.org
Subject: Re: bug#24542: 25.1.50; The symbol `@' and sexp scanning
Date: Tue, 03 Jan 2017 08:03:54 -0500

Michael Heerdegen <michael_heerdegen <at> web.de> writes:
>> `@' has the prefix syntax flag, the elisp manual says `(elisp) Syntax Flags':
>>
>>    * `p' identifies an additional prefix character for Lisp syntax.
>>      These characters are treated as whitespace when they appear between
>>                           ^^^^^^^^^^^^^^^^^^^^^
>>      expressions.  When they appear within an expression, they are
>>      handled according to their usual syntax classes.
>>
>> So (eq element '@) acts the same as (eq element ' ).
>
> I don't agree with your interpretation.  `@' is an (textual
> representation of an) expression in the above example, so "these
> characters" don't "appear between expressions" in this case.  Could be
> that this wording led to a wrong implementation, however.

Well, it's a bit ambiguous whether a character sequence composed soley
of prefix characters should be considered an expression or not.
Obviously the end result is wrong for Lisp, i.e., does not correspond
with what `read' thinks.

Message #20 received at 24542 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 24542 <at> debbugs.gnu.org, npostavs <at> users.sourceforge.net
Subject: Re: bug#24542: 25.1.50; The symbol `@' and sexp scanning
Date: Fri, 04 Sep 2020 15:05:36 +0200

Michael Heerdegen <michael_heerdegen <at> web.de> writes:

> Thanks for the patch.  Can anyone judge whether we can do this?

I tried the patch, and it fixes the

(setq a '@)

problem for me.  I don't really have a great overview of where
scan_lists is used for -- it seems like a very general solution to a
very specific problem.

Does anybody have any comments here?

diff --git a/src/syntax.c b/src/syntax.c
index 7f0fc341f6..10912dd5f2 100644
--- a/src/syntax.c
+++ b/src/syntax.c
@@ -2693,7 +2693,17 @@ scan_lists (EMACS_INT from0, EMACS_INT count, EMACS_INT depth, bool sexpflag)
 	    }
 
 	  if (prefix)
-	    continue;
+            {
+              int next_c = FETCH_CHAR_AS_MULTIBYTE (from_byte);
+              int next_syntax = SYNTAX_WITH_FLAGS (next_c);
+              bool next_prefix = SYNTAX_FLAGS_PREFIX (next_syntax);
+              enum syntaxcode next_code =
+		syntax_multibyte (next_c, multibyte_symbol_p);
+              if (next_prefix
+                  || next_code == Ssymbol
+                  || next_code == Sword)
+                continue;
+            }
 
 	  switch (code)
 	    {


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #23 received at 24542 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: michael_heerdegen <at> web.de, 24542 <at> debbugs.gnu.org,
 npostavs <at> users.sourceforge.net
Subject: Re: bug#24542: 25.1.50; The symbol `@' and sexp scanning
Date: Fri, 04 Sep 2020 16:31:52 +0300

> From: Lars Ingebrigtsen <larsi <at> gnus.org>
> Date: Fri, 04 Sep 2020 15:05:36 +0200
> Cc: 24542 <at> debbugs.gnu.org, npostavs <at> users.sourceforge.net
> 
> I tried the patch, and it fixes the
> 
> (setq a '@)
> 
> problem for me.  I don't really have a great overview of where
> scan_lists is used for -- it seems like a very general solution to a
> very specific problem.
> 
> Does anybody have any comments here?

Stefan, any comments?

Message #26 received at 24542 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: michael_heerdegen <at> web.de, Lars Ingebrigtsen <larsi <at> gnus.org>,
 24542 <at> debbugs.gnu.org, npostavs <at> users.sourceforge.net
Subject: Re: bug#24542: 25.1.50; The symbol `@' and sexp scanning
Date: Fri, 04 Sep 2020 10:22:41 -0400

>> Does anybody have any comments here?
> Stefan, any comments?

Checking for an immediately following word or symbol syntax doesn't see
sufficient if we consider cases like:

   '(sgasfg)

or

   , (sfgsdf)

or

   ,@ ;; entertained yet?
   \)

and then we have the really fun stuff like:

   (foo '@ bar)

vs

   (foo ,@ bar)

So maybe a better solution is a syntax-propertize rule which
distinguishes "@ after comma" from "@ not after comma"?


        Stefan

Message #31 received at 24542 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 24542 <at> debbugs.gnu.org
Subject: Re: bug#24542: 25.1.50; The symbol `@' and sexp scanning
Date: Wed, 10 Apr 2024 19:06:32 -0400

[Message part 1 (text/plain, inline)]

>     (eq element '@)
>
> in an elisp mode buffer (e.g. scratch).  Put point at the quote or the
> "@".  Eval
>     (goto-char (scan-sexps (point) 1))
> and you get an error like:
> Debugger entered--Lisp error: (scan-error "Containing expression ends prematurely" 15 16)

The patch below seems to fix it.
Any comments/objections?


        Stefan

[bug24542.diff (text/x-diff, inline)]

diff --git a/lisp/progmodes/elisp-mode.el b/lisp/progmodes/elisp-mode.el
index 9b4c3f994cd..10d7f01ebaf 100644
--- a/lisp/progmodes/elisp-mode.el
+++ b/lisp/progmodes/elisp-mode.el
@@ -40,9 +40,8 @@ 'emacs-lisp-mode-abbrev-table
 
 (defvar emacs-lisp-mode-syntax-table
   (let ((table (make-syntax-table lisp-data-mode-syntax-table)))
-    ;; These are redundant, now.
-    ;;(modify-syntax-entry ?\[ "(]  " table)
-    ;;(modify-syntax-entry ?\] ")[  " table)
+    ;; `syntax-propertize'. takes care of `,@'.
+    (modify-syntax-entry ?@ "_" table)
     table)
   "Syntax table used in `emacs-lisp-mode'.")
 
diff --git a/test/lisp/progmodes/elisp-mode-tests.el b/test/lisp/progmodes/elisp-mode-tests.el
index 1d1ef9981e5..3ef5a28e460 100644
--- a/test/lisp/progmodes/elisp-mode-tests.el
+++ b/test/lisp/progmodes/elisp-mode-tests.el
@@ -1131,5 +1131,14 @@ test-indentation
                         (emacs-lisp-mode)
                         (indent-region (point-min) (point-max)))))
 
+(ert-deftest elisp-tests-syntax-propertize ()
+  (with-temp-buffer
+    (emacs-lisp-mode)
+    (insert "(a '@)")
+    (should (equal (scan-sexps (+ (point-min) 3) 1) (1- (point-max))))
+    (erase-buffer)
+    (insert "(a ,@)")
+    (should-error (scan-sexps (+ (point-min) 3) 1))))
+
 (provide 'elisp-mode-tests)
 ;;; elisp-mode-tests.el ends here

Message #34 received at 24542 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 24542 <at> debbugs.gnu.org
Subject: Re: bug#24542: 25.1.50; The symbol `@' and sexp scanning
Date: Thu, 11 Apr 2024 05:42:08 +0200

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:

>  (defvar emacs-lisp-mode-syntax-table
>    (let ((table (make-syntax-table lisp-data-mode-syntax-table)))
> -    ;; These are redundant, now.
> -    ;;(modify-syntax-entry ?\[ "(]  " table)
> -    ;;(modify-syntax-entry ?\] ")[  " table)
> +    ;; `syntax-propertize'. takes care of `,@'.
> +    (modify-syntax-entry ?@ "_" table)
>      table)
>    "Syntax table used in `emacs-lisp-mode'.")

This seems to give good results for me - so far at least.  Also Isearch
symbol searching doesn't seem to be affected in any negative way (I
think there is lost "." in your comment, however).

Thanks,

Michael.

Message #39 received at 24542-done <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 24542-done <at> debbugs.gnu.org
Subject: Re: bug#24542: 25.1.50; The symbol `@' and sexp scanning
Date: Fri, 12 Apr 2024 13:29:16 -0400

>>  (defvar emacs-lisp-mode-syntax-table
>>    (let ((table (make-syntax-table lisp-data-mode-syntax-table)))
>> -    ;; These are redundant, now.
>> -    ;;(modify-syntax-entry ?\[ "(]  " table)
>> -    ;;(modify-syntax-entry ?\] ")[  " table)
>> +    ;; `syntax-propertize'. takes care of `,@'.
>> +    (modify-syntax-entry ?@ "_" table)
>>      table)
>>    "Syntax table used in `emacs-lisp-mode'.")
>
> This seems to give good results for me - so far at least.

Thanks, pushed to `master`, closing.


        Stefan

Message #47 received at 24542 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: 24542 <at> debbugs.gnu.org
Cc: monnier <at> iro.umontreal.ca
Subject: Re: bug#24542: 25.1.50; The symbol `@' and sexp scanning
Date: Fri, 12 Apr 2024 23:32:42 +0200

Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs <at> gnu.org> writes:

> Thanks, pushed to `master`, closing.

Thank you.

[ Is the omission of the test your original patch contained
intentional? ]


Michael.

Message #50 received at 24542 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 24542 <at> debbugs.gnu.org
Subject: Re: bug#24542: 25.1.50; The symbol `@' and sexp scanning
Date: Fri, 12 Apr 2024 21:55:00 -0400

> [ Is the omission of the test your original patch contained
> intentional? ]

No, indeed, thanks,


        Stefan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.