GNU bug report logs - #30939
shepherd: detailed output should be placed into well-known location and not tty

Previous Next

To reply to this bug, email your comments to 30939 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: ng0 <ng0 <at> n0.is>
To: bug-guix <at> gnu.org
Subject: shepherd: detailed output should be placed into well-known location
 and not tty
Date: Sun, 25 Mar 2018 18:35:55 +0000

Problem, not just when a service is misbehaving after successful system reconfigure:

$ sudo herd start smtpd
Password: 
Service smtpd could not be started.
herd: failed to start service smtpd



This is on virtual terminal in X11, as well as in /var/log/messages,
/var/log/shepherd.log, etc.
This is not enough. If I wanted more info, I'd expect that
sudo herd status smtpd would give it (which it does not), so the only
reliable source of information so far is tty 1. Can we fix that in
one of the next shepherd releases? Or is this something we have to
fix in Guix?

Message #8 received at 30939 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: ng0 <ng0 <at> n0.is>
Cc: 30939 <at> debbugs.gnu.org
Subject: Re: bug#30939: shepherd: detailed output should be placed into
 well-known location and not tty
Date: Mon, 26 Mar 2018 15:41:27 +0200

Hi ng0,

ng0 <ng0 <at> n0.is> skribis:

> Problem, not just when a service is misbehaving after successful system reconfigure:
>
> $ sudo herd start smtpd
> Password: 
> Service smtpd could not be started.
> herd: failed to start service smtpd
>
>
>
> This is on virtual terminal in X11, as well as in /var/log/messages,
> /var/log/shepherd.log, etc.
> This is not enough. If I wanted more info, I'd expect that
> sudo herd status smtpd would give it (which it does not), so the only
> reliable source of information so far is tty 1. Can we fix that in
> one of the next shepherd releases? Or is this something we have to
> fix in Guix?

So you’re saying that you’d like shepherd to show more info as to why
the service could not be started, right?

Thanks,
Ludo’.

Message #11 received at 30939 <at> debbugs.gnu.org (full text, mbox):

From: ng0 <ng0 <at> n0.is>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30939 <at> debbugs.gnu.org, ng0 <ng0 <at> n0.is>
Subject: Re: bug#30939: shepherd: detailed output should be placed into
 well-known location and not tty
Date: Mon, 26 Mar 2018 15:10:36 +0000

Hi Ludovic,

Ludovic Courtès transcribed 790 bytes:
> Hi ng0,
> 
> ng0 <ng0 <at> n0.is> skribis:
> 
> > Problem, not just when a service is misbehaving after successful system reconfigure:
> >
> > $ sudo herd start smtpd
> > Password: 
> > Service smtpd could not be started.
> > herd: failed to start service smtpd
> >
> >
> >
> > This is on virtual terminal in X11, as well as in /var/log/messages,
> > /var/log/shepherd.log, etc.
> > This is not enough. If I wanted more info, I'd expect that
> > sudo herd status smtpd would give it (which it does not), so the only
> > reliable source of information so far is tty 1. Can we fix that in
> > one of the next shepherd releases? Or is this something we have to
> > fix in Guix?
> 
> So you’re saying that you’d like shepherd to show more info as to why
> the service could not be started, right?
> 
> Thanks,
> Ludo’.

Must have been late and too many failed attempts at what I'm trying to do.
Yes. So I can't make any daemons I run out there fail, but for the
current case I have in Guix for this:

Sometimes I succeed building a system generation with an OpenSMTPD config-file
which has syntax error that aren't picked up at configure time. When I reboot,
not being aware of this, I have to switch to tty to read the reasons why it
crashed.
Because this is a desktop system, I have to start the service again to see
the error output directly from the daemon.

I think I know why this happens (that the output goes to tty), but nevertheless
it would be good if shepherd were more capable than beint captain obvious:
Start: "Oh, you see it is started". Crashed: "Oh, no has your daemon crashed?",
like it is now.
....
Okay, I just looked at some other daemon controls I run, and maybe it's good that
shepherd is limited in its output. It does this one job. What I'd like to have
as a sysadmin is the ability to tail something like say /var/log/shepherd.fail.log
and services which are failing log into this file (or a set of files in /var/log/shepherd/
in files like $daemonname.fail.log).
Given the absence of the kitchensink of tools in systemd, you got used to something like
"status" and immediate "HELLO! This is why I failed: (5 lines)". With shepherd, you can't
even grep for the failures in locations newcomers to the system would assume (like:
/var/log/shepherd.log (it is the daemon control application)).

Long store short, greping for failures to fix daemon configurations and not having to
look at tty 1 (which can be noisy depending on what you run, I have some notorious tty
spammers) would be good.
And not sacrifice the simplicity of Shepherd :)

Message #14 received at 30939 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: ng0 <ng0 <at> n0.is>
Cc: 30939 <at> debbugs.gnu.org
Subject: Re: bug#30939: shepherd: detailed output should be placed into
 well-known location and not tty
Date: Tue, 27 Mar 2018 09:36:20 +0200

Hello,

ng0 <ng0 <at> n0.is> skribis:

> Sometimes I succeed building a system generation with an OpenSMTPD config-file
> which has syntax error that aren't picked up at configure time. When I reboot,
> not being aware of this, I have to switch to tty to read the reasons why it
> crashed.
> Because this is a desktop system, I have to start the service again to see
> the error output directly from the daemon.

I think shepherd could capture stdout/stderr of the processes it starts
and make it available, in a way similar in spirit to what ‘journalctl’
does.  That would allow you to see the output of the daemon that failed.

That’s the only solution I can think of.  Of course we don’t have to do
that if the daemon writes error messages to syslog, but not all of them do.

Thanks,
Ludo’.

Message #17 received at 30939 <at> debbugs.gnu.org (full text, mbox):

From: ng0 <ng0 <at> n0.is>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30939 <at> debbugs.gnu.org, ng0 <ng0 <at> n0.is>
Subject: Re: bug#30939: shepherd: detailed output should be placed into
 well-known location and not tty
Date: Tue, 27 Mar 2018 09:15:29 +0000

Ludovic Courtès transcribed 839 bytes:
> Hello,
> 
> ng0 <ng0 <at> n0.is> skribis:
> 
> > Sometimes I succeed building a system generation with an OpenSMTPD config-file
> > which has syntax error that aren't picked up at configure time. When I reboot,
> > not being aware of this, I have to switch to tty to read the reasons why it
> > crashed.
> > Because this is a desktop system, I have to start the service again to see
> > the error output directly from the daemon.
> 
> I think shepherd could capture stdout/stderr of the processes it starts
> and make it available, in a way similar in spirit to what ‘journalctl’
> does.  That would allow you to see the output of the daemon that failed.

That sounds good.

> That’s the only solution I can think of.  Of course we don’t have to do
> that if the daemon writes error messages to syslog, but not all of them do.
> 
> Thanks,
> Ludo’.

Well, just a way to catch them would be good.

Thanks

Message #20 received at 30939 <at> debbugs.gnu.org (full text, mbox):

From: Clément Lassieur <clement <at> lassieur.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30939 <at> debbugs.gnu.org, ng0 <ng0 <at> n0.is>
Subject: Re: bug#30939: shepherd: detailed output should be placed into
 well-known location and not tty
Date: Tue, 27 Mar 2018 20:59:46 +0200

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hello,
>
> ng0 <ng0 <at> n0.is> skribis:
>
>> Sometimes I succeed building a system generation with an OpenSMTPD config-file
>> which has syntax error that aren't picked up at configure time. When I reboot,
>> not being aware of this, I have to switch to tty to read the reasons why it
>> crashed.
>> Because this is a desktop system, I have to start the service again to see
>> the error output directly from the daemon.
>
> I think shepherd could capture stdout/stderr of the processes it starts
> and make it available, in a way similar in spirit to what ‘journalctl’
> does.  That would allow you to see the output of the daemon that failed.

That would be great!

> That’s the only solution I can think of.  Of course we don’t have to do
> that if the daemon writes error messages to syslog, but not all of them do.
>
> Thanks,
> Ludo’.

Message #23 received at 30939 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30939 <at> debbugs.gnu.org, ng0 <ng0 <at> n0.is>
Subject: Re: bug#30939: shepherd: detailed output should be placed into
 well-known location and not tty
Date: Tue, 27 Mar 2018 16:13:52 -0400

ludo <at> gnu.org (Ludovic Courtès) writes:

> ng0 <ng0 <at> n0.is> skribis:
>
>> Sometimes I succeed building a system generation with an OpenSMTPD config-file
>> which has syntax error that aren't picked up at configure time. When I reboot,
>> not being aware of this, I have to switch to tty to read the reasons why it
>> crashed.
>> Because this is a desktop system, I have to start the service again to see
>> the error output directly from the daemon.
>
> I think shepherd could capture stdout/stderr of the processes it starts
> and make it available, in a way similar in spirit to what ‘journalctl’
> does.  That would allow you to see the output of the daemon that failed.

This would be very helpful.

      Mark

Message #28 received at 30939 <at> debbugs.gnu.org (full text, mbox):

From: Robert Vollmert <rob <at> vllmrt.net>
To: 30939 <at> debbugs.gnu.org
Subject: still relevant
Date: Wed, 26 Jun 2019 20:07:06 +0200

This came up again recently, compare the discussion here:

https://lists.gnu.org/archive/html/guix-devel/2019-06/msg00186.html

Here’s some code to wrap an executable manually to capture its output
and send it to syslog:

(define* (logger-wrapper name exec . args)
  "Return a derivation that builds a script to start a process with
standard output and error redirected to syslog via logger."
  (define exp
    #~(begin
        (use-modules (ice-9 popen))
        (let* ((pid    (number->string (getpid)))
               (logger #$(file-append inetutils "/bin/logger"))
               (args   (list "-t" #$name (string-append "--id=" pid)))
               (pipe   (apply open-pipe* OPEN_WRITE logger args)))
          (dup pipe 1)
          (dup pipe 2)
          (execl #$exec #$exec #$@args))))
  (program-file (string-append name "-logger") exp))

Message #33 received at 30939 <at> debbugs.gnu.org (full text, mbox):

From: conjaroy <conjaroy <at> gmail.com>
To: 30939 <at> debbugs.gnu.org
Subject: shepherd: detailed output should be placed into well-known location
 and not tty
Date: Sat, 18 Jul 2020 12:27:06 -0400

[Message part 1 (text/plain, inline)]

Hello -

I too have found that debugging is a challenge when a service's
stdout/stderr aren't captured automatically. From my point of view though,
the issue is not just that certain binaries lack syslog support: since a
service implementation's gexp can do much more than just exec a binary, and
since mistakes in gexps usually go unnoticed until a runtime, I've found it
easy to write scripts that trigger fatal Guile errors before the service
binary is even started (syntax errors, missing `use-modules` declarations,
etc.)

Will the solution proposed here capture output for this class of errors as
well?

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.