Greetings,
I implemented a simple common option (-K) for the client tools that if enabled, sets the file descriptor used to output errors to stdout instead of the default (and currently used) stderr.
This is done because when running a batch of commands with the output sent to a file, logs collected are not shown chronologically because the output is split between stdout and stderr. This occurs because both stdout and stderr do not have the same buffer behavior and unfortunately, there is no means within any shell to get them to behave in concert.
Here is an example of the output of ldapmodify saved in a file using the current code:
ldapmodify: Type or value exists (20) ldapmodify: Type or value exists (20) ldapmodify: Type or value exists (20) modifying entry "uid=User01,ou=MyOrganization,o=example.com "
modifying entry "uid=User03,ou=MyOrganization,o=example.com "
modifying entry "uid=User04,ou=MyOrganization,o=example.com "
modifying entry "uid=User03,ou=MyOrganization,o=example.com "
modifying entry "uid=User02,ou=MyOrganization,o=example.com "
modifying entry "uid=User05,ou=MyOrganization,o=example.com "
modifying entry "uid=User21,ou=MyOrganization,o=example.com "
modifying entry "uid=User22,ou=MyOrganization,o=example.com "
Now when instructing ldapmodify to redirect errors to stdout, the following output shows up.
modifying entry "uid=User01,ou=MyOrganization,o=example.com "
modifying entry "uid=User03,ou=MyOrganization,o=example.com " ldapmodify: Type or value exists (20)
modifying entry "uid=User04,ou=MyOrganization,o=example.com " ldapmodify: Type or value exists (20)
modifying entry "uid=User03,ou=MyOrganization,o=example.com "
modifying entry "uid=User02,ou=MyOrganization,o=example.com " ldapmodify: Type or value exists (20)
modifying entry "uid=User05,ou=MyOrganization,o=example.com "
modifying entry "uid=User21,ou=MyOrganization,o=example.com "
modifying entry "uid=User22,ou=MyOrganization,o=example.com "
As you can see, in the first case it is impossible to tell which of the multiple operations failed. With my little feature, now it is possible. Many persons on the web complain about this and I believe that my feature will be well received. Default for errors still remain stderr, so normal usage will remain undisturbed.
The change to the code is very simple. I change common.c and common.h, added the -K options and replaced stderr with errorfp (default to stderr) in many of the tools' source code. Very low risk of breaking anything. I have been using ldapsearch and ldapmodify a lot with my option. Why -K? Because it is one of the rare letters still available in the long list of options.
I would like to submit my changes for review and hopefully, if accepted, that it becomes part of the trunk. However, I do not know how to proceed. Though I am familiar with other configuration management tools, I am not familiar with git, thus if any operation with this tools is required, please provide me examples of commands I should do.
Best regards, Hans Deragon http://www.deragon.biz
Hans Deragon wrote:
I implemented a simple common option (-K) for the client tools that if enabled, sets the file descriptor used to output errors to stdout instead of the default (and currently used) stderr.
This is done because when running a batch of commands with the output sent to a file, logs collected are not shown chronologically because the output is split between stdout and stderr. (...)
I suggest instead a -B option to make stdout unbuffered or linebuffered.
Hallvard
Hallvard Breien Furuseth wrote:
Hans Deragon wrote:
I implemented a simple common option (-K) for the client tools that if enabled, sets the file descriptor used to output errors to stdout instead of the default (and currently used) stderr.
That would be stupid. Error messages belong on stderr, that's why it exists.
This is done because when running a batch of commands with the output sent to a file, logs collected are not shown chronologically because the output is split between stdout and stderr. (...)
There are two obvious fixes: 1) don't send batches of commands that ignore errors, so that the processing immediately stops at the failed entry. 2) change the error message text to include the input line number where it failed.
Also a possibility: don't send running status messages to stdout, send them only to stderr. Technically the only commandline tool that normally produces "output" is ldapsearch. Everything else is "diagnostic" and belongs on stderr anyway.
I suggest instead a -B option to make stdout unbuffered or linebuffered.
stdout is already linebuffered by default when directed to a tty.
Greetings M. Chu,
The most practical solution is to send running status messages to one stream, as you suggest, though I would send it to stdout, not stderr. In my view, stderr is for errors and diagnostics in the sense of debugging, not for normal operation. A tool that reports the status of its operations should send its output to stdout. Connection errors, timeouts and such abnormal errors should be reported to stderr. Duplicate values, existing values, entry not found and such are normal operational messages and should be sent to stdout.
I made my changes because they were the simplest, fasted solutions which do not modify the current tools default behavior. I also believe that in the context of logging, using one single stream to record all events in chronological order is very practical. Since we disagree on this matter and you are chef architect of OpenLDAP, you decision stands and I respect that. We will leave the tools as is.
Best regards, Hans Deragon
Hallvard Breien Furuseth wrote:
Hans Deragon wrote:
I implemented a simple common option (-K) for the client tools that if enabled, sets the file descriptor used to output errors to stdout instead of the default (and currently used) stderr.
That would be stupid. Error messages belong on stderr, that's why it exists.
This is done because when running a batch of commands with the output sent to a file, logs collected are not shown chronologically because the output is split between stdout and stderr. (...)
There are two obvious fixes:
- don't send batches of commands that ignore errors, so that the
processing immediately stops at the failed entry. 2) change the error message text to include the input line number where it failed.
Also a possibility: don't send running status messages to stdout, send them only to stderr. Technically the only commandline tool that normally produces "output" is ldapsearch. Everything else is "diagnostic" and belongs on stderr anyway.
I suggest instead a -B option to make stdout unbuffered or linebuffered.
stdout is already linebuffered by default when directed to a tty.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Hans Deragon wrote:
Greetings M. Chu,
The most practical solution is to send running status messages to one stream, as you suggest, though I would send it to stdout, not stderr. In my view, stderr is for errors and diagnostics in the sense of debugging, not for normal operation. A tool that reports the status of its operations should send its output to stdout. Connection errors, timeouts and such abnormal errors should be reported to stderr. Duplicate values, existing values, entry not found and such are normal operational messages and should be sent to stdout.
I made my changes because they were the simplest, fasted solutions which do not modify the current tools default behavior. I also believe that in the context of logging, using one single stream to record all events in chronological order is very practical. Since we disagree on this matter and you are chef architect of OpenLDAP, you decision stands and I respect that. We will leave the tools as is.
You still have several other alternatives to investigate. As Hallvard suggested, simply calling setlinebuf(stdout) in the tools may be sufficient.
Best regards, Hans Deragon
Hallvard Breien Furuseth wrote:
Hans Deragon wrote:
I implemented a simple common option (-K) for the client tools that if enabled, sets the file descriptor used to output errors to stdout instead of the default (and currently used) stderr.
That would be stupid. Error messages belong on stderr, that's why it exists.
This is done because when running a batch of commands with the output sent to a file, logs collected are not shown chronologically because the output is split between stdout and stderr. (...)
There are two obvious fixes: 1) don't send batches of commands that ignore errors, so that the processing immediately stops at the failed entry. 2) change the error message text to include the input line number where it failed.
Also a possibility: don't send running status messages to stdout, send them only to stderr. Technically the only commandline tool that normally produces "output" is ldapsearch. Everything else is "diagnostic" and belongs on stderr anyway.
I suggest instead a -B option to make stdout unbuffered or linebuffered.
stdout is already linebuffered by default when directed to a tty.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
-
Hallvard Breien Furuseth -
Hans Deragon -
Howard Chu