[Novalug] Filename handling: correctness vs. convenience

[Michael Henry]

James Ewing Cottrell 3rd wrote:
>  Well, your text seemed very thorough. But it's too long to actually read.

I agree that it's awfully long - sorry about that.  I was aiming for
enough detail in the explanation and motivation that most LUG users
would be able to follow along regardless of their experience level. 
Thanks for taking the trouble to read it.  One day I'll try to find a
better balance between detail and brevity.

>  I would opt for Convenience, along with Hunting Down and Killing 
anyone who created these files names. :)

:-)

>  But seriously, if you keep your filenames in variables and always 
double-quote the references, you will mostly be OK.

My main point, which I probably buried in too much verbiage, is that
there is a difference between "mostly OK" and "fully OK", and that the
line between the two is rarely pointed out in the Unix culture when
talking about filename handling in shell scripting.  I believe there are
plenty of cases where "mostly OK" is good enough, but only when the
script author recognizes that the solution is only "mostly OK" and knows
when he must use the "fully OK" solution.

>  Users tend to put spaces in their filenames, but rarely, newlines.

I agree for "regular" users.  Attackers, on the other hand,
intentionally create such files to exploit errors in buggy scripts.  It
is my purpose to raise awareness among the members of our community that
some of the commonly used shortcuts do not have full generality, and can
be the cause of security vulnerabilities when used in general-purpose
scripts.  In my view, the rareness of finding newlines in typical
filenames serves only to make it less likely a developer will find his
bug through testing, and perhaps less likely he'll even know it's a bug.

>  One mistake you did make was when an expansion exceeds the command 
line length. Typing "ls *.bak | while read ..." doesn't eliminate the 
error, it just moved it to a different part of the pipeline.

My mistake - thanks for catching that.  I'd intended to use the ``find``
command for that example, but then changed my mind and talked about
``find`` later.  I should have spent more time proofreading (another
downside of writing an excessively long email :-)).

Thanks for your comments,
Michael Henry