Subject: Two FTP issues

From: John C Klensin <[email-masked]>
Date: Mon, 30 Nov 2020 17:07:41 -0500

Hi.
Inspired (if that is the right term) by the recent discussions
about FTP, I went back and reread the spec (RFC 959) today and
made two rediscoveries of problems/issues.  Before I explain
them (and try to keep this short), I will pose the key question:
does anyone here have responsibility (or the inclination to fuss
with) FTP client and/or server software and, if so, would you be
inclined to make changes to fix bugs or near-bugs?

Anyone offended by the very idea of updates or clarifications to
FTP should please skip to the last paragraph.

Issues:

TYPE ASCII assumes that whatever text files exist on the sending
system can be converted to NVT ASCII.  Back in 1985 and earlier,
that could require some effort and maybe some small compromises,
but it wasn't conceptually difficult.  Today, with a great deal
of Unicode on the network, there are no possible conversions, at
least unless "NVT ASCII" is redefined to include assorted escape
sequences (clearly not the intent and, from a user standpoint,
almost certainly bad news, but see below).  So:

(1) Do we need a specific error code for "you specified 'TYPE
ASCII' but the file you want to transfer cannot be converted".
In principle, the same code would apply to TYPE EBCDIC, but I
haven't seen much of that the wild lately.  If the answer is
"no", what would we expect implementations to do under those
circumstances?

(2) In the interest of an increasingly internationalized
Internet, is it time to look again at a Unicode type, perhaps
using the long-expired draft-klensin-ftpext-typeu as a starting
point?  I haven't looked at it in seven or eight years, so don't
have an opinion as to whether it would be useful.

(3) I hope the answer to the question of Unicode in NVT ASCII is
not "escape sequence" but, if it is, do we need a specification,
however short, about which of the many possible escape sequences
is to be used and/or identified?

Request: I think the topic of whether anyone who thinks about
using FTP, much less updating it, needs to be dragged kicking
and screaming into the 21st century and/or suffers from some
mental, emotional, or moral problem has been covered thoroughly
in the last several weeks.  Please do not try to repeat it,
especially if the purpose of doing so is to prevent a discussion
among those who are interested.

thanks,
   john

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 30 Nov 2020 23:38:43 +0100

Hi John,

I believe a hard-earned piece of experience I took from using FTP is that you don’t want to do conversions during transit.

In theory, the sending system knows its format, can convert that to a common format, and the receiving system knows its own format and can do the second conversion.  That was certainly true when we had access methods, per-record byte counts etc.

In practice, systems today have files of various formats lying around, no useful metadata so no idea what the actual source format is, so they are likely to botch the conversion.  Getting the original bytes from the sending system and doing the conversion at the receiving end, outside the actual transfer, became the norm.  I don’t remember when ftp clients started to automatically send “TYPE I” on connection setup, but it sure made life so much easier (read: FTP became somewhat usable again).

This is probably one instance of the more general gateway fundamental.  Instead of needing gateways between various mail systems, everything converged to SMTP, or being as close to SMTP as possible (i.e. requiring minimal gatewaying/conversion).

This has also happened in text formats (with the exception of the remaining two line ends, and, with a lesser relevance, interpretation of HTs).  Anything that tries to make life easier for systems that aren’t UTF-8 yet is like adding another transition technology to IPv6: trying to be helpful, but in practice counterproductive.

Grüße, Carsten

---

From: John C Klensin <[email-masked]>
Date: Mon, 30 Nov 2020 18:26:00 -0500

--On Monday, November 30, 2020 23:38 +0100 Carsten Bormann
<[email-masked]> wrote:

> Hi John,
> 
> I believe a hard-earned piece of experience I took from using
> FTP is that you don't want to do conversions during transit.

I think we had figured that out by early in 1971. At least in
general, I think it is still true.

> In theory, the sending system knows its format, can convert
> that to a common format, and the receiving system knows its
> own format and can do the second conversion.  That was
> certainly true when we had access methods, per-record byte
> counts etc.
> 
> In practice, systems today have files of various formats lying
> around, no useful metadata so no idea what the actual source
> format is, so they are likely to botch the conversion.
> Getting the original bytes from the sending system and doing
> the conversion at the receiving end, outside the actual
> transfer, became the norm.  I don't remember when ftp
> clients started to automatically send "TYPE I" on
> connection setup, but it sure made life so much easier (read:
> FTP became somewhat usable again).

Even then, if someone asks for "type ascii" or, potentially,
"type unicode", an error message, maybe with a 3yz or 4yz code,
to the effect of "I don't have a clue what this file actually
is, suggest you ask for Type I and sort it out yourself".

> This is probably one instance of the more general gateway
> fundamental.  Instead of needing gateways between various mail
> systems, everything converged to SMTP, or being as close to
> SMTP as possible (i.e. requiring minimal
> gatewaying/conversion).
> 
> This has also happened in text formats (with the exception of
> the remaining two line ends, and, with a lesser relevance,
> interpretation of HTs).  Anything that tries to make life
> easier for systems that aren't UTF-8 yet is like adding
> another transition technology to IPv6: trying to be helpful,
> but in practice counterproductive.

There, and for the specific case of Unicode, probably we
disagree.  Keep in mind that a number of contemporary operating
systems use UTF-16, or even UTF-32, with some byte ordering,
internally.  They typically know they are doing that, if only to
be able to do an orderly conversation to UTF-8 for putting over
the wire or from UTF-8 or ASCII for incoming data.  Now, if I
retrieve a UTF-16 object in image moded, there better be a BOM
and it better be accurate or I'm probably in deep trouble.  And,
if I retrieve UTF-32 and know that it is text but not anything
else, I better not be on a system that that makes special use of
null bytes.  So I would assume that "TYPE U", interpreted as
always encoded in UTF-8, would work fine coming out of most
systems because they probably know how to do the conversation
already.  And, if someone requests "TYPE A", I'd much rather
than a system that notices, e.g., that some of the octets in the
file have their high bit on, return an error indication than
make something up (like zeroing out all of the high-order bits
or discarding a bit in the middle).

Since you used an email analogy, let me do that too.   While we
permit a MIME body part of application/octet-stream, which
basically translates into plain English as "I don't know or I'm
not willing to tell you -- your problem", we do not allow
text/plain charset="I don't have a clue what this is or how it
was encoded but I think it is text".  And, if the originating
system knows enough to specify that a body part is text/plain
and to specify a charset, it presumably knows enough to respond
intelligently to an fTP request for TYPE A / E / or presumably U
whether it knows how to do the conversion or not.  It is a
little different because the sending user or MUA might know even
if the operating system doesn't, but still...  And most of the
MUAs I know of will make an educated guess and, in practice,
usually get it right.

best,
   john

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 1 Dec 2020 01:36:42 +0100

> the wire or from UTF-8 or ASCII for incoming data.  

Generally, the Web (and the need for MIME sniffing) has shattered any hope I might have had there.  The problem really is that the FTP server is disconnected from all the application knowledge that made the file happen.

And that is very different from the situation…

> […] we do not allow
> and to specify a charset,

… where it is not the system, it is the application (mail client), often with some help of the user.
E.g., we have much better metadata on clipboard data than on files.
For files, the application can use heuristics whose failures at least *can* be corrected by someone sitting in front of the screen.
(But then, even with that I still get tons of mojibake in e-mail.)

Grüße, Carsten

---

From: John C Klensin <[email-masked]>
Date: Mon, 30 Nov 2020 20:29:53 -0500

--On Tuesday, December 1, 2020 01:36 +0100 Carsten Bormann
<[email-masked]> wrote:

>> disagree.  Keep in mind that a number of contemporary
>> operating systems use UTF-16, or even UTF-32, with some byte
>> ordering, internally.  They typically know they are doing
>> that, if only to be able to do an orderly conversation to
>> UTF-8 for putting over the wire or from UTF-8 or ASCII for
>> incoming data.  
> 
> Generally, the Web (and the need for MIME sniffing) has
> shattered any hope I might have had there.  The problem really
> is that the FTP server is disconnected from all the
> application knowledge that made the file happen.

But, unless those files were created, e.g., in the web, so is a
web browser.

> … where it is not the system, it is the application (mail
> client), often with some help of the user. E.g., we have much
> better metadata on clipboard data than on files. For files,
> the application can use heuristics whose failures at least
> *can* be corrected by someone sitting in front of the screen.

And I continue to have an interesting problem with the
vendor-preferred mail client on a well-known operating system
copying in the text of a previous message, which it had stored
in UTF-16, appending it to the message being composed in UTF-8
without changing the encoding, and then sending the result out
labeled charset=utf-8.  Receiving MUAs who sniff may be able to
make sense of that situation; those who actually follow the spec
are headed for trouble.

But one reasonable conclusion from that experience and your
comments above it that it is all hopeless and we should just
give up on independently developed standards letting the major
vendors and their browsers and MUAs do what they feel like and
either abandoning other applications or reverse-engineering
sufficiently that they conform to whatever those vendors are
doing.

It would save us a lot of meetings and arguments.

   john

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 1 Dec 2020 02:45:21 +0100

> it is all hopeless

You may be on to something…

But more seriously, the saying is “choose your battles”, and I think in this specific area (getting any processing, or even just the metadata right while serving up collections of files) it is really hard to make a difference.

Grüße, Carsten

---

From: John C Klensin <[email-masked]>
Date: Mon, 30 Nov 2020 22:07:22 -0500

--On Tuesday, December 1, 2020 02:45 +0100 Carsten Bormann
<[email-masked]> wrote:

> But more seriously, the saying is "choose your battles",
> and I think in this specific area (getting any processing, or
> even just the metadata right while serving up collections of
> files) it is really hard to make a difference.

Agreed.  I'm not really pushing this -- I certainly have not
posted any I-Ds on the subject in the last eight years.  The
questions were motivated by a hypothesis that there are some
people out there who think FTP is useful, are using it, and are
are using conforming implementations (e.g., the default really
is ASCII and not Image) and possibly a significant fraction of
its features.  Put differently, they are using FTP in a way that
takes advantage of its capabilities rather than using in a way
for which any number of other things, starting with rcp and/or
rsync and their clones, would do equally well.

Then _for that group_ (a group of which you are clearly not part
- and I don't have any problem with that at all), the questions
are whether a "cannot reliably convert to that TYPE" code and/or
an explicit Unicode TYPE would be helpful and worth supporting.
I agree with you that good-quality support would not be
straightforward on some (probably most) contemporary systems but
I'm not convinced that it is impossible if there were adequate
motivation.   However, the implementation issues are not even
worth exploring or probably even speculating about if no one is
interested.

best,
   john

---

From: "Theodore Y. Ts'o" <[email-masked]>
Date: Mon, 30 Nov 2020 22:07:59 -0500

> general, I think it is still true.

And yet, just recently on this mailing list, some of the people
arguing for the inherent superiority of the ftp protocol as compared
to http is that with ftp, it *Would* do conversions of text files
(e.g., crlf mangling, etc.)

Heh.

						- Ted

---

From: Joseph Touch <[email-masked]>
Date: Mon, 30 Nov 2020 20:11:39 -0800

> On Nov 30, 2020, at 7:07 PM, Theodore Y. Ts'o <[email-masked]> wrote:
> 
>> <[email-masked] <mailto:[email-masked]>> wrote:
>> 

Conversion can be useful. I can FTP a text file of Perl and run it just fine on different platforms.

Doing the same with HTTP either requires explicitly excising the surrounding HTML or doing CRLF conversion - still.

At least with FTP, it was a choice (ASCII vs binary).

Joe