SFTP: Incomplete file when size changes during transfer #4344
Description
I did this
Got a file using sftp but the received file was incomplete.
I expected the following
A complete file.
curl/libcurl version
curl 7.65.3 (x86_64-pc-cygwin) libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4) libssh/0.8.7/openssl/zlib nghttp2/1.37.0
Release-Date: 2019-07-19
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP TrackMemory UnixSockets
operating system
Client on Cygwin
Server is a HPE NonStop.
Under rather unique circumstances, curl using sftp gets less data than sftp does when used directly. curl seems to stop transfer at the reported eof rather than reading until eof. sftp gets the entire file. The file below is on a HPE NonStop system in their "Guardian" file system. This file ("an edit file") has spaces compressed on disk and put back upon read. That is, the reported size is sometimes less than the amount of data returned.
Here's how the file is reported:
: ssh bvoris@ssg-1.houston.hpecorp.net "ls -og /G/ipsv1/t9633abc/dfmtr"
-r-x------ 1 84518 May 11 2010 /G/ipsv1/t9633abc/dfmtr
: curl -vs sftp://bvoris@ssg-1.houston.hpecorp.net/G/ipsv1/t9633abc/dfmtr -O
* STATE: INIT => CONNECT handle 0x600077118; line 1356 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* STATE: CONNECT => WAITRESOLVE handle 0x600077118; line 1397 (connection #0)
* Trying 16.209.76.76:22...
* TCP_NODELAY set
* STATE: WAITRESOLVE => WAITCONNECT handle 0x600077118; line 1476 (connection #0)
* Connected to ssg-1.houston.hpecorp.net (16.209.76.76) port 22 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x600077118; line 1532 (connection #0)
* Marked for [keep alive]: SSH default
* User: bvoris
* Known hosts: /home/BVoris/.ssh/known_hosts
* SSH 0x600079200 state change from SSH_STOP to SSH_INIT (line 2115)
* SSH 0x600079200 state change from SSH_INIT to SSH_S_STARTUP (line 582)
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x600077118; line 1547 (connection #0)
* SSH 0x600079200 state change from SSH_S_STARTUP to SSH_HOSTKEY (line 595)
* SSH 0x600079200 state change from SSH_HOSTKEY to SSH_AUTHLIST (line 605)
* SSH 0x600079200 state change from SSH_AUTHLIST to SSH_AUTH_PKEY_INIT (line 628)
* Authentication using SSH public key file
* Completed public key authentication
* SSH 0x600079200 state change from SSH_AUTH_PKEY_INIT to SSH_AUTH_DONE (line 693)
* Authentication complete
* SSH 0x600079200 state change from SSH_AUTH_DONE to SSH_SFTP_INIT (line 807)
* SSH 0x600079200 state change from SSH_SFTP_INIT to SSH_SFTP_REALPATH (line 833)
* SSH CONNECT phase done
* SSH 0x600079200 state change from SSH_SFTP_REALPATH to SSH_STOP (line 850)
* STATE: PROTOCONNECT => DO handle 0x600077118; line 1566 (connection #0)
* DO phase starts
* SSH 0x600079200 state change from SSH_STOP to SSH_SFTP_QUOTE_INIT (line 2332)
* SSH 0x600079200 state change from SSH_SFTP_QUOTE_INIT to SSH_SFTP_GETINFO (line 868)
* SSH 0x600079200 state change from SSH_SFTP_GETINFO to SSH_SFTP_TRANS_INIT (line 1060)
* SSH 0x600079200 state change from SSH_SFTP_TRANS_INIT to SSH_SFTP_DOWNLOAD_INIT (line 1085)
* SSH 0x600079200 state change from SSH_SFTP_DOWNLOAD_INIT to SSH_SFTP_DOWNLOAD_STAT (line 1485)
* SSH 0x600079200 state change from SSH_SFTP_DOWNLOAD_STAT to SSH_STOP (line 1629)
* DO phase is complete
* STATE: DO => DO_DONE handle 0x600077118; line 1621 (connection #0)
* STATE: DO_DONE => PERFORM handle 0x600077118; line 1743 (connection #0)
{ [84518 bytes data]
* readwrite_data: we're done!
* nread <= 0, server closed connection, bailing
* STATE: PERFORM => DONE handle 0x600077118; line 1933 (connection #0)
* multi_done
* SSH 0x600079200 state change from SSH_STOP to SSH_SFTP_CLOSE (line 2390)
* SFTP DONE done
* SSH 0x600079200 state change from SSH_SFTP_CLOSE to SSH_STOP
sftp shows this version information:
debug1: Local version string SSH-2.0-OpenSSH_8.0
debug1: Remote protocol version 2.0, remote software version 1.37g sshlib: T9999L02_14JUL2017_comForte_SSH2_0104:\\SSG.$SSH00
debug1: no match: 1.37g sshlib: T9999L02_14JUL2017_comForte_SSH2_0104:\\SSG.$SSH00
Then, using sftp get /G/ipsv1/t9633abc/dfmtr dfmtr.sftp gives this:
: ls -do dfmtr*
-rw-r--r--+ 1 BVoris 84518 Sep 12 21:29 dfmtr
-rwx------+ 1 BVoris 91332 Sep 12 21:31 dfmtr.sftp*
Could curl be changed to just read the file until eof, like sftp?
Although this server will always give a short output for a given file, isn't it possible on other systems if some adds data after the size is got and before the transfer starts?