File Size Difference between FTP and sftp

[Larry Gray]

Testing the Co:Z SFTP client and server. One of the files I am trying to send is an SVC dump. With sftp, I do not get the same number of bytes transferred as I do with FTP. I have used binary mode with both. I have tried with the mainframe being both the client and the server to both the PC or a unix box. The same number of bytes are always transferred with sftp whichever the platform or whoever is the server. The file in question is approximately 367MB.

Larry Gray

[Larry Gray]

I tried a tersed file as a second test. Same problem. The new file is smaller than the original file.

Larry Gray

[dovetail]

Can you provide more details:

1) what kind of file or dataset do you start with on z/OS?
If a dataset, what are its DCB attributes and how many records does it have?

2) what are you comparing with FTP - the size of the result file on the receiving system? What specifically are the sizes / how do they differ?

Thanks.

[Larry Gray]

1. The issue is with datasets, not files in the hfs. The first dataset was a standard SVC dump, DCB=(RECFM=FBS,LRECL=4160,BLKSIZE=24960). For the second file, the dataset was a standard TERSEd file, DCB=(RECFM=FB,LRECL=1024,BLKSIZE=10240).

2. For file size, I am comparing the size on the receiving end. The sftp size is smaller that the one transfered by FTP. I also OPUT'd the TERSE file to the hfs. The file size in the hfs matched the file size of the FTP file, but not the sftp file.

For the tersed file, the file size of the OPUT copy and the FTP copy on the receiving end was 7354368. The file size when sent by sftp was 7354266. The sftp size was the same whether I did a put from the mainframe or a get from the unix server.

Larry Gray

[dovetail]

We gave some binary dataset round-trip tests that work fine, but I'll try to reproduce the problem using something that matches your DCB attributes.

Thanks.

[dovetail]

This is a mystery to me. I created a binary FBS/1024 dataset with 7182 records and did this:

$cozsftp kirk@ubuntu
sftp>lzopts mode=binary
sftp>put //KIRK.TEST21.DATA /tmp/test21.data
Uploading //KIRK.TEST21.DATA to /tmp/test21.data.copy
ZosDataset: Opening dataset KIRK.TEST21.DATA for read with options: shr
//KIRK.TEST21.DATA 100% 7182KB 276.2KB/s 00:26
ZosDataset: Closing dataset //KIRK.TEST21.DATA - 7182 records read, 7354368 bytes sent

sftp>quit

On the Unix host:
> ls -al test*
-rw-r--r-- 1 kirk kirk 7354368 2008-11-17 10:12 test21.data

Which looks good.

1) Are you setting lzopts like I did above? Can you run again and post the output (like I have above)?
2) What Unix sftp server are you running? I tested with a Ubuntu/Debian linux server running OpenSSH 4.7_p1.
3) Try transferring the dataset using cozsftp to the z/OS system itself, to a HFS dataset.
4) If possible, try transferring to a different Unix distro and see if you get the same (bad) results.

[Larry Gray]

I ran the sftp to another mainframe running your sftp server. I also ran an ftp as a comparison. Output below. The last below is the output from the ls -l command showing the two files. The file being sent has 7182 lines with an lrecl of 1024. That would make it 7354368 bytes.

# cozsftp teclag5@mvsbtst
Co:Z sftp version: 1.1.0-Beta (5.0p1) 2008-10-20
Copyright (C) Dovetailed Technologies, LLC. 2008. All rights reserved.
Connecting to mvsbtst...
sftp> lzopts mode=binary
mode=binary
sftp> ls /+mode=binary
/+mode=binary
sftp> put //TECLAG4.MPTF2834.SERVLINK mptf2834.servlink
Uploading //TECLAG4.MPTF2834.SERVLINK to /u/people/teclag4/mptf2834.servlink
ZosDataset[I]: Opening dataset TECLAG4.MPTF2834.SERVLINK for read with options: shr
ZosSmf119Record[I]: getaddrinfo failed: EDC5000I No error occurred. (errno2=0x0523011C) for: mvsbtst:22
//TECLAG4.MPTF2834.SERVLINK 99% 7182KB 1.6MB/s 00:00 ETA
ZosDataset[I]: Closing dataset //TECLAG4.MPTF2834.SERVLINK - 7182 records read, 7354266 bytes sent
sftp>

# ftp mvsbtst
IBM FTP CS V1R9
Connecting to: mvsbtst.0998.lowes.com 172.26.142.30 port: 21.
220-TSTFTPT1 IBM FTP CS V1R9 at mvsbtst.0998.lowes.com, 20:46:35 on 2008-11-17.
220 Connection will close if idle for more than 5 minutes.
NAME (mvsbtst:TECLAG4):
teclag5
>>> USER teclag5
331 Send password please.
PASSWORD:

>>> PASS
230 TECLAG5 is logged on. Working directory is "TECLAG5.".
Command:
cd /u/people/teclag4
>>> CWD /u/people/teclag4
250 HFS directory /u/people/teclag4 is the current working directory
Command:
bin
>>> TYPE I
200 Representation type is Image
Command:
put 'TECLAG4.MPTF2834.SERVLINK' mptf2834a.servlink
>>> SITE FIXrecfm 1024 LRECL=1024 RECFM=FB BLKSIZE=10240
200 SITE command was accepted
>>> PORT 172,26,142,16,4,10
200 Port request OK.
>>> STOR mptf2834a.servlink
125 Storing data set /u/people/teclag4/mptf2834a.servlink
250 Transfer completed successfully.
7354368 bytes transferred in 0.230 seconds. Transfer rate 31975.52 Kbytes/sec.
Command:

# ls -l mptf*
-rw-rw-rw- 1 TECLAG5 OMVSGRP 7354266 Nov 17 15:45 mptf2834.servlink
-rw-r----- 1 TECLAG5 OMVSGRP 7354368 Nov 17 15:47 mptf2834a.servlink

[Larry Gray]

In regard to the OpenSSH version, both the mainframe and the AIX box seem to be using OpenSSH_3.8.1p1. I used the ssh -V command to get this. I do not have the ability to upgrade the version on the AIX box.

Larry Gray

[dovetail]

OK - I found the problem. Sorry that it took this long.

We have a bug - the record transformation code is being setup to trim trailing spaces from the dataset records. This shouldn't be the case for "binary" mode. Our test cases didn't pick it up because we unfortunately didn't have a case of RECFM=F* with actual test data that had trailing spaces.

Its failing for you as both a client and as a server because the same code is used to read the dataset. We'll develop a fix and post a new beta version soon. Sorry for the inconvenience.

[dovetail]

This problem has been fixed in a new release.

See: http://dovetail.com/forum/viewtopic.php?t=727

Thank you very much for reporting this; we are sorry for the inconvenience.