sftp problem (posted for another person)

[john.mckown]

I am posting this for Richard Pinion. He indicated that he cannot seem to be able to log on to this forum. I have posted his entire message to me below. Has anyone had this problem? I cannot reproduce it myself on z/OS 1.12. I do _NOT_ know what he means by "CoZ sftp client". I don't know of any such beastie.

==== transcript below ===

We have the free version of CoZ, so we don't have support. I've tried to sign up on
their public forum, but it keeps asking for a code to prevent SPAM. But, I have no
idea where the code comes from.

Anyway, would you mind posting this question on the CoZ SFTP board. We have a
situation, when we use the CoZ SFTP client, using binary mode, to transfer a file from
a Linux based SFTP server to a z/OS ZFS file we get X'15' embedded in the ZFS file.
Which we don't want! We are using CoZ's TODSN on the mainframe to break
the records apart based on x'FF01', which was originally inserted by a VSE binary
FTP using STRUCT R.

However, when we use IBM's SFTP client, we don't get the X'15' and we are able
to break the ZFS file apart like we want.

I've gone through their documentation, and can't find a reason for the X'15'. I've
even tried linemode none (which isn't supposed to be used for a binary mode
transfer).

Thanks for your help, and I hope you don't mind. But I know you're a big fan
and big user of CoZ on the mainframe.

[dovetail]

Richard should be able register for the forum. The code is available on our support page.

Its not clear to me what the problem is - please ask Richard to post the Co:Z SFTP JCL/script that he is using, along with the output messages that he gets from running a test case.

[rpinion]

I'm not exactly the sharpest knife in the drawer, but I finally found the code to register . I'll past the %ISHELL VIEW of the ZFS file that has the X'15' characters embeded. At this location the data is not supposed to have a X'15', I have verified that by looking at the file on the non-mainframe SFTP server using HexD.

Select one or more files with / or action codes.

EUID=0 /hltheast/
Type Filename
_ Dir .
_ Dir ..
v File coztest
_ File test

+-----------------------------------------+
| Enter Record Length | s.
| |
| Record length . . . 32760 |
| |
| |
| |
+-----------------------------------------+

BROWSE /hltheast/coztest
Command ===>
*********************************************************** Top of Data ********************


-------------------------------------------------------------------------------------------
002SE928..9 E......023080 003S216.6 D......023080
44444444444FFFECFFF41F4444C000000FFFFFF4444444444444FFFEFFF4F44444C000000FFFFFF4444444444444
0000000000000225928B590000500C00C02308000000000000000032216B600000400C00C0230800000000000000


1 J E S 2 J O B L O G -- S Y S T E M L D A -- N
0
08.42.57 JOB09881 ---- FRIDAY, 23 JUN 2017 ----
08.42.57 JOB09881 IRR010I USERID LDARP1 IS ASSIGNED TO THIS JOB.
08.42.57 JOB09881 ICH70001I LDARP1 LAST ACCESS AT 07:45:38 ON FRIDAY, JUNE 2
08.42.57 JOB09881 $HASP373 LDARP1HE STARTED - INIT 1 - CLASS A - SYS
08.42.58 JOB09881 - --TIMINGS (MINS.)-
08.42.58 JOB09881 -JOBNAME STEPNAME PROCSTEP RC EXCP CPU SRB CLOC
08.42.58 JOB09881 -LDARP1HE DELETE DELETE 00 15 .00 .00 .0
08.43.24 JOB09881 -LDARP1HE STEP10B 00 38696 .04 .00 .4
08.43.24 JOB09881 -LDARP1HE ENDED. NAME-RWP TOTAL CPU TIME=
08.43.24 JOB09881 $HASP395 LDARP1HE ENDED - RC=0000
0------ JES2 JOB STATISTICS ------
- 23 JUN 2017 JOB EXECUTION DATE
- 34 CARDS READ
- 109 SYSOUT PRINT RECORDS
- 0 SYSOUT PUNCH RECORDS
- 12 SYSOUT SPOOL KBYTES
- 0.44 MINUTES EXECUTION TIME
1 //LDARP1HE JOB (LDA),'RWP',
// CLASS=A,
// MSGCLASS=X,REGION=0M,
// NOTIFY=&SYSUID
//**
IEFC653I SUBSTITUTION JCL - (LDA),'RWP',CLASS=A,MSGCLASS=X,REGION=0M,
2 //DELETE EXEC DELFILE,DELDSN=LDA.VTAPE.HLTHEAST.YTD.KMMRMMD
3 XXDELETE EXEC PGM=IEFBR14
4 XXDEL DD DSN=&DELDSN,
XX DISP=(MOD,DELETE),UNIT=SYSDA,SPACE=(TRK,(1,0))
IEFC653I SUBSTITUTION JCL - DSN=LDA.VTAPE.HLTHEAST.YTD.KMMRMMD,DISP=(
5 XXSYSOUT DD SYSOUT=*
6 XXSYSPRINT DD SYSOUT=*
XX*
XX*** DISP=(MOD,DELETE,DELETE),UNIT=SYSDA
//**
//** upload file from sftp server
//**
7 //STEP10B EXEC PGM=COZBATCH
8 //STEPLIB DD DISP=SHR,DSN=LDA.COZ.LOADLIB
9 //STDIN DD *
10 //UPLOAD DD PATH='/hltheast/coztest',
// PATHOPTS=(OWRONLY,OCREAT,OTRUNC),
// PATHMODE=SIRWXU
STMT NO. MESSAGE
2 IEFC001I PROCEDURE DELFILE WAS EXPANDED USING SYSTEM LIBRARY USER.PRO
ICH70001I LDARP1 LAST ACCESS AT 07:45:38 ON FRIDAY, JUNE 23, 2017
IEF236I ALLOC. FOR LDARP1HE DELETE DELETE
IGD103I SMS ALLOCATED TO DDNAME DEL
IEF237I JES2 ALLOCATED TO SYSOUT
IEF237I JES2 ALLOCATED TO SYSPRINT
IEF142I LDARP1HE DELETE DELETE - STEP WAS EXECUTED - COND CODE 0000
IGD105I LDA.VTAPE.HLTHEAST.YTD.KMMRMMD DELETED, DDNAME=DEL
IEF285I LDARP1.LDARP1HE.JOB09881.D0000102.? SYSOUT
IEF285I LDARP1.LDARP1HE.JOB09881.D0000103.? SYSOUT
IEF373I STEP/DELETE /START 2017174.0842
IEF032I STEP/DELETE /STOP 2017174.0842
CPU: 0 HR 00 MIN 00.00 SEC SRB: 0 HR 00 MIN 00.00 SEC
VIRT: 4K SYS: 244K EXT: 0K SYS: 9896K
ATB- REAL: 416K SLOTS: 0K
VIRT- ALLOC: 16M SHRD: 0M
IEF236I ALLOC. FOR LDARP1HE STEP10B
IEF237I 0A9B ALLOCATED TO STEPLIB
IEF237I JES2 ALLOCATED TO STDIN
IGD103I SMS HFS FILE ALLOCATED TO DDNAME UPLOAD
IEF237I JES2 ALLOCATED TO SYSOUT
IEF237I 0AA6 ALLOCATED TO SYS00004
IEF285I TCPIP.TCPIP.DATA KEPT
IEF285I VOL SER NOS= LDA023.
IEF237I 0D05 ALLOCATED TO SYS00006
IEF285I TCPIP.STANDARD.TCPXLBIN KEPT
IEF285I VOL SER NOS= LDARES.
IEF285I LDARP1.LDARP1HE.JOB09881.D0000104.? SYSOUT
IEF142I LDARP1HE STEP10B - STEP WAS EXECUTED - COND CODE 0000
IEF285I LDA.COZ.LOADLIB KEPT
IEF285I VOL SER NOS= LDAV09.
IEF285I LDARP1.LDARP1HE.JOB09881.D0000101.? SYSIN
IGD104I HFS FILE WAS RETAINED, DDNAME IS (UPLOAD )
FILENAME IS (/hltheast/coztest)
IEF373I STEP/STEP10B /START 2017174.0842
IEF032I STEP/STEP10B /STOP 2017174.0843
CPU: 0 HR 00 MIN 02.77 SEC SRB: 0 HR 00 MIN 00.01 SEC
VIRT: 84K SYS: 756K EXT: 10544K SYS: 10688K
ATB- REAL: 428K SLOTS: 0K
VIRT- ALLOC: 18M SHRD: 0M
IEF375I JOB/LDARP1HE/START 2017174.0842
IEF033I JOB/LDARP1HE/STOP 2017174.0843
CPU: 0 HR 00 MIN 02.77 SEC SRB: 0 HR 00 MIN 00.01 SEC
1
CoZBatch[N]: Copyright (C) 2005-2013 Dovetailed Technologies LLC. All rights re
CoZBatch[N]: version 2.4.1 2013-06-24
CoZBatch: executing progname=login-shell="-/bin/sh"
Co:Z SFTP version: 2.4.1 (5.0p1) 2013-06-24
Copyright (C) Dovetailed Technologies, LLC. 2008-2013. All rights reserved.
ZosSettings[W]: error reading configuration file (//.ssh/cozsftp_config) - EDC5
as a function parameter. (errno2=0xC00B0286)
fromdsn(LDA.HLTHEAST.PARMLIB(PW))[N]: 1 records/80 bytes read; 8 bytes written
cozsftp> lzopts mode=binary,servercp=ISO8859-1 #
mode=binary servercp=ISO8859-1
cozsftp> cd '/home/jail/home/healtheast/HECS01-01/To_LDA'
cozsftp> get HE_MR_HIST_YTD //DD:UPLOAD
Fetching /home/jail/home/healtheast/HECS01-01/To_LDA/HE_MR_HIST_YTD to //DD:UPL
ZosDataset: Opening dataset DD:UPLOAD for write
ZosDataset: Closing dataset ///hltheast/coztest - 63317248 bytes received, 6
CoZBatch: returning rc=exitcode=0

[dovetail]

With mode=binary, servercp is ignored. The file will just be transmitted in binary. The bytes should be the same between the remote file and the target data set.

[rpinion]

That is what I think should be happening too. Oh well, at least we have a working solution. I just wish I could determine if it is something I'm doing incorrectly, or something else.

[rpinion]

Did some more testing and have learned that if I use the following to create the output file, I get X'15' characters.

get $remotefile //DD:UPLOAD
EOB
//UPLOAD DD PATH='/hltheast/coztest',
// PATHOPTS=(OWRONLY,OCREAT,OTRUNC),
// FILEDATA=('BINARY'),
// PATHMODE=SIRWXU

But if I do this, I do NOT get the X'15' characters.

cd '/home/jail/home/healtheast/HECS01-01/To_LDA'
get $remotefile /hltheast/coz1

I don't need to understand the "whys", just good to know that I can again use CoZ!

[dovetail]

Its difficult to tell what is actually going on - it could be that there are options from your profile that are being applied that we can't see that might affect things.

Either do this:

1) install a more current version of Co:Z, which will print out the options in effect for each transfer

2) enable a trace by adding this to the beginning of your script:

export COZ_LOG=T

along with this, it would very helpful to see a small section of the data (assuming that it is not confidential) on both ends:

- ISPF with HEX ON
- remote server hex
- for example with Linux use the "xxd" command

please send these diagnostics to info at dovetail.com

[rpinion]

Since I have been able circumvent the problem by writing directly to the HFS/ZFS file, I don't think I'll pursue the issue any longer. When you think about it, it was not the most intelligent thing to do, write to //DD and then have the DD statement use HFS/ZFS. The only reason I did it this way, this was the first time I ever needed to write to a HFS/ZFS file. All CoZ transfers in the past have been to fixed length record data sets, TRSMAIN'ed data sets.

As I stated earlier, this data set came from a VSE system. And they are sending us VB data sets. As far as we know, TRSMAIN is not available on VSE (actually it is, but it only works with library members, and the VSE admin refused to try it). We had them FTP from the VSE machine using STRUCT R which inserts X'FF01' characters between the records. Your todsn -l 0xFF01 program was perfect for
reconstructing the VB records!

[dovetail]

Oh! I completely missed this - that you were pointing your DD to a zFS file.
Yeah, that doesn't make much sense.

What you saw was the x'15's were being added by the z/OS access method, as part of its conversion from QSAM to zFS.