Appending part of the source file name to target file name

Discussion of Co:Z sftp, a port of OpenSSH sftp for z/OS
Post Reply
giltjr
Posts: 25
Joined: Wed Jul 07, 2010 9:06 pm

Appending part of the source file name to target file name

Post by giltjr »

We have been running V1.7.2 forever and we just upgraded to V7.0.2. We have one remote site where something weird is happening. I have tracked it down what I think is how they are responding to sftp function.

Our job has:

Code: Select all

     cd /data/isr/input/    
     put //DD:DOWNLOAD $RFIL
Under V1.7.2 the job output shows:

Code: Select all

     cozsftp> cd /data/isr/input/                              
     cozsftp> put //DD:DOWNLOAD isr_050922.ZIP                 
     Uploading //DD:DOWNLOAD to /data/isr/input/isr_050922.ZIP 
But going to this one host (at least so far it is just this one) we see this under V7.0.2:

Code: Select all

     cozsftp> cd /data/isr/input/                                          
     cozsftp> put //DD:DOWNLOAD isr_051022.ZIP                             
     Uploading //DD:DOWNLOAD to /data/isr/input/isr_051022.ZIP/DD:DOWNLOAD 

Notice "/DD:DOWNLOAD" is append to the remote file name. When I send to a test sftp server using V7.0.2 everything works as normal.

I turned on ssh debug (-vvv) and to the test server where everything works I see:

Code: Select all

     %10:26:46.201486% debug3: remote_is_dir: path="/data/isr/input/isr_051122.ZIP" isDir=0 
And later on:

Code: Select all

      %10:26:46.203645% debug3: Sent message SSH2_FXP_OPEN I:5 O:x1a P:/data/isr/input/isr_051122.ZIP
So everything seems normal and correct. But going to the server that is having the problem I see:

Code: Select all

     %10:26:16.673519% debug3: remote_is_dir: path="/data/isr/input/isr_051122.ZIP" isDir=1
and later on:

Code: Select all

   %10:26:16.675440% debug3: Sent message SSH2_FXP_OPEN I:5 O:x1a P:/data/isr/input/isr_051122.ZIP/DD:DOWNLOAD

I'm not sure what the "remote_is_dir" is or means or why their server seems to return "1" and my test server returns a "0".

I noticed under V1.7.2 I don't see the "remote_is_dir" in the ssh debug, so I'm assuming that this is something that was added between our ancient level of V1.7.2 and V7.0.2.

I hope this makes sense.
coz
Posts: 391
Joined: Fri Jul 30, 2004 5:29 pm

Re: Appending part of the source file name to target file name

Post by coz »

Please refer to this section of the documentation: https://dovetail.com/docs/sftp/dsn-determination.html
giltjr
Posts: 25
Joined: Wed Jul 07, 2010 9:06 pm

Re: Appending part of the source file name to target file name

Post by giltjr »

Thanks for the update, but I'm still confused. The section you refered me to seems to deal with how the name is determined when the file is being placed on a z/OS system. We are sending from z/OS using the cozsftp client to a non-z/OS system, so I don't think this section applies here.

So to must make sure. because I was not clear before, we are a z/OS system using cozsftp client to send to a non-z/OS system. Our job is setup to do:

cd /data/isr/input/
put //DD:DOWNLOAD $RFIL

And we set RFIL to isr_050922.ZIP.

Under Coz:SFTP v1.7.2 the results were:

cozsftp> put //DD:DOWNLOAD isr_050922.ZIP
Uploading //DD:DOWNLOAD to /data/isr/input/isr_050922.ZIP

And the remote file name ends up being /data/isr/input/isr_051022.ZIP


Under Coz:SFTP v7.0.2 we see:

cozsftp> put //DD:DOWNLOAD isr_051022.ZIP
Uploading //DD:DOWNLOAD to /data/isr/input/isr_051022.ZIP/DD:DOWNLOAD

So the file name on the remote system ends up being /data/isr/input/isr_051022.ZIP/DD:DOWNLOAD which is not correct.

We have thousands of jobs going to hundreds of hosts and this only occurs with one host.
giltjr
Posts: 25
Joined: Wed Jul 07, 2010 9:06 pm

Re: Appending part of the source file name to target file name

Post by giltjr »

O.K., I think I know what is going on. Somewhere along the lines one of the versions of CoZ:SFTP or one of its "sub-programs" dealing with sftp added a function that check to see if the remote path on the put command is a directory or not, this is the "remote_is_dir" function/call.

If the remote server returns "0" that means the remote path on the put command is NOT a directory and what you have specified on the put command is a full path and filename.

If the remote server returns "1" that means the remote path on the put command is a directory and sftp appends the local file name to the end.

Although we are specifying a file name, this customers server is returning that name is a directory, so the source file name is being appended to what Coz:SFTP believes is a directory name.

Does the above make sense to you, I not sure my description is clear.

We are trying to contact the customer to see what server they are using and let them know it has a problem.

In the mean time we are trying a work around by copying the source file on our system from MVS to OMVS and make the name the same as we put on the customers system. Something like:

!cp "//'OUR.MVS.FILE.NAME'" /tmp/isr_050222.ZIP
cd /data/isr/input
put /tmp/isr_050222.ZIP

And see what happens.

Does all of this make sense to you?
giltjr
Posts: 25
Joined: Wed Jul 07, 2010 9:06 pm

Re: Appending part of the source file name to target file name

Post by giltjr »

Oh, the "-vvv" did not show the "remote_is_dir", I used export COZ_LOG=F in order to see that.
coz
Posts: 391
Joined: Fri Jul 30, 2004 5:29 pm

Re: Appending part of the source file name to target file name

Post by coz »

I've finally had a chance to look at this in more detail.

As you have discovered, the remote server is declaring that a .ZIP file is a directory object. From an SFTP perspective, this is almost certainly wrong and will break almost every existing SFTP client. For example, the upstream OpenSSH implementation will do the same thing that we currently do, which is to append the source filename to what the remote system reports as a directory. This will fail on the remote side, unless the sftp-server there does something dodgy. I suspect that the remote OS is Windows along with some custom sftp-server implementation, so I guess anything is possible :)

After digging through 12 years of code, I can confirm that *very* early versions of Co:Z ignored the directory designation reported from the remote server when the source object was a data set and just attempted to replace the target name. In the case of your .ZIP file, this worked, albeit by accident. This error was fixed nine years ago, in Co:Z release 2.4.0. It is documented in the release notes.
Post Reply