File transfer from client to server fails

[tombuch]

In the log for the COZ SFTP server we get:

Co:Z sftp-server version: 1.7.2 (5.0p1) 2010-07-23
Copyright (C) Dovetailed Technologies, LLC. 2008. All rights reserved.
Ý08.199¨ session opened for local user RETFTP2 from Ý204.58.216.95¨
Ý08.200¨ received client version 6
Ý08.227¨ debug3: request 148: stat
Ý08.227¨ stat name "D.UTT.C7298.RP240.MNTH.TEST"
Ý08.227¨ debug3: request 148: sent status 2
Ý08.227¨ sent status No such file
Ý08.238¨ debug3: request 149: open flags 11
Ý08.238¨ open "D.UTT.C7298.RP240.MNTH.TEST" flags READ,WRITE,CREATE mode 0666
ZosPosixFileÝE¨: Open flags (O_RDWR) not supported in text mode: D.UTT.C7298.RP240.MNTH.TEST 0x0083, 0x0000
Ý08.239¨ debug3: request 149: sent status 2
Ý08.239¨ sent status No such file
Ý08.250¨ debug3: request 150: open flags 11
Ý08.250¨ open "./D.UTT.C7298.RP240.MNTH.TEST" flags READ,WRITE,CREATE mode 0666
ZosPosixFileÝE¨: Open flags (O_RDWR) not supported in text mode: ./D.UTT.C7298.RP240.MNTH.TEST 0x0083, 0x0000
Ý08.251¨ debug3: request 150: sent status 2
Ý08.251¨ sent status No such file
Ý15.118¨ debug1: read eof
Ý15.119¨ session closed for local user RETFTP2 from Ý204.58.216.95¨


Is there a work around this O_RDWR?

[dovetail]

Apparently, the remote sftp client is doing a "put", but it is sending a SFTP OPEN packet with open mode = O_RDWR.

What SFTP client are you using? We have seen this problem before with an OpenVMS implementation (from "Process Software"). Please contact them to see if they have a fix. We believe that opening a file "RDWR" when doing a sftp "put" is a mis-interpretation of the spec.

Please see this related thread, which is similar but with a dataset and not a Unix file: http://dovetail.com/forum/viewtopic.php?f=8&t=1012

BTW: if the remote client uses "binary" mode, we will allow the file to be used RDWR. With text mode, this is not possible, since we must edit the incoming data stream which would possibly involve replacing line terminators with those of differing lengths.

[tombuch]

At times in the middle of a transfer, I get the following:

ZosDatasetÝE¨: dataset write error: seek not allowed
Ý95.098¨ process_write: write failed
Ý95.098¨ debug3: request 1522: sent status 4
Ý95.098¨ sent status Failure
Ý95.142¨ debug1: request 1523: write "//D.UTT.C7298.RP240.MNTH.TEST" (handle 0) off 48672000 len 32000
ZosDatasetÝE¨: dataset write error: seek not allowed
Ý95.142¨ process_write: write failed

And then the connection is reset....

Is this the same problem?

Thank you...

[dovetail]

When writing to datasets, the Co:Z SFTP server requires that the data be written sequentially. The SFTP client sends a stream of data and we convert it on-the-fly to records - conerting codepages, decoding line-termintors, trimming etc.

Nearly all SFTP clients work with this properly using the client's "put" command. But your client (still unnamed?) apparently writes non-sequentially (seeks).

[tombuch]

The client is Chilkat client.

Here is part of the initial log for the client:


Transfer was started by (deleted text)on computer PRUNED: 01/06/12 8:01:18 AM
Sending (deleted text)((deleted text)) to (deleted text)...
SFTPConnectToSCDC==>SFTP Object Unlocked!
SFTPConnectToSCDC==>SFTP Connect to (deleted text) was successful!
SFTPConnectToSCDC==>SFTP private OpenSSH key loaded successful!
SFTPConnectToSCDC==>SFTP From Open SSH Private key was successful!
SFTPConnectToSCDC==>SFTP Authenticate Public key for userretftp2 with (deleted text) was successful!
ChilkatLog:
AuthenticatePk:
DllDate: Aug 3 2011
UnlockPrefix: T02042012SSH
Username: PRUNED:(deleted text)
Architecture: Little Endian; 32-bit
Language: ActiveX
SshVersion: SSH-2.0-OpenSSH_3.8.1p1
SftpVersion: 0
login: retftp2
SentServiceReq: ssh-userauth
ssh-userauth service accepted.
Using a DSA key.
keyType: DSA private key
Getting DSA key parts...
publicKeyBlobSize: 817
msgPayloadSize: 876
Sent public-key request.
OK to proceed with publickey authentication.
dssSigLen: 40
Sent public-key request with signature.
Public-key authentication succeeded.
Success.

SFTPConnectToSCDC==>SFTP Initialize SFTP with (deleted text) was successful!
SendRP240MonthlyToSCDC==>ChilkatLog:
UploadFileByName:
DllDate: Aug 3 2011
UnlockPrefix: T02042012SSH
Username: PRUNED:(deleted text)
Architecture: Little Endian; 32-bit
Language: ActiveX
SshVersion: SSH-2.0-OpenSSH_3.8.1p1
SftpVersion: 3
ChilkatObjectID: 100

[dovetail]

I'm not familiar with this SFTP client, but it seems to do two things are not compatible:

1) When doing a "put", it (sometimes?) opens the file in O_RDWR mode. This IMO is just wrong
2) It sometimes(?) uploads a file non-sequentially. In other words, each write packet needs to start at the file offset that follows the previous write packet.

[tombuch]

I thought I fixed the problem... I changed the parms in sftp-server.rc

I changed:

export SFTP_ZOS_OPTIONS="mode=text,pad=' ',notrim,linerule=crlf"

to:

export SFTP_ZOS_OPTIONS="mode=text"


But now it is failing again... Here is the log:

Ý73.551¨ process_write: write failed
Ý73.551¨ debug3: request 4471: sent status 4
Ý73.551¨ sent status Failure
Ý73.576¨ debug1: request 4472: write "//D.NAME" (handle 0) off 143040000 len 32000
ZosDatasetÝE¨: dataset write error: seek not allowed
Ý73.576¨ process_write: write failed
Ý73.576¨ debug3: request 4472: sent status 4
Ý73.576¨ sent status Failure
Ý73.604¨ debug1: request 4473: write "//D.NAME" (handle 0) off 143072000 len 32000
ZosDatasetÝE¨: dataset write error: seek not allowed
Ý73.604¨ process_write: write failed
Ý73.604¨ debug3: request 4473: sent status 4
Ý73.604¨ sent status Failure
Ý73.604¨ debug3: request 4474: close handle 0
Ý73.604¨ close "//D.NAME" bytes read 0 written 142720000
ZosDatasetÝI¨: Closing dataset //D.NAME before completion - 142720000 bytes received, 136053 records written
Ý73.868¨ debug3: request 4474: sent status 0
Ý73.868¨ sent status Success
Ý73.868¨ debug1: read eof
Ý73.868¨ session closed for local user RETFTP2 from Ý111.34.222.123¨

I have changed the IP address and the dataset name above...

Is there any other documentation that could help with this?
This only fails with big files....338050 records of 1047 record length...

Any advise would help...

Thank you...

Tom

[dovetail]

Co:Z SFTP streams data directly to z/OS datasets, so data must be written to these datasets in sequential order. This is also a restriction when writing to z/OS Unix files in text mode with certain options (like trimming, changing line terminators, or any transformation that would change the length of data as it is written).

The following message indicates that your SFTP client (Chilkat???) is trying to write data that is not contiguous:

ZosDatasetÝE¨: dataset write error: seek not allowed

Background: In the SFTP protocol, each "write" packet includes the data, the length, and the offset in the file to write it at. When writing to datasets, it is not allowed to skip or back up (to "seek").

To see what blocks are being written, you can turn on a trace of the ZosDataset component by issuing this command from the client before doing the "put":

ls /+loglevel=ZosDataset=T

This will create log messages something like this:

ZosDataset (write, off=0 len=32768)
ZosDataset (write, off=32768 len=32768)
...

Each write must follow contiguously the previous when writing to a ZosDataset. I cannot imagine why an SFTP client would not choose to write data contiguously - I have not seen one that didn't. Perhaps there is some kind of "checkpoint / restart" facility in this client, or maybe just a bug.

[dovetail]

BTW - you could also temporarily turn on ZosDataset tracing by adding this to sftp-server.rc -

export SFTP_ZOS_OPTIONS="mode=text,loglevel=ZosDataset=T"