No results in response to an ls command

[DClassic53]

One of my users is having a problem with no response to an ls command to list the files in the remote directory. When he is given the name of a file that is there, a get for the filename is successful. He just can not list the file(s) in the directory. The remote ssh server is windows based. Any suggestions?

[dovetail]

Just to verify: you are using the Co:Z SFTP client on z/OS connecting to an SFTP server on Windows.

The best thing to do would be to run with -vvv added to the cozsftp command line to generate a trace of the packets are being sent and received.

If the answer doesn't reveal itself, please post an isolated test case log with -vvv (please scrub any possibly confidential information).

[DClassic53]

Here is the session with -vvv ...


$ cozsftp -vvv 'site.ext\user'@sftp.site.com
Co:Z SFTP version: 2.1.1 (5.0p1) 2012-03-16
Copyright (C) Dovetailed Technologies, LLC. 2011. All rights reserved.
Connecting to sftp.site.com...
[94.199] debug3: connect_to_server arg=/bin/ssh
[94.199] debug3: connect_to_server arg=-oForwardX11 no
[94.199] debug3: connect_to_server arg=-oForwardAgent no
[94.199] debug3: connect_to_server arg=-oClearAllForwardings yes
[94.199] debug3: connect_to_server arg=-v
[94.199] debug3: connect_to_server arg=-v
[94.199] debug3: connect_to_server arg=-v
[94.199] debug3: connect_to_server arg=-lsite.ext\\user
[94.199] debug3: connect_to_server arg=-oProtocol 2
[94.199] debug3: connect_to_server arg=-s
[94.199] debug3: connect_to_server arg=sftp.site.com
[94.199] debug3: connect_to_server arg=sftp
[94.210] debug2: setting ssh _CEE_RUNOPTS=HEAP(8M,1M,,FREE),ENVAR("_CEE_REALLOC_CONTROL=256K,25")
OpenSSH_5.0p1, OpenSSL 1.0.1c 10 May 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: RNG is ready, skipping seeding
debug1: zsshSmfSetConnSmfStatus: SMF status is 0
debug1: Rhosts Authentication disabled, originating port will not be trusted.
debug2: ssh_connect: needpriv 0
debug1: Connecting to sftp.site.com [999.999.999.999] port 22.
debug1: Connection established.
debug1: cipher_init: none from source OpenSSL
debug1: cipher_init: none from source OpenSSL
debug3: zsshGetpw: passwd name=MYUSER, uid=40095, gid=36, dir=/u/myuser, shell=/bin/sh
debug3: Not a RSA1 key file /u/myuser/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /u/myuser/.ssh/id_rsa type 1
debug1: identity file /u/myuser/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 in RemotelyAnywhere 11.0.2673
debug1: match: OpenSSH_5.3 in RemotelyAnywhere 11.0.2673 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.0
debug2: fd 3 setting O_NONBLOCK
debug3: RNG is ready, skipping seeding
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug1: mac_setup_by_id: hmac-md5 from source OpenSSL
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: mac_setup_by_id: hmac-md5 from source OpenSSL
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 129/256
debug2: bits set: 524/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: check_host_in_hostfile: filename /u/myuser/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug3: check_host_in_hostfile: filename /u/myuser/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug1: Host 'sftp.site.com' is known and matches the RSA host key.
debug1: Found key in /u/myuser/.ssh/known_hosts:1
debug2: bits set: 521/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: cipher_init: aes128-ctr from source OpenSSL
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: cipher_init: aes128-ctr from source OpenSSL
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /u/myuser/.ssh/id_rsa (1753D900)
debug2: key: /u/myuser/.ssh/id_dsa (0)
debug1: Authentications that can continue: password,keyboard-interactive
debug3: start over, passed a different list password,keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug2: input_userauth_info_req
Windows Authentication
debug2: input_userauth_info_req: num_prompts 1
Password:
debug3: packet_send2: adding 32 (len 23 padlen 9 extra_pad 64)
debug1: Authentication succeeded (keyboard-interactive).
debug2: fd 4 setting O_NONBLOCK
debug2: fd 5 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Entering interactive session.
debug2: callback start
debug2: client_session2_setup: id 0
debug1: Sending subsystem: sftp
debug2: channel 0: request subsystem confirm 1
debug2: fd 3 setting TCP_NODELAY
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
[00.773] debug2: Remote version: 3
[00.773] debug2: Init extension: "posix-rename@openssh.com"
[00.773] debug2: Init extension: "statvfs@openssh.com"
[00.773] debug2: Init extension: "fstatvfs@openssh.com"
[00.773] debug3: Sent message fd 6 T:16 I:1
[00.895] debug3: SSH_FXP_REALPATH . -> /C
cozsftp> cd /E/DCOM/Out/Images
[18.606] debug3: Sent message fd 6 T:16 I:2
[18.648] debug3: SSH_FXP_REALPATH /E/DCOM/Out/Images -> /E/DCOM/Out/Images
[18.648] debug3: Sent message fd 6 T:17 I:3
[18.693] debug3: Received stat reply T:105 I:3
cozsftp> pwd
Remote working directory: /E/DCOM/Out/Images
cozsftp> ls
[24.828] debug3: Sent message fd 6 T:7 I:4
[24.873] debug3: Received stat reply T:105 I:4
[24.873] debug3: Sent message fd 6 T:7 I:5
[25.215] debug3: Received stat reply T:101 I:5
[25.215] debug1: Couldn't stat remote file: No such file or directory
cozsftp> cd ..
[34.994] debug3: Sent message fd 6 T:16 I:6
[35.043] debug3: SSH_FXP_REALPATH /E/DCOM/Out/Images/.. -> /E/DCOM/Out
[35.043] debug3: Sent message fd 6 T:17 I:7
[35.086] debug3: Received stat reply T:105 I:7
cozsftp> pwd
Remote working directory: /E/DCOM/Out
cozsftp> ls
[41.680] debug3: Sent message fd 6 T:7 I:8
[41.726] debug3: Received stat reply T:105 I:8
[41.726] debug3: Sent message fd 6 T:7 I:9
[41.775] debug3: Received stat reply T:101 I:9
[41.775] debug1: Couldn't stat remote file: No such file or directory
cozsftp>

[dovetail]

From the trace, I can see that the Windows server is responding to a "STAT" request with an error.
It looks like after the ls, there is a STAT that works, but the second one fails.

I suspect that it is some permission kind of problem with the Windows SFTP server or filesystem. The filesystem appears to be a removable memory card or something.

The server log on the Windows system will hopefully have more information.

[DClassic53]

So far we are not having any luck when using cozsftp to connect to one of our partners. Their response to the trace (and your comments) elicited the following response:

"The SFTP vendor has not provided any options for the Dillard's Issue we have. Please read this article which describes the elevated SFTP actions Dillards is trying to perform. As discussed, Dillard's and Coach manually tested SFTP using other SFTP clients (PSFTP and Filezilla). Can you please request the vendor to bypass the STAT command from their client configuration?

http://www.thegeekstuff.com/2009/07/uni ... ttributes/

Also can you please get me the below info when you get a chance...

- What type of UNIX they are using (Linux / BSD / Solaris / other) ?
- Can we get info on the SFTP client used and is it possible to download it for test?"

I'm not sure that I know how to interpret their response ...

[dovetail]

The thegeekstuff.com link seems to refer to the Unix "stat" command.

The "stat" command (packet) being sent by the cozsftp "ls" command is "SSH_FXP_STAT", which is part of the SFTP standard.
http://tools.ietf.org/html/draft-ietf-secsh-filexfer-03

It sounds to me like their proposed solution is not to send "SSH_FXP_STAT" to their server. Curious.

I would suggest that you test with another OpenSSH sftp client (Linux, Unix, even Windows via cygwin). You should be able to reproduce this problem, since all of these will send SSH_FXP_STAT packets.... it is part of the standard.

Some sftp clients will send an "SSH_FXP_OPENDIR" packet for a "ls" or "dir" command. PSFTP is one of them. Perhaps their server only implements part of the standard. OpenSSH (and many other implementations) will first send a "SSH_FXP_STAT" packet for the current directory, to verify that it is a directory. If it is, then it will send "SSH_FXP_OPENDIR".

I have NEVER heard of a SFTP implementation that didn't work with OpenSSH. The cozsftp client is based on OpenSSH, and it isn't sending anything to this server that any OpenSSH client wouldn't send.

[DClassic53]

The remote server is: Remote protocol version 2.0, remote software version OpenSSH_5.3 in RemotelyAnywhere 11.0.2673

From a Ubuntu workstation here:

sftp 'coach.ext\lmorgan'@sftp.xxxxx.com
Windows Authentication
Password:
Connected to sftp.xxxxx.com.
sftp> ls -ltr
drwxrwxrwx 1 root system 0 Apr 26 2012 Acxiom
drwxrwxrwx 1 root system 0 May 18 2012 AAFES

Everything lists out and the permissions are full blown

I then used Putty to access my host here and access the destination sftp server using:

from sftp from Ported tools on z/OS:
$ sftp 'coach.ext\lmorgan'@sftp.xxxxx.com
Connecting to sftp.xxxxx.com...
Windows Authentication
Password:
sftp> ls
sftp> exit

from cozsftp from the Co:Z Toolkit on z/OS
$ cozsftp 'coach.ext\lmorgan'@sftp.xxxxx.com
Co:Z SFTP version: 2.1.1 (5.0p1) 2012-03-16
Copyright (C) Dovetailed Technologies, LLC. 2011. All rights reserved.
Connecting to sftp.xxxxx.com...
Windows Authentication
Password:
cozsftp> ls
cozsftp> quit

I never get a response to the ls command when coming from my z/OS system.

[dovetail]

Some suggestions;

1) Please post "-vvv" traces for both the working Ubuntu and failing cozsftp client.

2) In the earlier trace that showed the server rejecting the "stat" commands, investigate the cause of the error on the server. Nothing in the earlier trace indicated an error other than the server sending an error back for the stat commands. (SSH_FXP_STAT). Perhaps seeing traces of both the working Ubuntu client and the failing cozsftp client will explain.

3) test the same connection, with a -vvv trace from the same z/OS system, but use the IBM Ported Tools sftp client and post this. This command is "sftp" instead of "cozsftp". You won't be able to do any dataset transfers, but you will be able to login and do remote "ls", "cd", etc. I am suggesting this because we have seen some issues with servers that have security rules based on what the client IP is.