CozBatch and FIREWALL Issue

[mac.mcmullin]

We are having random errors on a connection from mainframe to our firewall to vendor sftp server. It appears we send a packet then do not get a TCP ACK returned and then time out. The exact same job is reran and completes successfully. We use PaloAlto firewall. We are at:OpenSSH_6.4, OpenSSL 1.0.1c 10 May 2012
debug1: Reading configuration data /etc/ssh/ssh_config
Are there any firewall friendly entries in ssh_config or Coz that could fix this intermittent issue? I have reviewed with our Firewall support and vendor but other then -vvv I cannot find any tracing that incorporates end to end TCP packet tracing or denial of service. I sanitized the trace site names below. I took z/os TCP traces, but nothing indicative of an issue on mainframe TCPIP.

Enter Password for bu15.corpedef.xfrp:
debug2: input_userauth_info_req: num_prompts 1
debug1: read_passphrase: can't open /dev/tty: EDC5128I No such device. (errno2=0x056201A9)
debug1: permanently_drop_suid: 0
/products/coz/run/bin/read_passwd_dsn.sh prompt: "Password:"
fromdsn(LM.PARMLIB(CS115S07))[N]: 1 records/80 bytes read; 37 bytes written in 0.007 seconds (5.162 KBytes/sec).
debug3: packet_send2: adding 8 (len 50 padlen 6 extra_pad 64)
debug3: Received SSH2_MSG_IGNORE
debug3: Received SSH2_MSG_IGNORE
debug3: Received SSH2_MSG_IGNORE
debug3: Received SSH2_MSG_IGNORE
debug3: Received SSH2_MSG_IGNORE
Received disconnect from 192.131.64.177: 11: Idle connection
[43.832] Connection closed

Same job that completes successfully:
Enter Password for bu15.dukedef.xfrp:
debug2: input_userauth_info_req: num_prompts 1
debug1: read_passphrase: can't open /dev/tty: EDC5128I No such device. (errno2=0x056201A9)
debug1: permanently_drop_suid: 0
/products/coz/run/bin/read_passwd_dsn.sh prompt: "Password:"
fromdsn(LM.PARMLIB(CS115S07))[N]: 1 records/80 bytes read; 37 bytes written in 0 milliseconds.
debug3: packet_send2: adding 8 (len 50 padlen 6 extra_pad 64)
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to name.com ([xxx.xxx.xx.xxx]:22).
debug3: options.client_smf = none

My options are:
22.707] debug3: connect_to_server arg=/bin/ssh
[22.707] debug3: connect_to_server arg=-oForwardX11 no
[22.707] debug3: connect_to_server arg=-oForwardAgent no
[22.707] debug3: connect_to_server arg=-oClearAllForwardings yes
[22.707] debug3: connect_to_server arg=-oBatchMode=no
[22.707] debug3: connect_to_server arg=-oConnectTimeout=2400
[22.707] debug3: connect_to_server arg=-oServerAliveInterval=2400
[22.707] debug3: connect_to_server arg=-oStrictHostKeyChecking=no
[22.707] debug3: connect_to_server arg=-v
[22.707] debug3: connect_to_server arg=-v
[22.707] debug3: connect_to_server arg=-v
[22.707] debug3: connect_to_server arg=-obatchmode yes
[22.707] debug3: connect_to_server arg=-lbu15.xxxxef.xfrp
[22.707] debug3: connect_to_server arg=-oProtocol 2
[22.707] debug3: connect_to_server arg=-s
[22.707] debug3: connect_to_server arg=xxx.com
[22.707] debug3: connect_to_server arg=sftp

JCL has:
HOME=/var/mss
echo $HOME
export TMPDIR=/var/tmpdir1
echo tmpdir is $TMPDIR
coz_bin="/products/coz/run/bin"
ruser="bu15.dukedef.xfrp"
server="datadelivery-mft.onefiserv.com"
servercp="ISO8859-1"
loglevel=I
#
export PASSWD_DSN='//LM.PARMLIB(CS115S07)'
export SSH_ASKPASS=$coz_bin/read_passwd_dsn.sh
export DISPLAY=all

ssh_opts="-oBatchMode=no" # ALLOWS SSH TO USE SSH_ASKPASS PROGRAM
ssh_opts="$ssh_opts -oConnectTimeout=2400"
ssh_opts="$ssh_opts -oServerAliveInterval=2400"
ssh_opts="$ssh_opts -oStrictHostKeyChecking=no" # accept initial key
#
$coz_bin/cozsftp $ssh_opts -vvv -b- $ruser@$server <<EOB
lzopts mode=text,servercp=$servercp
cd Outbox
pwd
ls
EOB

[dovetail]

SSH doesn't have FW_FRIENDLY settings (or issues) like FTP, since it only uses a single socket connection.
Most firewall have no issues; you just need to make sure that this traffic (TCP socket, dest port 22) is allowed.

I assume that you have done a Comm Server packet trace to see that TCP ACKs are not being returned? If so, then you probably do have some kind of network problem, although it could be anywhere in the path.

Note: Co:Z SFTP uses IBM Ported Tools OpenSSH (or z/OS 2.2 OpenSSH) for it underlying ssh connection. If you think that you have a z/OS network issue, you might open an problem with either Comm Server or z/OS OpenSSH.

[mac.mcmullin]

I opened an SR with IBM, they said:
Greetings Mac,
Looking at this from the OpenSSH client (running on zOS) point of view, we see that the client is receiving a disconnect from the remote
server:
Received disconnect from 192.131.xxx.xxx: 11: Idle connection
It is not readily apparent as to why from the traces.
I did not see a server trace from the same time period as the client trace. Is it possible to get a server side trace that we can use to compare to the entries in the client trace?

Another thing I see is that it looks like your C:OS SFTP version is a little downlevel:
Co:Z SFTP version: 3.0.0 (5.0p1)
There is a more recent version of Co:Z SFTP (4.1) available. The 3.0.0 release of Co:Z SFTP is using OpenSSH version 5.0p1. The OpenSSH level that your zOS OpenSSH is running is OpenSSH_6.4 (which is more recent that the level Co:Z is running).
You may want to consider bringing Co:Z up to the same OpenSSH 6.4 release level (this may be provided in Co:Z SFTP 4.1).

You asked about firewall support in zOS OpenSSH. OpenSSH does not have a 'firewall friendly' mode, but does support port forwarding. If you suspect a firewall issue, you may want to check with your firewall administrator to see if port forwarding is needed on the client side.

I have since downloaded and setup Co:Z FTP version 4.1, we are z/OS 2.1 and planning z/OS 2.2 in 2017. We are at Co:Z SFTP version: 3.0.0 (5.0p1). We are using DNS versus hard coded IP address, but seems like we successfully connect, but then somewhere during the connection, we miss a TCP service packet and timeout. I will try to coordinate packet traces witht eh vendor, but is sporadic issue and when same job reran is successful.

[dovetail]

I don't think that the version of Co:Z SFTP has anything whatsoever to do with this problem.

The cozsftp command starts an IBM ssh command as a child process to do the secure SSH connection. The ssh connection is failing / disconnecting.