We are getting this error about once every 100 or so executions of PGM=COZLNCH. Any insight would be appreciated? fyi, we have controls in place to keep only one coz launcher job running at one time.
1[18:36:00.726465] CoZLauncher[N]: version: 2.4.1 2013-06-24
[18:36:00.729635] CoZLauncher[N]: Copyright (C) Dovetailed Technologies, LLC. 2006-2013. All rights reserved.
[18:36:00.729645] CoZLauncher[D]: target_user="xx12345", target_host="server1", target_port="NULL"
[18:36:00.729667] CoZLauncher[D]: sysname=S001, user=xx12345, uid=000000000, euid=000000000, pid=393518, ppid=1
[18:36:00.729672] CoZLauncher[D]: region size requested = 32768K, Actual below/above limit = 9192K / 1083392K
[14:36:00.743103] CoZLauncher: Agent output WTO is OFF
[14:36:00.743117] CoZLauncher: ssh tunnelling is ON
[14:36:00.743121] CoZLauncher[D]: server-path=/usr/local/coz/bin/cozserver
[14:36:00.743124] CoZLauncher[D]: server-ports=8040-8048
[14:36:00.743127] CoZLauncher[D]: ssh-options=-oStrictHostKeyChecking=no
[14:36:00.743130] CoZLauncher[D]: server-env-PASSWD_DSN=//OP.CARD(ACARD01)
[14:36:00.743133] CoZLauncher[D]: server-env-SSH_ASKPASS=/usr/local/coz/bin/read_passwd_dsn.sh
[14:36:00.743136] CoZLauncher[D]: server-env-DISPLAY=none
[14:36:00.743138] CoZLauncher[D]: ssh-path=/bin/ssh
[14:36:00.743141] CoZLauncher[D]: ssh-tunnel=true
[14:36:00.743143] CoZLauncher[D]: ssh-shareas=YES
[14:36:00.743146] CoZLauncher[D]: agent-path=/opt/dovetail/coz/bin/cozagent
[14:36:00.743152] CoZLauncher[D]: agent-output-wto=false
[14:36:00.743154] CoZLauncher[D]: server-env-COZ_TRSUB_US-ASCII=ISO8859-1
[14:36:00.743304] CoZLauncher[D]: Set server environment variable:'PASSWD_DSN'='//OP.CARD(ACARD01)'
[14:36:00.743309] CoZLauncher[D]: Set server environment variable:'SSH_ASKPASS'='/usr/local/coz/bin/read_passwd_dsn.sh'
[14:36:00.743313] CoZLauncher[D]: Set server environment variable:'DISPLAY'='none'
[14:36:00.743317] CoZLauncher[D]: Set server environment variable:'COZ_TRSUB_US-ASCII'='ISO8859-1'
[14:36:00.743321] CoZLauncher[D]: COZ_RECV_MSG_WAITALL was not found, setting to "N"
[14:36:00.743514] CoZLauncher[D]: Server port range: 8040-8048
[14:36:00.743521] CoZLauncher[D]: Attempting to start socket listener on port 8040
[14:36:00.768158] CoZLauncher: CoZServer listener socket bound to: 127.0.0.1:8040
[14:36:00.768294] CoZLauncher[D]: server_cmd...
/usr/local/coz/bin/cozserver -sockfd 3
[14:36:00.772182] CoZLauncher[D]: Started CoZServer process: 17171274
[14:36:00.778455] CoZLauncher[D]: arg[3] = '-oStrictHostKeyChecking=no'
[14:36:00.778470] CoZLauncher[D]: agent_cmd...
/bin/ssh -l zubat01 -oStrictHostKeyChecking=no -R 8040:127.0.0.1:8040 server1 /opt/dovetail/coz/bin/cozagent
[14:36:00.780466] CoZLauncher[D]: Started CoZAgent process: 394073
[14:36:00.780527] CoZLauncher[D]: About to start target command: ''
[14:36:00.883813] CoZLauncher[D]: First agent message received:
=== Help ensure information systems are used solely for authorized purposes.
===
=== This is a business information system for authorized use only.
=== Unauthorized use and other violations of company policy may result in
=== corrective action, up to and including immediate termination, and/or
=== criminal prosecution. Use of this system and all files on it may be
=== monitored, copied, and disclosed at any time
===
=== By using this system you agree to the above terms and conditions.
===
fromdsn(OP.CARD(ACARD01))[N]: 1 records/80 bytes read; 9 bytes written in 0 milliseconds.
FOTS2266 Warning: remote port forwarding failed for listen port 8040
cozagent[N]: version: 1.0.11 2010-10-27
[14:36:01.606561] CoZLauncher[D]: CoZAgent: completed with RC=103
todsn-client(11820)[E]: server exit_code=106
fromdsn-client(11819)[E]: server exit_code=106
todsn-client(11820)[E]: error: CoZServer terminated before all input was read
todsn-client(11822)[E]: server exit_code=106
todsn-client(11822)[E]: error: CoZServer terminated before all input was read
cozagent[E]: Stdin DD Reader(11819) ended with RC=106
cozagent[E]: Stdout DD Writer(11820) ended with RC=103
cozagent[E]: Stderr DD Writer(11822) ended with RC=103
[14:36:01.606977] CoZLauncher[D]: SSH process times: elapsed=0 secs, user cpu=0.030000 secs, sys cpu=0.010000 secs
[14:36:01.606989] CoZLauncher[D]: Killing child process CoZServer (17171274) with signal 15
[14:36:01.607787] CoZLauncher[E]: CoZAgent process (394073) ended with RC=103
[14:36:01.607806] CoZLauncher[E]: xx12345@server1 target command '<default shell>' ended with RC=103
[14:36:01.607811] CoZLauncher[D]: Killing child process CoZServer (17171274) with signal 15
[14:36:01.607829] CoZLauncher[D]: Waiting for CoZServer
[14:36:01.608658] CoZLauncher[D]: serverStderrThread completed
[14:36:01.608964] CoZLauncher[D]: CoZServer process (17171274) ended with RC=0
[14:36:01.610327] CoZLauncher[D]: agentStdoutThread completed
[14:36:01.610610] CoZLauncher[D]: agentStderrThread completed
[14:36:01.611284] CoZLauncher[E]: CoZLauncher ended with RC=103
1CoZAgent: xx12345@server1 target program '/bin/bash' PID: 11818
CoZAgent: completed with RC=103
CozLauncher port forwarding failed
Re: CozLauncher port forwarding failed
this looks to us to be a case where two jobs followed in quick succession. The second should pick a different port, but we have discovered a timing issue that causes it to pick the same port even though that port is still in use by SSHD on the target system.
We are developing a fix for this -- would you be willing to test a pre-release version (this would eventually be version 2.4.5) ?
We are developing a fix for this -- would you be willing to test a pre-release version (this would eventually be version 2.4.5) ?
Re: CozLauncher port forwarding failed
Has this timing issue been resolved? We just encountered an issue with the same error message. I have requested a rerun to see if it works. What can be done to allow multiple jobs to work from z/OS to a single Linux server, using Co:Z Launcher?
Re: CozLauncher port forwarding failed
Are you sure that you are not using the same ports on multiple LPARs going to the same server?
Please post your COZCFG/D parameter settings.
Please post your COZCFG/D parameter settings.
Re: CozLauncher port forwarding failed
###############################################################################
# Co:Z Installation specific settings
###############################################################################
# Note: The convention in this file is to show defaults for properties
# as commented-out #property-name=value settings.
# The path on the server of the CozServer executable. If this executable is
# not available by symbolic link at /bin as recommended by the z/OS installer,
# change this property to be the absolute path of the cozserver executable,
# which is typically <COZINST>/bin/cozserver
# The following record was customized for the AT&T environment. ATT
server-path=/usr/local/bin/cozserver
# The range of ports from which CoZServer selects to bind and communicate
# with the target program.
# If ssh-tunnel=true, the target program will connect to the selected
# port on the loopback adapter. Otherwise the target program will connect
# to CoZServer directly at this port.
server-ports=8040-8048
###############################################################################
# Additional settings
###############################################################################
# If supplied, specifies the location of the z/OS ssh client executable.
#ssh-path=/bin/ssh
# Additional ssh options to be supplied to ssh.
#ssh-options=
# If true, target program IO requests (via fromdsn and todsn) are tunnelled
# over ssh via reverse port forwarding. If false, direct socket connects are
# made to the server.
#ssh-tunnel=true
# By default, the ssh client runs in a separate address space from the
# launcher. Versions earlier than IBM's 1.2 Ported Tools may allow ssh to run
# in the same address space if a value of MUST is specified.
#ssh-shareas=YES
# Custom Language Environment (LE) options to set for the ssh client process
# The following options work around a problem that causes out-of-memory
# conditions in Ported Tools OpenSSH. See IBM APAR OA34819.
ssh-le-options=HEAP(8M,1M,,FREE),ENVAR("_CEE_REALLOC_CONTROL=256K,25")
# It is possible to authenticate the ssh client with a RACF Digital Certificate
# (either RSA or DSA) instead of the traditional OpenSSH keypairs. In order
# to use this option, an OpenSSH version of the Digital Certificate public key
# must be present in the target-system's authorized_keys file. See the user's
# guide for details.
# To enable this form of authentication, set the property below is set with
# the RACF key ring and (optional) label:
#saf-cert=KEYRING
#saf-cert=KEYRING:LABEL
# The executable path on the target of the CozAgent executable.
# The client installation process places the Co:Z executables at
# /opt/dovetail/coz/bin by default.
#agent-path=/opt/dovetail/coz/bin/cozagent
# Command line options to CoZAgent
#agent-options=
# If true, messages written by the CoZAgent are written to the operator
# console. If false, they are written to the launcher's stdout(DD://SYSPRINT)
#agent-output-wto=false
# The external IP address of the CoZServer running on z/OS.
# If ssh-tunnel=false, the target program will connect
# to this address. If ssh-tunnel=true, this value is ignored.
# If this property is not supplied, it will default to the result of
# gethostname().
#server-host=
# If supplied, the server will accept connections on this address.
# If not supplied, the default is 0.0.0.0, which is all addresses.
# If ssh-tunnel=true, this value is ignored.
#server-ip-stack=0.0.0.0
# Customized server environment variables that will be set prior to launching
# the CoZServer. These environment variables will also be adopted by the
# Launcher itself.
#server-env-MY_VAR=my_value
server-env-COZ_TRSUB_US-ASCII=ISO8859-1
# Co:Z Installation specific settings
###############################################################################
# Note: The convention in this file is to show defaults for properties
# as commented-out #property-name=value settings.
# The path on the server of the CozServer executable. If this executable is
# not available by symbolic link at /bin as recommended by the z/OS installer,
# change this property to be the absolute path of the cozserver executable,
# which is typically <COZINST>/bin/cozserver
# The following record was customized for the AT&T environment. ATT
server-path=/usr/local/bin/cozserver
# The range of ports from which CoZServer selects to bind and communicate
# with the target program.
# If ssh-tunnel=true, the target program will connect to the selected
# port on the loopback adapter. Otherwise the target program will connect
# to CoZServer directly at this port.
server-ports=8040-8048
###############################################################################
# Additional settings
###############################################################################
# If supplied, specifies the location of the z/OS ssh client executable.
#ssh-path=/bin/ssh
# Additional ssh options to be supplied to ssh.
#ssh-options=
# If true, target program IO requests (via fromdsn and todsn) are tunnelled
# over ssh via reverse port forwarding. If false, direct socket connects are
# made to the server.
#ssh-tunnel=true
# By default, the ssh client runs in a separate address space from the
# launcher. Versions earlier than IBM's 1.2 Ported Tools may allow ssh to run
# in the same address space if a value of MUST is specified.
#ssh-shareas=YES
# Custom Language Environment (LE) options to set for the ssh client process
# The following options work around a problem that causes out-of-memory
# conditions in Ported Tools OpenSSH. See IBM APAR OA34819.
ssh-le-options=HEAP(8M,1M,,FREE),ENVAR("_CEE_REALLOC_CONTROL=256K,25")
# It is possible to authenticate the ssh client with a RACF Digital Certificate
# (either RSA or DSA) instead of the traditional OpenSSH keypairs. In order
# to use this option, an OpenSSH version of the Digital Certificate public key
# must be present in the target-system's authorized_keys file. See the user's
# guide for details.
# To enable this form of authentication, set the property below is set with
# the RACF key ring and (optional) label:
#saf-cert=KEYRING
#saf-cert=KEYRING:LABEL
# The executable path on the target of the CozAgent executable.
# The client installation process places the Co:Z executables at
# /opt/dovetail/coz/bin by default.
#agent-path=/opt/dovetail/coz/bin/cozagent
# Command line options to CoZAgent
#agent-options=
# If true, messages written by the CoZAgent are written to the operator
# console. If false, they are written to the launcher's stdout(DD://SYSPRINT)
#agent-output-wto=false
# The external IP address of the CoZServer running on z/OS.
# If ssh-tunnel=false, the target program will connect
# to this address. If ssh-tunnel=true, this value is ignored.
# If this property is not supplied, it will default to the result of
# gethostname().
#server-host=
# If supplied, the server will accept connections on this address.
# If not supplied, the default is 0.0.0.0, which is all addresses.
# If ssh-tunnel=true, this value is ignored.
#server-ip-stack=0.0.0.0
# Customized server environment variables that will be set prior to launching
# the CoZServer. These environment variables will also be adopted by the
# Launcher itself.
#server-env-MY_VAR=my_value
server-env-COZ_TRSUB_US-ASCII=ISO8859-1
Re: CozLauncher port forwarding failed
If you use this:
server-ports=8040-8048
across several LPARs, you will have port collisions that cause this error.
This is because, when using ssh-tunnel=true, here is what happens:
1) The Co:Z Launcher job finds an unused port in the range, say 8048
2) It starts a socket listener on 127.0.0.1:8048
3) it starts an ssh session to the target server, with: -R 127.0.0.1:8048:127.0.0.1:8048.
this sets up a local listener on the target server on port 8048 that connects back into z/OS to the listener at 127.0.0.1:8048.
This falls apart when you use the same port range, since multiple jobs on different LPARs could start a job to the same server using the same port.
If this is your situation, you need to:
a) reserve a port range on the target server, for use by ALL z/OS client LPARs, and
b) partition this range so that each z/OS LPAR only uses the part of the range reserved for that LPAR.
As shown in: http://dovetail.com/docs/coz/config.htm ... g_launcher
you can do something like this and share the same COZCFGD dataset accross LPARs:
server-ports-SYSA=8040-8059
server-ports-SYSB=8060-8079
server-ports-SYSC=8080-8099
...
(SYSX is the SYSNAME of the z/OS LPAR)
server-ports=8040-8048
across several LPARs, you will have port collisions that cause this error.
This is because, when using ssh-tunnel=true, here is what happens:
1) The Co:Z Launcher job finds an unused port in the range, say 8048
2) It starts a socket listener on 127.0.0.1:8048
3) it starts an ssh session to the target server, with: -R 127.0.0.1:8048:127.0.0.1:8048.
this sets up a local listener on the target server on port 8048 that connects back into z/OS to the listener at 127.0.0.1:8048.
This falls apart when you use the same port range, since multiple jobs on different LPARs could start a job to the same server using the same port.
If this is your situation, you need to:
a) reserve a port range on the target server, for use by ALL z/OS client LPARs, and
b) partition this range so that each z/OS LPAR only uses the part of the range reserved for that LPAR.
As shown in: http://dovetail.com/docs/coz/config.htm ... g_launcher
you can do something like this and share the same COZCFGD dataset accross LPARs:
server-ports-SYSA=8040-8059
server-ports-SYSB=8060-8079
server-ports-SYSC=8080-8099
...
(SYSX is the SYSNAME of the z/OS LPAR)