I am trying to run the following:
//STEPLIB JCLLIB ORDER=('DVCC.COZ.SAMPJCL')
//RUNCOZK EXEC PROC=COZPROC,
// ARGS='adsm@dledm.itg.ti.com'
//COZCFG DD *
ssh-tunnel=false
ssh-options=-vvv
agent-path=/apps/adsm/coz/bin/cozagent.sh
agent-options=-LD,t
target-env-COZ_LOG=D,t
//STDIN DD *
/apps/adsm/coz/bin/fromdsn -b -l rdw //DD:INPUT \
| gzip > /apps/adsm/coz/DRP.ABARS.DMGTEST.C.C01V0015.gz
/*
//INPUT DD DSN=DRP.ABARS.DMGTEST.C.C01V0015,DISP=SHR
There are a few things happening that I am trying to figure out how to get around. First, it can't find fromdsn unless I have it fully qualified as above. Second, the default shell is /bin/csh. How do I change it to /bin/sh for interpreting the STDIN commands? Third, gzip cannot seem to find the piped input file. Here are the error messages:
11:48:10.439496ì todsn-client(6105)Dì: clientCodePage 'ISO8859-1' (via COZ_CLIENT_CODEPAGE)
11:48:10.439521ì todsn-client(6106)Dì: clientCodePage 'ISO8859-1' (via COZ_CLIENT_CODEPAGE)
11:48:10.493522ì fromdsn-client(6104)Dì: fromdsn(DD:STDIN): , rc=0, msg=""
11:48:10.497998ì fromdsn-client(6104)Dì: server exit_code=0
11:48:10.500319ì fromdsn-client(6104)Dì: transmitted byte counts agree; server sent/received = 109/0, client
received/sent = 109/0
debug2: channel 0: written 460 to efd 6
fromdsn(GZIP)Eì: DatasetHandler: Error in fopen(//'GZIP', rb,type=record,noseek,recfm=*) - EDC5049I The specified file
name could not be located. (errno2=0xC00B0641)
todsn(DD:STDOUT)Nì: 1309 bytes read; 24 records/1289 bytes written in 0.002 seconds (639.160 KBytes/sec).
todsn(DD:STDERR)Nì: 1133 bytes read; 15 records/1118 bytes written in 0.002 seconds (553.223 KBytes/sec).
debug2: channel 0: rcvd ext data 213
debug2: channel 0: rcvd ext data 74
debug2: channel 0: rcvd ext data 60
debug2: channel 0: rcvd ext data 60
debug2: channel 0: rcvd ext data 133
debug2: channel 0: rcvd ext data 133
debug2: channel 0: rcvd ext data 91
debug2: channel 0: rcvd ext data 140
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
11:48:10.746894ì cozagentEì: Target Program(6103) ended with RC=103
11:48:10.746946ì todsn-client(6105)Dì: todsn(DD:STDOUT): , rc=0, msg=""
11:48:10.746990ì cozagentDì: STDIN DD Reader(6104) ended with RC=0
11:48:10.747219ì todsn-client(6106)Dì: todsn(DD:STDERR): , rc=0, msg=""
11:48:10.748444ì todsn-client(6106)Dì: server exit_code=0
11:48:10.757252ì todsn-client(6105)Dì: server exit_code=0
11:48:10.758491ì todsn-client(6106)Dì: transmitted byte counts agree; server sent/received = 0/1133, client
received/sent = 0/1133
11:48:10.758615ì todsn-client(6105)Dì: transmitted byte counts agree; server sent/received = 0/1309, client
received/sent = 0/1309
11:48:10.953118ì cozagentDì: At least one child I/O process still running, waiting again
11:48:10.953229ì cozagentDì: STDOUT DD Writer(6105) ended with RC=0
11:48:10.953252ì cozagentDì: STDERR DD Writer(6106) ended with RC=0
debug2: channel 0: written 904 to efd 6
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd close
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug3: channel 0: will not send data after close
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cfd -1)
debug3: channel 0: close_fds r -1 w -1 e 6 c -1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: fd 2 clearing O_NONBLOCK
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.6 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 103
CoZLauncherEì: adsm@dledm.itg.ti.com target command '<default shell>' ended with RC=103
oZAgent: adsm@dledm target program '/bin/csh' PID: 6103
oZAgent: completed with RC=103
11:48:10.416373ì cozagentDì: About to execute shell -/bin/csh
tty: : Invalid argument
tty: : No such device or address
11:48:10.660097ì fromdsn-client(6132)Iì: version: 1.1.0 2012-03-16
11:48:10.660426ì fromdsn-client(6132)Dì: cmd_arg1ì = -b
11:48:10.660491ì fromdsn-client(6132)Dì: cmd_arg2ì = -l
11:48:10.660502ì fromdsn-client(6132)Dì: cmd_arg3ì = rdw
11:48:10.660511ì fromdsn-client(6132)Dì: cmd_arg4ì = //DD:INPUT
11:48:10.660518ì fromdsn-client(6132)Dì: cmd_arg5ì = |
11:48:10.660528ì fromdsn-client(6132)Dì: cmd_arg6ì = gzip
11:48:10.660539ì fromdsn-client(6132)Dì: Using getaddrinfo() to start connection to server SDV01 at port 8040
11:48:10.668083ì fromdsn-client(6132)Dì: clientCodePage 'ISO8859-1' (via COZ_CLIENT_CODEPAGE)
11:48:10.737013ì fromdsn-client(6132)Eì: fromdsn(GZIP): Error opening input dataset, rc=103, msg="EDC5049I The specified file name
11:48:10.739269ì fromdsn-client(6132)Eì: server exit_code=103
11:48:10.742799ì fromdsn-client(6132)Dì: transmitted byte counts agree; server sent/received = 0/0, client received/sent = 0/0
arning: no access to tty; thus no job control in this shell...
Pipe to gzip not working
Re: Pipe to gzip not working
| First, it can't find fromdsn unless I have it fully qualified as above
On the target system, cozagent will set PATH to include the Co:Z Target bin directory before starting your login shell. Apparently your login shell is setting PATH from scratch rather than adding to the current setting. Check the profile scripts for your shell (csh) to correct this
| Second, the default shell is /bin/csh. How do I change it to /bin/sh for interpreting the STDIN commands?
Do you want csh as your default login shell? If not, you would have to change your userid to have a different default shell.
If you want a different shell just for running the Co:Z Launcher, you can specify the program/shell name using the option:
target-command /bin/bash -l
The above example will run bash with the -l (login shell) switch. When bash is run as a login shell, it will execute the following profile scripts (if found) -
/etc/profile
~/.bash_profile
~/.bash_login
~/.profile
see the bash man page under "INVOCATION" for more information.
| Third, gzip cannot seem to find the piped input file. Here are the error messages:
I looks like your shell does not recognize the pipe (|) character.
From the trace, you can see that the fromdsn command is seeing arguments: "-b -l rdw //DD:INPUT | gzip" -- which is not right if your shell interpreted the | correctly.
since "gzip" is the last argument, it is being taken as the data set name.
I'm not sure why this is happening.... maybe you should get the right shell running first and then see what happens.
On the target system, cozagent will set PATH to include the Co:Z Target bin directory before starting your login shell. Apparently your login shell is setting PATH from scratch rather than adding to the current setting. Check the profile scripts for your shell (csh) to correct this
| Second, the default shell is /bin/csh. How do I change it to /bin/sh for interpreting the STDIN commands?
Do you want csh as your default login shell? If not, you would have to change your userid to have a different default shell.
If you want a different shell just for running the Co:Z Launcher, you can specify the program/shell name using the option:
target-command /bin/bash -l
The above example will run bash with the -l (login shell) switch. When bash is run as a login shell, it will execute the following profile scripts (if found) -
/etc/profile
~/.bash_profile
~/.bash_login
~/.profile
see the bash man page under "INVOCATION" for more information.
| Third, gzip cannot seem to find the piped input file. Here are the error messages:
I looks like your shell does not recognize the pipe (|) character.
From the trace, you can see that the fromdsn command is seeing arguments: "-b -l rdw //DD:INPUT | gzip" -- which is not right if your shell interpreted the | correctly.
since "gzip" is the last argument, it is being taken as the data set name.
I'm not sure why this is happening.... maybe you should get the right shell running first and then see what happens.
Re: Pipe to gzip not working
I have already tried your first and second suggestions but I still can't seem to get it to work. I will keep working on it.
thanks,
Sandy
thanks,
Sandy
Re: Pipe to gzip not working
Here's some more information that I hope will help -
(First question) Try putting an "env" command at the beginning of the STDIN script. This will allow you to see the environment variables that have been set up by your shell.
Again, cozagent will set PATH to include the Co:Z target toolkit bin directory (the same directory that includes the cozagent command). If your login shell wipes out the PATH rather than adding to it then you will need to configure your login script(s) to set it.
(Second question) you are unable to use the target-command setting to start a different shell? What trace output do you see?
(First question) Try putting an "env" command at the beginning of the STDIN script. This will allow you to see the environment variables that have been set up by your shell.
Again, cozagent will set PATH to include the Co:Z target toolkit bin directory (the same directory that includes the cozagent command). If your login shell wipes out the PATH rather than adding to it then you will need to configure your login script(s) to set it.
(Second question) you are unable to use the target-command setting to start a different shell? What trace output do you see?
Re: Pipe to gzip not working
Thanks for your help on this. I have this part working now, including the shell, the path, and the gzip. Now I will do some benchmark timings on this portion before proceeding to the Co:Z SFTP step. Will I be able to pipe the gz file directly into SFTP, or will I need to create the intermediate gz file on the unix box first?
thanks,
Sandy
thanks,
Sandy
Re: Pipe to gzip not working
I'm not clear on what you are trying to accomplish.
If you want to gzip a dataset and then sftp it, then perhaps consider doing both the gzip and sftp transfer on the gateway *nix box?
You can use the curl command as a convenient front-end to sftp for piping data.
There is an article in zJournal describing this kind of thing:
http://enterprisesystemsmedia.com/artic ... ge-gateway
So, for example:
You will need to install cURL (with sftp enabled support) on your gateway Unix box, and will need a user key on the gateway box for logging into the target server. There are also techniques that you could use to pull the key or password for the target server from a z/OS file or dataset.
If you want to gzip a dataset and then sftp it, then perhaps consider doing both the gzip and sftp transfer on the gateway *nix box?
You can use the curl command as a convenient front-end to sftp for piping data.
There is an article in zJournal describing this kind of thing:
http://enterprisesystemsmedia.com/artic ... ge-gateway
So, for example:
Code: Select all
//STEPLIB JCLLIB ORDER=('DVCC.COZ.SAMPJCL')
//RUNCOZK EXEC PROC=COZPROC,
// ARGS='adsm@dledm.itg.ti.com'
//COZCFG DD *
ssh-tunnel=false
#ssh-options=-vvv
agent-path=/apps/adsm/coz/bin/cozagent.sh
#agent-options=-LD,t
#target-env-COZ_LOG=D,t
//INPUT DD DSN=DRP.ABARS.DMGTEST.C.C01V0015,DISP=SHR
//STDIN DD *
set -o pipefail
fromdsn -b -l rdw //DD:INPUT |
gzip |
curl -T- sftp://target.server.com/targetdir/targetfile.gz
//Re: Pipe to gzip not working
Thank you for the article and the example using curl. Very interesting reading.
I was able to do a comparison of sending a file using cozbatch versus our existing process of using xmit and ftp. I was surprised to find that the cozbatch processing took longer (15.92 cpu secs versus 10.29 cpu secs). Can you shed some light on why this is happening? I was expecting cozbatch to be less cpu.
This was my cozbatch job:
//STEPLIB JCLLIB ORDER=('DVCC.COZ.SAMPJCL')
//RUNCOZK EXEC PROC=COZPROC,
// ARGS='adsm@dledm.itg.ti.com'
//COZCFG DD *
ssh-tunnel=false
ssh-options=-vvv
agent-path=/apps/adsm/coz/bin/cozagent.sh
agent-options=-LD,t
target-env-COZ_LOG=D,t
target-command = /bin/sh
//STDIN DD *
env
dsn=DRP.ABARS.DMGTEST.D.C01V0015
remotedir="/apps/adsm/coz"
fromdsn -b -l rdw //$dsn ! gzip > $remotedir/$dsn.gz
/*
I was able to do a comparison of sending a file using cozbatch versus our existing process of using xmit and ftp. I was surprised to find that the cozbatch processing took longer (15.92 cpu secs versus 10.29 cpu secs). Can you shed some light on why this is happening? I was expecting cozbatch to be less cpu.
This was my cozbatch job:
//STEPLIB JCLLIB ORDER=('DVCC.COZ.SAMPJCL')
//RUNCOZK EXEC PROC=COZPROC,
// ARGS='adsm@dledm.itg.ti.com'
//COZCFG DD *
ssh-tunnel=false
ssh-options=-vvv
agent-path=/apps/adsm/coz/bin/cozagent.sh
agent-options=-LD,t
target-env-COZ_LOG=D,t
target-command = /bin/sh
//STDIN DD *
env
dsn=DRP.ABARS.DMGTEST.D.C01V0015
remotedir="/apps/adsm/coz"
fromdsn -b -l rdw //$dsn ! gzip > $remotedir/$dsn.gz
/*
Re: Pipe to gzip not working
(FWIW, you are using Co:Z Launcher, not Co:Z Batch)
It would be difficult to say without a significant amount of analysis.
One major difference is that, although you are running the data transfer without encryption (ssh-tunnel=false), you are still setting up an ssh session. If IBM Ported Tools OpenSSH is not tuned / configured properly it can use alot of CPU, especially to start a session.
It could also be that the CPU difference is I/O related; having to do with the block sizes / record lengths of the data that you are actually transferring. Co:Z currently uses the IBM C library for dataset I/O, which might not be as efficient. You say that you are creating XMITs first with FTP; is the I/O and CPU time required to do that considered?
It would be difficult to say without a significant amount of analysis.
One major difference is that, although you are running the data transfer without encryption (ssh-tunnel=false), you are still setting up an ssh session. If IBM Ported Tools OpenSSH is not tuned / configured properly it can use alot of CPU, especially to start a session.
It could also be that the CPU difference is I/O related; having to do with the block sizes / record lengths of the data that you are actually transferring. Co:Z currently uses the IBM C library for dataset I/O, which might not be as efficient. You say that you are creating XMITs first with FTP; is the I/O and CPU time required to do that considered?
Re: Pipe to gzip not working
Yes, the XMIT step took 7.43 sec, and the FTP step took 2.86 sec.
Also, when I ran the same coz launcher job but with a very small file, it took .12 sec, so not much time is attributed to the ssh connection.
I'm not sure how to proceed from here. What would be involved in further analysis of this?
thanks,
Sandy
Also, when I ran the same coz launcher job but with a very small file, it took .12 sec, so not much time is attributed to the ssh connection.
I'm not sure how to proceed from here. What would be involved in further analysis of this?
thanks,
Sandy
Re: Pipe to gzip not working
The difference is probably due to I/O library differences (C library vs FTP), although I don't think that what you are seeing is typical.
What DCB attributes are used on the files (original dataset, XMIT dataset)?
What DCB attributes are used on the files (original dataset, XMIT dataset)?
Re: Pipe to gzip not working
Original dataset:
General Data
Management class . . : MD010703
Storage class . . . : SCTMMLG
Volume serial . . . : VDR001 +
Device type . . . . : 3390
Data class . . . . . : DCTMMDRP
Organization . . . : PS
Record format . . . : U
Record length . . . : 0
Block size . . . . : 32760
1st extent bytes . : 531236160
Secondary bytes . . : 531234816
Data set name type :
SMS Compressible. . : NO
Current Allocation
Allocated bytes . . : 1,425,813,480
Allocated extents . : 3
Current Utilization
Used bytes . . . . : 1,425,813,480
Used extents . . . : 3
XMIT dataset:
General Data
Management class . . : MD020703
Storage class . . . : SCPRDOLG
Volume serial . . . : DDV02E +
Device type . . . . : 3390
Data class . . . . . : DCTMMDRP
Organization . . . : PS
Record format . . . : FB
Record length . . . : 80
Block size . . . . : 3120
1st extent bytes . : 816894000
Secondary bytes . . : 531234816
Data set name type :
SMS Compressible. . : NO
Current Allocation
Allocated bytes . . : 2,193,048,000
Allocated extents . : 6
Current Utilization
Used bytes . . . . : 2,193,007,440
Used extents . . . : 6
General Data
Management class . . : MD010703
Storage class . . . : SCTMMLG
Volume serial . . . : VDR001 +
Device type . . . . : 3390
Data class . . . . . : DCTMMDRP
Organization . . . : PS
Record format . . . : U
Record length . . . : 0
Block size . . . . : 32760
1st extent bytes . : 531236160
Secondary bytes . . : 531234816
Data set name type :
SMS Compressible. . : NO
Current Allocation
Allocated bytes . . : 1,425,813,480
Allocated extents . : 3
Current Utilization
Used bytes . . . . : 1,425,813,480
Used extents . . . : 3
XMIT dataset:
General Data
Management class . . : MD020703
Storage class . . . : SCPRDOLG
Volume serial . . . : DDV02E +
Device type . . . . : 3390
Data class . . . . . : DCTMMDRP
Organization . . . : PS
Record format . . . : FB
Record length . . . : 80
Block size . . . . : 3120
1st extent bytes . : 816894000
Secondary bytes . . : 531234816
Data set name type :
SMS Compressible. . : NO
Current Allocation
Allocated bytes . . : 2,193,048,000
Allocated extents . : 6
Current Utilization
Used bytes . . . . : 2,193,007,440
Used extents . . . : 6
Re: Pipe to gzip not working
So, its a RECFM=U dataset... that might explain some of the problem if the average record length is small.
The IBM C library doesn't perform well with small unblocked records.
Do this from a z/OS Unix shell and post the results:
> time fromdsn -b -l rdw //original.dsn > /dev/null
The IBM C library doesn't perform well with small unblocked records.
Do this from a z/OS Unix shell and post the results:
> time fromdsn -b -l rdw //original.dsn > /dev/null
Re: Pipe to gzip not working
# time fromdsn -b -l rdw //DRP.ABARS.DMGTEST.D.C01V0015 > /dev/null
fromdsn(DRP.ABARS.DMGTEST.D.C01V0015)Nì: 87083 records/2175683079 bytes read; 2176031411 bytes written in 28.259 seconds (73.428 MB
ytes/sec).
real 0m28.61s
user 0m 1.37s
sys 0m 0.46s
fromdsn(DRP.ABARS.DMGTEST.D.C01V0015)Nì: 87083 records/2175683079 bytes read; 2176031411 bytes written in 28.259 seconds (73.428 MB
ytes/sec).
real 0m28.61s
user 0m 1.37s
sys 0m 0.46s
Re: Pipe to gzip not working
That's a little less than 2 seconds of CPU. So, that doesn't account for the difference (or at least not all of the 5 second difference).
So, I can't say much more until we try to recreate this test. I think that we have enough data, but I can't promise when we'll get to it.
So, I can't say much more until we try to recreate this test. I think that we have enough data, but I can't promise when we'll get to it.
Re: Pipe to gzip not working
I guess I'm confused as I thought that the execution load was supposed to be offloaded from the mainframe and onto the remote machine. But the cpu time seems directly proportional to the amount of data I am sending, so cpu consumption on the mainframe consists of more than just establishing the data connection. Can you provide more detail as to what pieces are executing on the mainframe versus the remote machine?
thanks,
Sandy
thanks,
Sandy