Can dataset pipes do this?

Discussion of the Co:Z Toolkit Dataset Pipes utilities
Post Reply
sdean4
Posts: 25
Joined: Wed Nov 07, 2012 2:05 pm

Can dataset pipes do this?

Post by sdean4 »

I have a large number of files that I need to FTP from z/os on the mainframe to a remote unix box. This process consumes a lot of resources on the mainframe. I was wondering if I could use Dataset Pipes on a local LINUX box (which has excess capacity in an IFL) to do this FTP instead? Ideally I would like to leave my mainframe data where it is on z/os and have Linux access that data in-place, gzip the z/os data to a local Linux file, and then FTP the gzipped data to the remote UNIX location. Obviously I want all of the resource consumption (CPU) to occur on the Linux IFL. I don't know anything about Dataset Pipes or the Launcher to know if this is something it could do. Thanks!
dovetail
Site Admin
Posts: 2022
Joined: Thu Jul 29, 2004 12:12 pm

Re: Can dataset pipes do this?

Post by dovetail »

Can you elaborate on some of the details?

- what kind of datasets? (text / binary , fixed/var records)
- how are the datasets specified?
- specifically what kind of FTP are your currently doing (FTP? FTP/TLS? SFTP?) ; please elaborate on why the CPU consumption is high
sdean4
Posts: 25
Joined: Wed Nov 07, 2012 2:05 pm

Re: Can dataset pipes do this?

Post by sdean4 »

Our files are originally VB files, which we convert to FB by running it through TSO TRANSMIT. Then we use FTP to send it to a unix box where it is gzipped for longterm storage. Then when we need to access the datasets, we gunzip it, FTP back to the mainframe (a different mainframe from the source), and change it back to its original VB format by doing a TSO RECEIVE. We do the first half of this process many times daily, and some of the files are quite large (500GB). Since we are cpu-constrained on the mainframe, ANYTHING we can port to a unix box is a plus for us.
I was also wondering if we could omit the XMIT/RECEIVE steps and use dataset pipes to gzip the VB files directly. If so, I assume we would need dataset pipes on that second mainframe as well? Or would we need dataset pipes on the second mainframe regardless of the presence of XMIT/RECEIVE?
thanks,
Sandy
dovetail
Site Admin
Posts: 2022
Joined: Thu Jul 29, 2004 12:12 pm

Re: Can dataset pipes do this?

Post by dovetail »

I'm starting to get the idea, but you didn't answer my question about what kind of FTP. Are you using encryption? Is encryption required between the mainframe and the Unix server?
dovetail
Site Admin
Posts: 2022
Joined: Thu Jul 29, 2004 12:12 pm

Re: Can dataset pipes do this?

Post by dovetail »

About the cheapest/fastest way that I can think to do this is to use a z/OS Hybrid batch job (using Co:Z Launcher and Dataset Pipes) that directly reads your VB binary datasets and gzips them on the target Linux box. CPU time can be significantly reduced if the data traffic is sent in a clear socket rather than tunneled in SSH (option ssh-tunnel=false)

Example 1 (send dataset to gziped file on *nix)

//RUNCOZK EXEC PROC=COZPROC,
// ARGS='kirk@linux1.dovetail.com'
//COZCFG DD *
ssh-tunnel=false
//STDIN DD *
# customize these:
dsn=HLQ.MY.DSN
serverDir=/home/user/whatever

set -e # fail if any command has non-zero exit code
set -o pipefail # pipeline exit code is last non-zero exit code
cd $serverDir
fromdsn -b -l rdw //$dsn |
gzip > $dsn.gz
//


( you could factor this into a reusable shell script on the target system; transfer multiple files in the same step, etc)

Example 2 (pull dataset from gziped file on *nix)

//RUNCOZK EXEC PROC=COZPROC,
// ARGS='kirk@linux1.dovetail.com'
//COZCFG DD *
ssh-tunnel=false
//STDIN DD *
# customize these:
dsn=HLQ.MY.DSN
serverDir=/home/user/whatever

set -e # fail if any command has non-zero exit code
set -o pipefail # pipeline exit code is last non-zero exit code
cd $serverDir
gunzip -c $dsn.gz |
todsn -b -l rdw //$dsn
//
sdean4
Posts: 25
Joined: Wed Nov 07, 2012 2:05 pm

Re: Can dataset pipes do this?

Post by sdean4 »

This sounds great! I can hardly wait to give this a try. We are currently using plain FTP. Encryption is not required as everything is internal. So the use of OpenSsh is not required then?
dovetail
Site Admin
Posts: 2022
Joined: Thu Jul 29, 2004 12:12 pm

Re: Can dataset pipes do this?

Post by dovetail »

Yes, OpenSSH is still required, since it is used to start the remote connection and login to the remote server securely.

Also, I should probably have pointed out that on the todsn command you will need to know and code the DCB and space allocation using BPXWDYN parameters. Something like:

todsn -b -l rdw -x 'recfm(v,b) lrecl(1028) space(10,10) cyl release' //$dsn

A couple of points:

- to use Co:Z Launcher, your remote system will need to have the Co:Z Target System Toolkit installed.
- to use ssh-tunnel=false, you will need the ports in the "server-ports" property range open from the server to z/OS
- it isn't shown in the example, but you need to set up SSH user and host authentication between z/OS and the remote server

So, there's a bit to set up and understand if you haven't worked with IBM Ported Tools OpenSSH before.
FWIW, for our commercial support customers, we will walk you through all of this.

PS> There is a new 10 minute video introducing 'z/OS Hybrid Batch" on our Webinars page: http://dovetail.com/webinars.html
sdean4
Posts: 25
Joined: Wed Nov 07, 2012 2:05 pm

Re: Can dataset pipes do this?

Post by sdean4 »

Thank you for all the help and information. I have gradually been working through all the video clips to become more familiar with the product. Can you send me email and/or give me a call so we can discuss commercial support options?
Post Reply