Get All files in a Linux Directory to members in a PDS

Discussion of Co:Z sftp, a port of OpenSSH sftp for z/OS
Post Reply
mdgilmore
Posts: 5
Joined: Thu Dec 08, 2016 12:28 pm

Get All files in a Linux Directory to members in a PDS

Post by mdgilmore »

I have a Linux Directory with several files in it. I want to issue one Get to copy all the files into individual members in a PDS on z/OS.
So far, I have the JCL to the point that the whole directory is uploaded with one GET. But I can't get it to create members within a single PDS. It creates new datasets for each file with the last qualifiers being the filename and file extension.

JCL:
//SFTPIN DD *
host=[host_name]
user=[user_id]
pwdsn=[dataset_with_password]
lzopts="l=l4,dsntype=library"
. $script_dir/sftp_connect.sh << EOB
ASCII
cd /export/share/isc_security/OBFUSCATION/Mike/test
GET * //!SYS0033.OBF.DB2
EOB

Results:
[user_id].OBF.DB2
[user_id].OBF.DB2.A10064.CSV
[user_id].OBF.DB2.A10325.CSV
[user_id].OBF.DB2.A10327.CSV
[user_id].OBF.DB2.A10460.CSV
[user_id].OBF.DB2.A10706.CSV
[user_id].OBF.DB2.A10708.CSV
[user_id].OBF.DB2.A10772.CSV
[user_id].OBF.DB2.A10782.CSV
[user_id].OBF.DB2.A10861.CSV
[user_id].OBF.DB2.A10862.CSV
[user_id].OBF.DB2.A11086.CSV
[user_id].OBF.DB2.A11174.CSV
...
...
[user_id].OBF.DB2.B00059.CSV
[user_id].OBF.DB2.B00069.CSV
[user_id].OBF.DB2.B00079.CSV
[user_id].OBF.DB2.B00082.CSV
[user_id].OBF.DB2.B00083.CSV
[user_id].OBF.DB2.B00090.CSV
[user_id].OBF.DB2.B00094.CSV

Where the first dataset is the pre-allocated empty PDS that I wand the members created it.

Mikeg
dovetail
Site Admin
Posts: 2022
Joined: Thu Jul 29, 2004 12:12 pm

Re: Get All files in a Linux Directory to members in a PDS

Post by dovetail »

It appears that your linux directory has files with names like "Annnnn.CSV" in either upper or lower case.

When you try to get these names into a PDS, these names are not valid PDS member names, so it can't create or update members. This is because there isn't a feature in Co:Z SFTP that will for example drop suffixes, shorten names, change invalid member name characters.

So what it does it to create new data set names in the data set space. Since you used the "!" prefix character, it probably should have just failed.

For what you want to do, there are a couple of options:

1) change or copy the file names on linux to be valid PDS member names by dropping the ".csv", then do the Co:Z SFTP "get"
You could do this with some bash scripting, but the Linux "mmv" command/package lets you do fancy stuff like this:

Code: Select all

cd  csvdir
mkdir -p memberlinks && cd memberlinks
mmv -s ../\*.csv  \#1
this will create a set of symlinks: Annnnn -> ../Annnnn.csv in the memberlinks subdirectory

Then you can do a get csvdir/memberlinks/* into the PDS

2) upload the Linux files to z/OS unix files.
- aside: If you have *many* files, you might want to transfer as a tar and then use z/OS pax, since otherwise each sftp file transfer will be a separate allocate/open/write/close/free
- then use the new z/OS "putpds" shell command, which does have a mechanism for making member names from "member.suffix" names.
This also allows you to do a bunch of other fancy stuff, and it's really fast for large numbers of members.


Another option would be to use "Co:Z Hybrid Batch", where your batch job launched a bash script on linux that generated "todsn" commands for each Annnnn.csv -> MY.PDS(Annnnn). Or this could create a tar and upload it for a subsequent step to unpax and putpds.
Post Reply