fromdsn -l option misbehaving

jacobsm · Post by **jacobsm** » Fri Jan 14, 2011 9:46 am

I'm using -l 0x0d0a on an fromdsn command with the -t ASCII option but when the file gets to my workstation the last two bytes are 0x150a.

I've tried several things, when I reverse the bytes using -l0a0d my workstation shows 0x0a15.

Can you assist?

Mark Jacobs

Post by **dovetail** » Fri Jan 14, 2011 10:01 am

Mark,

How are you executing the fromdsn command? (locally on z/OS, via the CO:Z launcher, or remotely initiated via SSH)?

If you execute locally on z/OS, try:

fromdsn -l crlf -t ASCII > /tmp/local.ascii.txt

and then use ISPF 3.17 and display the data in HEX / ASCII mode to see if it is what you expected. You might also try "ISO8859-1" instead of ASCII.

jacobsm · Post by **jacobsm** » Fri Jan 14, 2011 10:42 am

I'm executing the fromdsn command under zos then executing an sftp to my workstation.

This command fromdsn -l crlf -t ISO8859-1 DD:IN | todsn //DD:OUT gives me this

00000000 2f 2a 20 4c 49 42 3a 20 43 50 41 43 2e 50 41 52 |/* LIB: CPAC.PAR|
00000010 4d 4c 49 42 28 49 4b 4a 54 53 4f 30 30 29 20 20 |MLIB(IKJTSO00) |
00000020 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 | |
00000040 20 20 20 2a 2f 15 0a 2f 2a 20 44 4f 43 3a 20 54 | */../* DOC: T|

This fromdsn -l 0x0d0a -t ISO8859-1 DD:IN | todsn //DD:OUT gives the same

00000000 2f 2a 20 4c 49 42 3a 20 43 50 41 43 2e 50 41 52 |/* LIB: CPAC.PAR|
00000010 4d 4c 49 42 28 49 4b 4a 54 53 4f 30 30 29 20 20 |MLIB(IKJTSO00) |
00000020 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 | |
00000040 20 20 20 2a 2f 15 0a 2f 2a 20 44 4f 43 3a 20 54 | */../* DOC: T|

This fromdsn -t ISO8859-1 //DD:IN | todsn //DD:OUT gives

00000000 2f 2a 20 4c 49 42 3a 20 43 50 41 43 2e 50 41 52 |/* LIB: CPAC.PAR|
00000010 4d 4c 49 42 28 49 4b 4a 54 53 4f 30 30 29 20 20 |MLIB(IKJTSO00) |
00000020 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 | |
00000040 20 20 20 2a 2f 0a 2f 2a 20 44 4f 43 3a 20 54 48 | */./* DOC: TH|

I've run several combination's and we can't seem to get each line terminated by x'0d0a' as the client expects.

Mark Jacobs

Post by **dovetail** » Fri Jan 14, 2011 11:38 am

Mark,

The problem above is that todsn defaults to text mode, so it is looking for newlines in the data to make records (in EBCDIC), but EBCDIC newlines are also 0D0A.

If you add "-b" to your todsn commands above, it wiill treat the data as binary and the todsn target dataset will have the ascii data, including 0D0As and the record boundaries will be meaningless (each record will be filled to capacity). So this data could be sent in binary to an ASCII host, ignoring record boundaries.

But, why not just use cozsftp directly without pre-processing the data with fromdsn/todsn - it has the same capabilities without the extra copies/processing:

Code: Select all

> cozsftp ....
sftp> ls /+mode=text
sftp> ls /+linerule=crlf
sftp> ls /+servercp=ASCII   (the default would be ISO8859-1)
sftp> put //DD:IN  /path/to/remote/file

jacobsm · Post by **jacobsm** » Fri Jan 14, 2011 12:27 pm

Thanks, I'll try the -b option.

I don't think cozsftp will work for this job since the batch job is initiated on the zos side and puts the data to a server. We usually don't allow a server to connect to the zos mainframe (there are exceptions however.)

Mark Jacobs

Post by **dovetail** » Fri Jan 14, 2011 12:36 pm

Check out the sample JCLs of running the Co:Z SFTP client in a batch job (RUNSFTPx).

jacobsm · Post by **jacobsm** » Fri Jan 14, 2011 1:54 pm

Almost got it to what the client expects. We're having x'15' inserted into the converted data stream, which looks like ever x'401' bytes, and one at the end of the file.

[mark@bc008988 /usr/home/mark]$ hd srchfor.list | grep ' 15 '
00000400 15 20 20 20 41 4c 55 20 20 20 20 20 20 41 4c 54 |. ALU ALT|
00000800 20 15 20 20 20 20 20 20 20 20 20 20 20 20 20 20 | . |
00000c00 20 20 15 20 20 20 20 20 20 2f 2a 20 45 4e 44 45 | . /* ENDE|
00001000 2f 20 2b 15 0d 0a 20 20 20 52 45 43 45 49 56 45 |/ +... RECEIVE|
00001400 41 52 44 20 15 20 20 20 20 20 20 20 20 20 20 20 |ARD . |
00001800 30 30 20 20 20 15 20 20 20 20 20 20 20 20 20 20 |00 . |
00001c00 20 20 20 20 20 20 15 2a 2f 20 2b 0d 0a 20 20 20 | .*/ +.. |
00002000 2a 20 56 41 4e 47 55 15 41 52 44 20 20 20 20 20 |* VANGU.ARD |
00002400 20 54 45 52 4d 20 20 20 15 20 20 54 45 52 4d 49 | TERM . TERMI|
00002800 0d 0a 20 20 20 41 41 43 4d 15 44 30 31 34 20 20 |.. AACM.D014 |
00002c00 44 55 4c 45 52 20 20 2a 2f 20 15 20 2b 0d 0a 20 |DULER */ . +.. |
00003000 4e 49 54 4f 52 3d 43 50 55 20 49 15 4e 46 4f 20 |NITOR=CPU I.NFO |
00003400 20 20 20 20 20 2f 2a 20 50 4c 41 54 15 49 4e 55 | /* PLAT.INU|
00003800 53 45 4e 44 28 4f 4e 29 20 20 20 20 20 15 20 20 |SEND(ON) . |
00003c00 20 20 20 20 20 20 28 4e 59 49 53 43 2c 53 15 59 | (NYISC,S.Y|
00004000 43 54 4c 28 4e 41 4d 45 53 2e 54 45 58 54 29 15 |CTL(NAMES.TEXT).|
00004090 4e 48 45 52 45 28 4f 4e 29 0d 0a 15 |NHERE(ON)...|

Post by **dovetail** » Fri Jan 14, 2011 2:06 pm

Its not clear to me where/who is inserting this into the data stream.

If you do the "fromdsn" and pipe to a z/OS Unix file do you see the bad data?

If not, then if you pipe fromdsn into todsn with -b and look at the dataset, what then?

If still not, then the data is being corrupted when you send it to the "client'. How is that done?

jacobsm · Post by **jacobsm** » Fri Jan 14, 2011 2:45 pm

I looked at the file after the fromdsn command executed, and before it was piped to the todsn command.

It was fine, no x'15's inserted.

The file created by the todsn command and before the sftp to my workstation does have the x'15''s inserted into the file every 1024 bytes.

Post by **dovetail** » Sun Jan 16, 2011 7:48 pm

To figure out what is going wrong when you try this, let me do the following example.

The following series of commands are executed in the z/OS Unix shell (either TSO OMVS, or a TTY Unix Shell, SSH shell, or in COZBATCH/BPXBATCH).

Note: I'm using ISPF BROWSE to display the datasets or files in hex, but you may need to copy and paste the display into a fixed-width font hex editor if you want to see it aligned properly.

First, make a variable length MVS dataset. I'll use the the /etc/profile Unix file as input:

Code: Select all

todsn //KIRK.ETC.PROF < /etc/profile

Since the above command takes all of the defaults, it reads the stream of bytes from stdin (redirected from the file /etc/profile) and writes records to the dataset. The default source codepage and target is based on your current Unix locale, which for most is IBM-1047. The default LineRule is to use any line ending (newline, cr, or combination of the two) to break up source lines into records.

So the resulting dataset will be in EBCDIC and records will be determined by the newline (X'15') characters in the source file /etc/profile:

Code: Select all

BROWSE    KIRK.ETC.PROF
*********************************************************** Top of Data 
                                                                        
                                                                        
 -----------------------------------------------------------------------
#***********************************************************************
755555555555555555555555555555555555555555555555555555555555555555555555
BCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
 -----------------------------------------------------------------------
#       Licensed Materials - Property of IBM                           *
74444444D8889A884D8A89889A464D99989AA4984CCD4444444444444444444444444445
B0000000393552540413599132000796759380660924000000000000000000000000000C
 -----------------------------------------------------------------------
#       5647-A01                                                       *
74444444FFFF6CFF44444444444444444444444444444444444444444444444444444445
B0000000564701010000000000000000000000000000000000000000000000000000000C
 -----------------------------------------------------------------------
#       (C) Copyright IBM Corp. 1992, 2006                             *
744444444C54C99A9888A4CCD4C99944FFFF64FFFF444444444444444444444444444445
B0000000D3D0367899783092403697B01992B0200600000000000000000000000000000C
 -----------------------------------------------------------------------
#                                                                      *
744444444444444444444444444444444444444444444444444444444444444444444445
B0000000000000000000000000000000000000000000000000000000000000000000000C
 -----------------------------------------------------------------------
#***********************************************************************
755555555555555555555555555555555555555555555555555555555555555555555555
BCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
 -----------------------------------------------------------------------
# This is a sample profile defining system wide variables. The          
74E88A48A484A89998499988984888898984AAAA894A8884A8988898A44E88          
B03892092010214735079669350456959570282354069450519912352B0385          
 -----------------------------------------------------------------------
# variables set here may be overridden by a user's personal .profile    
74A8988898A4A8A48898498A48849A8998888948A484AA897A4989A9989449998898    
B05199123520253085950418025065599944550280104259D20759265130B7966935    
 -----------------------------------------------------------------------
# in their $HOME directory.                                             
74894A888945CDDC488988A99A4                                             
B0950385990B86450499533698B                                             
 -----------------------------------------------------------------------
#                                                                       
7                                                                       
B                                                                       
 -----------------------------------------------------------------------
# In order to customize this profile, you must first copy this sample   
74C94998894A948AAA998A84A88A4999889864A9A49AAA4889AA4899A4A88A4A89998   
B09506945903603423649950389207966935B08640442306992303678038920214735   
 -----------------------------------------------------------------------
# to /etc/profile. More information on this profile may be found in the 
74A9468A86999889844D99848989998A8994994A88A49998898498A488489A984894A88 
B0360153317966935B04695095669413965065038920796693504180250664540950385

Now, the next step will be to take this dataset and create a stream of bytes that are in ASCII, with the source record boundaries transformed to ASCII CRLF characters (rather than just LF characters, which is the default). For now, we will direct this stream of bytes to a Unix file:

Code: Select all

fromdsn -l crlf -t IS8859-1 //KIRK.ETC.PROF > /tmp/etc.prof.ascii

Code: Select all

 BROWSE    /S0W1/tmp/etc.prof.ascii                                                                        Line 00000000 Col 001 133
*********************************************************** Top of Data *************************************************************
                                                                                                                                     
                                                                                                                                     
 -----------------------------------------------------------------------------------------------------------------------------------
..................................................................................<ÑÄÁ>ËÁÀ.(/ÈÁÊÑ/%Ë...&Ê?øÁÊÈ`.?Ã.ñâ(...............
2222222222222222222222222222222222222222222222222222222222222222222222220022222222466667662467676667222576767772662444222222222222222
3AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADA30000000C935E3540D145291C30D002F052490F6092D000000000000000
 -----------------------------------------------------------------------------------------------------------------------------------

A FIND x'15' command in this file yields no results.

The next step is to create a new dataset with the binary contents of the /tmp/etc/prof.ascii file. The todsn -b option will disable translation, and the data from the input file will be dumped into the records without respect to line terminators. Each record will be filled to capacity, and since we don't specify a DCB when we create the dataset, it will be RECFM=VB,LRECL=1028, so that the max record length is 1024:

Code: Select all

todsn -b //KIRK.ETC.PROF.ASCII.BIN  < /tmp/etc.prof.ascii

Code: Select all

 BROWSE    KIRK.ETC.PROF.ASCII.BIN                                                                         Line 00000000 Col 001 133
*********************************************************** Top of Data *************************************************************
                                                                                                                                     
                                                                                                                                     
 -----------------------------------------------------------------------------------------------------------------------------------
..................................................................................<ÑÄÁ>ËÁÀ.(/ÈÁÊÑ/%Ë...&Ê?øÁÊÈ`.?Ã.ñâ(...............
2222222222222222222222222222222222222222222222222222222222222222222222220022222222466667662467676667222576767772662444222222222222222
3AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADA30000000C935E3540D145291C30D002F052490F6092D000000000000000
 -----------------------------------------------------------------------------------------------------------------------------------
À.....È?.Í>Ä?__Á>È.?Ê.Ä?__Á>È.ÈÇÁ.ÀÁËÑÊÁÀ.%Ñ>ÁË........î/ÊÑ/Â%ÁË._ÍËÈ.ÂÁ..ÁÌø?ÊÈÁÀ..Ñ>.?ÊÀÁÊ.Ã?Ê.ÈÇÁ.Î/ÊÑ/Â%ÁË.È?.ÂÁ..../Î/Ñ%/Â%Á.Ã?Ê
6222227627666666672672666666727662667676626666720020022567666667267772662267767766226626766726672766276766666727626600226766666662667
4073704F05E3FDD5E40F203FDD5E40485045392540C9E53EDA3DA30612912C530D53402507580F2454709E0F245206F204850612912C5304F025DA301619C12C506F2
 -----------------------------------------------------------------------------------------------------------------------------------
ÁÊÃ?Ê_/>ÄÁ.Ã?Ê.ÍËÁÊË.ÃÊ?_.ñë&ã.?Ê.ÍËÁÊË.ÏÑÈÇ.À/È/.ËÁÈË./%%?Ä/ÈÁÀ.È?....ëèá&<ñâ.àà.ËÈ/ÈÁ_Á>ÈË..ËøÁÄÑÃ`..ëèá&<ñâ.>?>Á.......èÇÑË.øÁÊÃ?Ê
6766766666266727767726766245542672776772767626676276772666666766276002255454442442776766667722776666722554544436666222002256672767667
526F2D1E3506F2053523062FD093060F205352307948041410354301CCF3145404FDA303450C92044034145D5E43C03053969023450C92DEFE520EDA30489300526F2
 -----------------------------------------------------------------------------------------------------------------------------------
.......................................................&ë....<|å+ (á....&ïà........ÁÌø?ÊÈ.&ë.........................................
3333333333333333333333333333333333333333333333333333300553322444444423225542322320067767725530020022333333333333333333333333333333333
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDA031D74CF7E1D57A740747A70E7DA580F240031DA0DA30DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
 -----------------------------------------------------------------------------------------------------------------------------------
.Î/ÊÑ/Â%Á.....................................................ëøÁÄÑÃÑÁË.ÈÇÁ.%ÑËÈ.?Ã.ÀÑÊÁÄÈ?ÊÑÁË.ÈÇ/È.ÈÇÁ.Ë`ËÈÁ_.ËÁ/ÊÄÇÁË.Ã?Ê./>....ÁÌ
2767666660022222222222222222222222222222222222222222222222002257666666727662667726626676676766727667276627777662766766672667266002267
0612912C5DA3000000000000000000000DDDDDDDDDDDDDDDDDDDDDDDDDDA3030539695304850C9340F60492534F2953048140485039345D03512385306F201EDA3058
 -----------------------------------------------------------------------------------------------------------------------------------
Ê?>_Á>È.Î/ÊÑ/Â%Á.......................................................ëøÁÄÑÃÑÁË.ÈÇÁ.%ÑËÈ.?Ã.ÀÑÊÁÄÈ?ÊÑÁË.ÈÇ/È.ÈÇÁ.Ë`ËÈÁ_.ËÁ/ÊÄÇÁË.Ã?Ê
7666667276766666002222222222222222222222222222222222222222222222222002257666666727662667726626676676766727667276627777662766766672667
2FED5E40612912C5DA300000000000000000000DDDDDDDDDDDDDDDDDDDDDDDDDDDDDA3030539695304850C9340F60492534F2953048140485039345D03512385306F2
 -----------------------------------------------------------------------------------------------------------------------------------
ÎÑÊ?>_Á>È.Î/ÊÑ/Â%Á.....................................................ëÁÈË.ÈÇÁ.>/_Á.?Ã.ÈÇÁ.ÍËÁÊ.Ë._/Ñ%Â?Ì.ÃÑ%Á./>À.Á>/Â%ÁË._/Ñ%....>
7676666672767666660022222222222222222222222222222222222222222222222002256772766266662662766277672726666667266662666266666672666600226
692FED5E40612912C5DA3000000000000000000000DDDDDDDDDDDDDDDDDDDDDDDDDDA30354304850E1D50F6048505352730D19C2F8069C501E405E12C530D19CDA30E
 -----------------------------------------------------------------------------------------------------------------------------------
ÈÇÁ.Ä?_øÑ%ÁÊ./>À.?Ê.ÊÍ>ÈÑ_Á.%ÑÂÊ/Ê`.Á%Á_Á>ÈË.ÍËÁ......ÀÑÃÃÁÊÁ>È.Î/%ÍÁË..ÈÇÁ>.ÈÇÁ./øøÊ?øÊÑ/ÈÁ..ÁÌø?ÊÈ..%Ñ>ÁË.ËÇ?Í%À.ÂÁ......ËÁÈ.È?.ÈÇÁ
7662666766672666267277676662666767726666667727760022226666676672766767227666276626777677667622677677226666727667662660022227672762766
48503FD09C5201E4FF2025E49D50C92212905C5D5E430535DA30004966525E4061C553C0485E048501002F02914502580F2420C9E53038F5C4025DA300035404F0485
 -----------------------------------------------------------------------------------------------------------------------------------
.....ËÁÈ..è?.ËÁÈ./>`.ø/ÊÈÑÄÍ%/Ê.Ä....ÄÄ.?Ê.Ä...Î/ÊÑ/Â%Á.ÀÑÃÃÁÊÁ>È%`.......¦ÍËÈ.Ä?ÀÁ.ÈÇÁ.>ÁÄÁËË/Ê`..ÁÌø?ÊÈ..ËÈ/ÈÁ_Á>È..ÍËÑ>Å.ÈÇÁ./øøÊ?
0222276722562767266727677667667263322662672622276766666266666766767200222267772666627662666677677226776772277676666722776662766267776
A3000354E04F035401E900124935C120389C0330F203BB0612912C504966525E4C9CDA3000A53403F4504850E5353312902580F242034145D5E408539E7048501002F
 -----------------------------------------------------------------------------------------------------------------------------------
Ê.....................................................ÁÌø?ÊÈ..#¬ä(&'¬ä<ñâ¬&êáãñì..äâä.............&ÊÁ%Ñ>,ÁÊ./>À.ÊÍ>ÈÑ_Á.%ÑÂÊ/Ê`......
7300222222222222222222222222222222222222222222220022226776772275445754444555444532444200200200222257666666726662776766626667677300222
2ADA3000DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDA3000580F2404BF3D0DF3C92F025698D23232DA3DA3DA3000025C9EB5201E4025E49D50C922129ADA300
 -----------------------------------------------------------------------------------------------------------------------------------
.....................................................................................................................................
2222222222222222222222222222222222222222222222222222222222222222222222002002233333333333333333333333333333333333333333333333333333333
3333333333333333333333333333333333333333333333333333333333333333333333DA3DA30DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
 -----------------------------------------------------------------------------------------------------------------------------------
********************************************************** Bottom of Data ***********************************************************

A FIND x'15' command in this dataset also yields no matches.

Now, what you will see is that each record except the last in this dataset is length=1024. So I suspect that the sftp command that you are using to send this dataset to the remote system is treating it as text and inserting X'15' separators after each record.

Combining the above steps, you can take an EBCDIC dataset with RECFM=VB records and create a a dataset in ASCII where the record boundaries are meaningless:

Code: Select all

fromdsn -l crlf -t ISO8859-1 //KIRK.ETC.PROF | 
    todsn -b //KIRK.ETC.PROF.ASCII.BIN

If you were to use the Co:Z SFTP client to send the resulting dataset to a remote system, you would want to set "mode=binary" so that the records boundaries weren't replaced with separators, and so that no codepage transform was done.

I don't see a reason to go through these steps before using sftp to send a file. If you want to send a RECFM=VB text dataset to a remote ASCII server with CRLF (0D0A) line separators, you could use something like this (a modified version of our RUNSFTP sample JCL):

Code: Select all

//RUNSFTP EXEC PGM=COZBATCH
//STDIN DD *
                                                                            
# Customize these ... 
coz_bin="/opt/dovetail/coz/bin" 
remoteuser="uid" 
server="remote.host.name" 
servercp="ISO8859-1" 
remotefile="/path/to/file" 
                                                                            
# These can be used to read the ssh password from a (secured) dataset 
# if you don't want to setup public/private keypairs 
export PASSWD_DSN='//COZUSER.PASSWD(SITE1)' 
export SSH_ASKPASS=$coz_bin/read_passwd_dsn.sh 
export DISPLAY=none 
                                                                            
ssh_opts="-oBatchMode=no"      # allows ssh to use SSH_ASKPASS program 
ssh_opts="$ssh_opts -oConnectTimeout=60" 
ssh_opts="$ssh_opts -oServerAliveInterval=60" 
ssh_opts="$ssh_opts -oStrictHostKeyChecking=no" # accept initial host keys 
                                                                            
# Invoke the Co:Z sftp client with an in-line batch of commands 
# that downloads a remote file to a local DD.   
# Note that "-oBatchMode=no" must be specified before "-b" 
#      since ssh opts are first-sticky 
                                                                            
$coz_bin/cozsftp  $ssh_opts -b- $remoteuser@$server <<EOB 
lzopts mode=text,servercp=$servercp
lzopts linerule=crlf
put //DD:INDD $remotefile
EOB 
                                      
//INDD DD DSN=HLQ.MY.DATASET,DISP=SHR
//