Convert to MF format using MFRDW

Discussion of the Co:Z Toolkit Dataset Pipes utilities
Post Reply
tsdjim
Posts: 64
Joined: Fri May 07, 2010 2:21 am

Convert to MF format using MFRDW

Post by tsdjim »

I have a z/os variable blocked file with a record consisting of a single record consisting of 01234567890 and I need to convert this to Microfocus format with a 128 byte header and a rdw for each record, keeping the file as EBCDIC in the target file. I send the file to Linux with the following command.

fromdsn -k -l mfrdw DD:IN > /iscsi_sda1/zos/bin/mfrdwcoz.txy.cvtd

I also used another utility from simotime to convert the same file to MF format, which I use it regularly and it works fine. When I compare the two file
from Dovetail and simotime, they do not match. Have I specified the fromdsn command correctly.

Here are the two files on Linux after conversion (hex mode with od -x):

Simotime conversion (correct)
0000000 3000 007c 0000 0000 3132 3038 3231 3131
0000020 3539 3333 3230 3132 3038 3231 3131 3539
0000040 3333 3230 003e 0001 0000 0000 0000 0000
0000060 0100 0000 0000 0000 7ff8 0000 0004 0000
0000100 0000 0000 0000 0000 0000 0000 0000 0100
0000120 0000 0000 0000 0000 0000 0000 0000 0000
0000140 0000 0000 0000 0000 0000 0000 5806 3e85
0000160 0000 0000 0000 0000 0000 0000 0000 0000
0000200 4000 000a f0f1 f2f3 f4f5 f6f7 f8f9 0000
0000220

fromdsn conversion (does not match above)
0000000 307e 0000 0000 0000 0000 0000 0000 0000
0000020 0000 0000 0000 0000 0000 0000 0000 0000
0000040 0000 0000 003e 0001 0000 0000 0000 0000
0000060 0100 0000 0000 0000 004c 0000 0000 0000
0000100 0000 0000 0000 0000 0000 0000 0000 0000
*
0000200 400c f0f1 f2f3 f4f5 f6f7 f8f9
coz
Posts: 391
Joined: Fri Jul 30, 2004 5:29 pm

Re: Convert to MF format using MFRDW

Post by coz »

The MicroFocus header for variable records has two layouts depending on the maximum LRECL. If the maximum is < 4095, the first four bytes of the header (length of the file header) contain the value: 0x307E0000. If the maximum is >= 4095 the first 4 bytes contain the value: 0x3000007C.

Offset 54 (0o66) contains the maximum record length. In your Simotime example below, the maximum LRECL is reported as: 0x00007ff8; this produces the initial 4 byte sequence of 0x3000007C. In your fromdsn example, the maximum LRECL is reported as: 0x0000004c. This produces the initial 4 byte sequence of 0x307E0000. Both of these are consistent with the spec.

The next step is to determine if the maximum LRECL is being reported correctly. What is the actual DCB information for your DD:IN?
tsdjim
Posts: 64
Joined: Fri May 07, 2010 2:21 am

Re: Convert to MF format using MFRDW

Post by tsdjim »

The max LRECL of the previous post dataset was lrecl=80

Here are the results when the same dataset has max LRECL=32700

simotime with max LRECL=32700
0000000 3000 007c 0000 0000 3132 3038 3232 3037
0000020 3134 3232 3939 3132 3038 3232 3037 3134
0000040 3232 3939 003e 0001 0000 0000 0000 0000
0000060 0100 0000 0000 0000 7ff8 0000 0004 0000
0000100 0000 0000 0000 0000 0000 0000 0000 0100
0000120 0000 0000 0000 0000 0000 0000 0000 0000
0000140 0000 0000 0000 0000 0000 0000 5806 3e85
0000160 0000 0000 0000 0000 0000 0000 0000 0000
0000200 4000 000a f0f1 f2f3 f4f5 f6f7 f8f9 0000
0000220

fromdsn with max LRECL=32700
0000000 3000 007c 0000 0000 0000 0000 0000 0000
0000020 0000 0000 0000 0000 0000 0000 0000 0000
0000040 0000 0000 003e 0001 0000 0000 0000 0000
0000060 0100 0000 0000 0000 7fb8 0000 0000 0000
0000100 0000 0000 0000 0000 0000 0000 0000 0000
*
0000200 4000 000e f0f1 f2f3 f4f5 f6f7 f8f9 0000
0000220
coz
Posts: 391
Joined: Fri Jul 30, 2004 5:29 pm

Re: Convert to MF format using MFRDW

Post by coz »

AFAIK, we are in conformance with the MicroFocus file structure specification, found here: http://supportline.microfocus.com/docum ... fhfile.htm

Can you please point out where you think the issues are? In particular, simotime is filling in header fields that are reserved for index files only - we leave these as zeros as per the spec.
tsdjim
Posts: 64
Joined: Fri May 07, 2010 2:21 am

Re: Convert to MF format using MFRDW

Post by tsdjim »

The reason I started out on this investigation was because a fromdsn converted file gave the following error in Microfocus, however the simotime converted file works fine:

error code: 9/018 (ANS74), pc=0, call=1, seg=0
18 Read part record error: EOF before EOR or file open in wrong mode

I am giving as much information as I could gather below:

The file in question is a large variable blocked file with 18M records and has max LRECL=3156

After conversion there is a mismatch in the files from simotime and fromdsn:
fromdsn size=2747757916 bytes, simotime size=2768233948bytes, a difference of 20476032 bytes

Here is the message from fromdsn during conversion:
fromdsn(DD:IN)?N18802519 :? records/2681306987 bytes read; 2747757916 bytes written

Here is the hex display at the head of each file:
simotime file:

0000000 3000 007c 0000 0000 3132 3038 3232 3233
0000010 3436 3432 3039 3132 3038 3232 3233 3436
0000020 3432 3039 003e 0001 0000 0000 0000 0000
0000030 0100 0000 0000 0000 7ff8 0000 0004 0000
0000040 0000 0000 0000 0000 0000 0000 0000 0100
0000050 0000 0000 0000 0000 0000 0000 0000 0000
0000060 0000 0000 0000 0000 0000 0000 5806 3e85
0000070 0000 0000 0000 0000 0000 0000 0000 0000
0000080 4000 0038 0000 0000 000f f0f1 0000 002c
0000090 f0f1 e2c6 e3f0 f0f0 d4d7 02fe f2f0 f1f2
00000a0 f0f8 f2f2 f2f0 f1f2 f0f8 f2f2 f2f0 f1f2
00000b0 f0f8 f2f2 f2f0 f1f2 f0f8 f2f2 4000 02c5
00000c0 0000 1000 017f f1f1 0000 02b9 f1f1 f0f1
00000d0 02b9 f1f0 f0f0 f0f1 f7f1 f001 9800 101c
00000e0 f2f0 0201 2082 8c02 0100 321c 0201 2072
00000f0 6cf4 f0f1 f0f2 f8f0 f0f0 f000 0054 2436
0000100 6d01 9771 001c 0196 4051 4c00 0000 000c
0000110 f0f0 f1f5 f1f1 f0f6 0201 2072 9c00 0000
0000120 000c 0000 0000 0c02 0120 801c 0201 2080
0000130 1cf1 f200 0003 2000 001c 4040 4040 4040
0000140 4040 4040 4040 4040 4040 4040 4040 4040
*
0000160 4040 4040 4040 4040 4040 4040 da75 debc
0000170 b856 40bd 56cf de73 ba56 4040 4040 4040
0000180 4040 4040 4040 4040 4040 4040 4040 4040
*
00001a0 4040 4040 4040 4040 40bd 56cf de73 ba56
00001b0 4040 b1de b0cf 4040 4040 4040 4040 4040
00001c0 4040 4040 4040 4040 4040 4040 4040 4040
*
00001f0 4040 e8e2 4040 4040 4040 4040 4040 4040
0000200 4040 4040 4040 4040 4040 4040 4040 4040
*
0000250 4040 4040 4040 4040
0000258


fromdsn file:

0000000 307e 0000 0000 0000 0000 0000 0000 0000
0000010 0000 0000 0000 0000 0000 0000 0000 0000
0000020 0000 0000 003e 0001 0000 0000 0000 0000
0000030 0100 0000 0000 0000 0c50 0000 0000 0000
0000040 0000 0000 0000 0000 0000 0000 0000 0000
*
0000080 403a 0000 0000 000f f0f1 0000 002c f0f1
0000090 e2c6 e3f0 f0f0 d4d7 02fe f2f0 f1f2 f0f8
00000a0 f2f2 f2f0 f1f2 f0f8 f2f2 f2f0 f1f2 f0f8
00000b0 f2f2 f2f0 f1f2 f0f8 f2f2 0000 42c7 0000
00000c0 1000 017f f1f1 0000 02b9 f1f1 f0f1 02b9
00000d0 f1f0 f0f0 f0f1 f7f1 f001 9800 101c f2f0
00000e0 0201 2082 8c02 0100 321c 0201 2072 6cf4
00000f0 f0f1 f0f2 f8f0 f0f0 f000 0054 2436 6d01
0000100 9771 001c 0196 4051 4c00 0000 000c f0f0
0000110 f1f5 f1f1 f0f6 0201 2072 9c00 0000 000c
0000120 0000 0000 0c02 0120 801c 0201 2080 1cf1
0000130 f200 0003 2000 001c 4040 4040 4040 4040
0000140 4040 4040 4040 4040 4040 4040 4040 4040
*
0000160 4040 4040 4040 4040 4040 da75 debc b856
0000170 40bd 56cf de73 ba56 4040 4040 4040 4040
0000180 4040 4040 4040 4040 4040 4040 4040 4040
*
00001a0 4040 4040 4040 40bd 56cf de73 ba56 4040
00001b0 b1de b0cf 4040 4040 4040 4040 4040 4040
00001c0 4040 4040 4040 4040 4040 4040 4040 4040
*
00001f0 e8e2 4040 4040 4040 4040 4040 4040 4040
0000200 4040 4040 4040 4040 4040 4040 4040 4040
*
0000250 4040 4040 4040 4040
0000258


Here is the hex display at the tail of each file:

fromdsn:
0025520 000001 171760 000050 000012 000000 171763 000050 000012
0025540 000000 171765 000050 000005 000000 171770 000051 000036
0025560 000000 172360 000145 000144 000000 172363 000144 000004
0025600 000000 172365 000057 000372 000000 172760 001061 000144
0025620 000017 172762 001145 000001 000001 172766 001261 000017
0025640 000000 173360 000104 000024 000000 173760 000116 000062
0025660 000000 173765 000246 000062 000002 174360 000054 000050
0025700 000002 174760 000074 000002 000000 177777 177777 177777
0025720 177777 000000
0025724

simotime:
0025660 000001 171365 000334 000002 000001 171760 000050 000012
0025700 000000 171763 000050 000012 000000 171765 000050 000005
0025720 000000 171770 000051 000036 000000 172360 000145 000144
0025740 000000 172363 000144 000004 000000 172365 000057 000372
0025760 000000 172760 001061 000144 000017 172762 001145 000001
0026000 000001 172766 001261 000017 000000 173360 000104 000024
0026020 000000 173760 000116 000062 000000 173765 000246 000062
0026040 000002 174360 000054 000050 000002 174760 000074 000002
0026060 000000 177777 177777 177777 177777
0026072
coz
Posts: 391
Joined: Fri Jul 30, 2004 5:29 pm

Re: Convert to MF format using MFRDW

Post by coz »

We appear to have a difference with simotime in our interpretation of the spec regarding the Standard Record Header length field. Given that your simotime file is being read properly by MicroFocus, there's a good chance that we have it wrong. This is a fairly simple fix - would you be willing to test this for us if we make an intermediate build available?

If so, please send an email to info@dovetail.com and I will provide instructions for getting the build.

By the way, it appears that simotime insists on using 4 byte record length fields, even when the maximum LRECL is < 4095. This is the reason the simotime transfer is 20M larger than fromdsn.
Post Reply