I am migrating from Co:Z SFTP 1.3.2 to 2.0.1. I implemented this on our development LPARs without issue. When I moved it into our production LPARs, I would get an abend 4093 return code 1C. I was able to pinpoint this to the inclusion of the HEAP size override that I included in the cozsftp_config:
ssh-le-options=HEAP(12M,1M,,FREE),ENVAR("_CEE_REALLOC_CONTROL=256K,25")
I can get past this by upping my region size from 32m to 64m, but I do not understand why this happens in production but not test.
I compared my LE options between development and production - they were identical. I opened a ticket with IBM LE and provided them an LE trace to see if they could find any large allocations. IBM found nothing in the trace to indicate this and stated:
"I considered the possibility of there being more than one instance of
Language Environment involved and that we didn't get a dump from the
failing instance, but that seems unlikely. COZBATCH probably uses the
Unix system call loadhfs (BPX1LOD, BPX4LOD -- Load a program into
storage by path name) to load the target executable and subsequently
transfer control to it to avoid issues with child enclaves that would
arise if COZBATCH were itself an LE program. "
I am assuming there must be more than one instance of LE since the job I dumped only had one step, the one that I traced and dumped and the cause of the 4093-1C was never indicated in that trace. Could you explain to me what is happening here, and what might possibly be different between our deveopment LPARs and produciton LPARs that would cause this to occur.
Thank you
Getting abend 4093 code=1c
Re: Getting abend 4093 code=1c
This line:
ssh-le-options=HEAP(12M,1M,,FREE),ENVAR("_CEE_REALLOC_CONTROL=256K,25")
sets LE options that are used when cozsftp invoke's IBM Ported Tools OpenSSH - the ssh command.
We made this change to avoid out-of-memory conditions in IBM's product as documented in this APAR:
http://www-01.ibm.com/support/docview.w ... sg1OA34819
Even though this was a doc APAR, IBM released a PTF for Ported Tools 1.2 which is supposed to address this problem.
http://www-01.ibm.com/support/docview.w ... sg1OA37278
The ssh command is invoked using a Unix fork()/exec(), so it will run in a separate OMVS address space. So, COZBATCH, the Unix shell, and cozsftp run in the original address space but IBM Ported Tools ssh runs in a separate address, where I assume that you are getting the LE Storage ABEND. This is likely occurring since the ssh address space will inherit the REGION size from the original address space.
So, please see if you are running Ported Tools 1.2, and if you have the latest maintenance (PTF for OA37278) installed. If so, you may be able to remove the LE HEAP override and get by with a smaller REGION size for ssh. I would encourage you to ask IBM for REGION size and LE options overrides that are required to properly run the ssh command. Previous to this latest PTF, IBM had recommended REGION=64MB as a workaround for OA34819; but I don't know if that is still the recommendation. Setting the initial HEAP size to 12MB before forking ssh was an attempt to work around the OA34819 problem, but if you do that you end up using more during initialization since ssh starts child processes during startup and each of these would be getting 12MB of initial HEAP for a while.
I would go ahead and use REGION=64MB on the original COZBATCH jobstep.
The original AS with COZBATCH + Unix Shell + cozsftp will run fine in 16MB.
IBM's previous recommendation for ssh was 64MB. After it is running it should need much less.
ssh-le-options=HEAP(12M,1M,,FREE),ENVAR("_CEE_REALLOC_CONTROL=256K,25")
sets LE options that are used when cozsftp invoke's IBM Ported Tools OpenSSH - the ssh command.
We made this change to avoid out-of-memory conditions in IBM's product as documented in this APAR:
http://www-01.ibm.com/support/docview.w ... sg1OA34819
Even though this was a doc APAR, IBM released a PTF for Ported Tools 1.2 which is supposed to address this problem.
http://www-01.ibm.com/support/docview.w ... sg1OA37278
The ssh command is invoked using a Unix fork()/exec(), so it will run in a separate OMVS address space. So, COZBATCH, the Unix shell, and cozsftp run in the original address space but IBM Ported Tools ssh runs in a separate address, where I assume that you are getting the LE Storage ABEND. This is likely occurring since the ssh address space will inherit the REGION size from the original address space.
So, please see if you are running Ported Tools 1.2, and if you have the latest maintenance (PTF for OA37278) installed. If so, you may be able to remove the LE HEAP override and get by with a smaller REGION size for ssh. I would encourage you to ask IBM for REGION size and LE options overrides that are required to properly run the ssh command. Previous to this latest PTF, IBM had recommended REGION=64MB as a workaround for OA34819; but I don't know if that is still the recommendation. Setting the initial HEAP size to 12MB before forking ssh was an attempt to work around the OA34819 problem, but if you do that you end up using more during initialization since ssh starts child processes during startup and each of these would be getting 12MB of initial HEAP for a while.
I would go ahead and use REGION=64MB on the original COZBATCH jobstep.
The original AS with COZBATCH + Unix Shell + cozsftp will run fine in 16MB.
IBM's previous recommendation for ssh was 64MB. After it is running it should need much less.
Re: Getting abend 4093 code=1c
Sorry, I didn't address the question about differences between LPARs.
Here is one guess - does the working LPAR have /dev/random (ICSF random support) enabled? If so, it will be used by IBM Ported Tools in favor of "ssh-rand-helper" which starts a boat load of processes in order to generate some entropy. ICSF + /dev/random is MUCH faster/better for ssh startup. If you don't have it, it might explain why you need a larger region for ssh to start.
Here is one guess - does the working LPAR have /dev/random (ICSF random support) enabled? If so, it will be used by IBM Ported Tools in favor of "ssh-rand-helper" which starts a boat load of processes in order to generate some entropy. ICSF + /dev/random is MUCH faster/better for ssh startup. If you don't have it, it might explain why you need a larger region for ssh to start.
Re: Getting abend 4093 code=1c
Thank you very much for your quick response. We are not using ICSF random support at all, so that wouldn't be the difference, it may just have to remain one of those unknowns!
We are at Ported Tools 1.2 but do not yet have OA37278 applied. I had looked at OA37278 in the past and saw that it was supposed to fix some of the issues addressed by OA34819, but it didn't go into any detail, so I wasn't sure what it did to fix the problem.
When I hit the problem that is reported in OA34819, we bumped the region size up in our jobs (some are using 64M already, others 32M). This worked fine until I added the override to the LE Options for heap size when I went to Co:Z 2.0.1 and ran the a test job in our production LPARs. So I am thinking - if we have already taken one of the suggested actions to overcome the issue that OA34819 addressed by bumping up our region size to no longer get the 'FOTS2050 xrealloc: out of memory' messages - does this mean that I would not need to have the LE option override in cozsftp_config, even though I do not have OA37278 applied?
We are at Ported Tools 1.2 but do not yet have OA37278 applied. I had looked at OA37278 in the past and saw that it was supposed to fix some of the issues addressed by OA34819, but it didn't go into any detail, so I wasn't sure what it did to fix the problem.
When I hit the problem that is reported in OA34819, we bumped the region size up in our jobs (some are using 64M already, others 32M). This worked fine until I added the override to the LE Options for heap size when I went to Co:Z 2.0.1 and ran the a test job in our production LPARs. So I am thinking - if we have already taken one of the suggested actions to overcome the issue that OA34819 addressed by bumping up our region size to no longer get the 'FOTS2050 xrealloc: out of memory' messages - does this mean that I would not need to have the LE option override in cozsftp_config, even though I do not have OA37278 applied?
Re: Getting abend 4093 code=1c
So you are running PT 1.2 without OA37278 applied with REGION=64MB, and ssh-le-options=HEAP(12,1M,,FREE) and still getting an LE abend 4093 return code 1C?
First, you need to capture this dump and see what program it is for. IBM's comment that there is more than one LE environment is true. The IBM Ported Tools ssh program does a fork/exec of several child processes of its own, including things like your _SSH_ASKPASS program (if any), ssh-rand-helper, etc. etc. If the ABEND is for COZBATCH or cozsftp, please let us know and we will arrange to get the dump to look at. But I suspect that that is not the case.
If the ABEND is in IBM's program, then you should forward to them to analyze. You could try removing the HEAP option from ssh-le-options to see if that helps. This was our attempt to avoid the OA34819 problem (xrealloc fails). Removing this option will very likely allow you to avoid these ABEND4093/1C. Since you already have REGION=64MB, then you will probably also avoid the xrealloc problem anyway. If you do get the xrealloc problem, then it will clearly be IBM's problem again - you will have followed the suggestions in OA34819 of running with REGION=64MB.
First, you need to capture this dump and see what program it is for. IBM's comment that there is more than one LE environment is true. The IBM Ported Tools ssh program does a fork/exec of several child processes of its own, including things like your _SSH_ASKPASS program (if any), ssh-rand-helper, etc. etc. If the ABEND is for COZBATCH or cozsftp, please let us know and we will arrange to get the dump to look at. But I suspect that that is not the case.
If the ABEND is in IBM's program, then you should forward to them to analyze. You could try removing the HEAP option from ssh-le-options to see if that helps. This was our attempt to avoid the OA34819 problem (xrealloc fails). Removing this option will very likely allow you to avoid these ABEND4093/1C. Since you already have REGION=64MB, then you will probably also avoid the xrealloc problem anyway. If you do get the xrealloc problem, then it will clearly be IBM's problem again - you will have followed the suggestions in OA34819 of running with REGION=64MB.
Re: Getting abend 4093 code=1c
I apologize, I don't think I made myself clear. I am the only one who has tested this so far, so I was running a COZBATCH job that had a region of 32M and got the 4093-1c abend. Bumping the region up to 64M resolved it for me. We have some jobs that are already running 64M to bypass the xrealloc issue and I do not know if they would have the 4093-1C problem or not since they have not been tested with this change.
Your last paragraph pretty much answered my last question though. Since we have bypassed the xrealloc problem already with the region size option, we should not need the HEAP LE option override. Without that option we do not get the 4093-1C, and at this point I am not seeing the xrealloc issue.
So I believe that you have addressed my concerns. Since we have already bypassed the xrealloc problem with larger region sizes for those batch jobs experiencing the problem, we do not need the ssh-le-options HEAP override. Since that override causes abends and is not needed, it will be removed from the cozsftp_config member. If we see any xrealloc problems that cannot be resolved by the larger region size, we will address that issue to IBM.
Thank you very much for your help with this.
Your last paragraph pretty much answered my last question though. Since we have bypassed the xrealloc problem already with the region size option, we should not need the HEAP LE option override. Without that option we do not get the 4093-1C, and at this point I am not seeing the xrealloc issue.
So I believe that you have addressed my concerns. Since we have already bypassed the xrealloc problem with larger region sizes for those batch jobs experiencing the problem, we do not need the ssh-le-options HEAP override. Since that override causes abends and is not needed, it will be removed from the cozsftp_config member. If we see any xrealloc problems that cannot be resolved by the larger region size, we will address that issue to IBM.
Thank you very much for your help with this.
Re: Getting abend 4093 code=1c
Hello Good People - I recently inherited a z/OS v2.3 LPAR that just does INBOUND SFTP only. Back in Feb/Mar 2019 time frame before I arrived they upgraded from z/OS v2.2 to v2.3 and these U4039,reason code=1c sftp abends started to appear. They run hundreds of SFTPs a day but 1-3 fail usually in the early AM.
+++++NFOS0071 STARTED 04.15.00 267666-2-266040**
IEA995I SYMPTOM DUMP OUTPUT 913
USER COMPLETION CODE=4093 REASON CODE=0000001C
TIME=04.15.00 SEQ=11580 CPU=0000 ASID=0079
PSW AT TIME OF ERROR 078D1000 8B560BB6 ILC 2 INTC 0D
NO ACTIVE MODULE FOUND
NAME=UNKNOWN
DATA AT PSW 0B560BB0 - 00181610 0A0DA7F4 001C1811
AR/GR 0: 00000000/84000000 1: 00000000/84000FFD
2: 00000000/0000001C 3: 00000000/00000001
4: 00000000/0001D41C 5: 00000000/0001CF585
6: 00000000/0001C358 7: 00000000/0B560B7A
8: 00000000/80000000 9: 00000000/00005000
A: 00000000/00000001 B: 00000000/8B560AD8
C: 00000000/10019018 D: 00000000/10015538
E: 00000000/8001404E F: 00000001/0000001C
END OF SYMPTOM DUMP
BPXP018I THREAD 1088780000000001, IN PROCESS 66097, ENDED 914
WITHOUT BEING UNDUBBED WITH COMPLETION CODE 84000FFD
, AND REASON CODE 0000001C.
IEF450I NFOS0071 *OMVSEX - ABEND=S000 U4093 REASON=0000001C 915
Was there any critical 'show stopper' upgrade changes or DOC reviewed that might need to be applied for V2.3?
The SSHD REGION size is set to REGION=0M on the jobcard.
I cant find anywhere that the cozsftp_config has this coded - should it be added?
ssh-le-options=HEAP(12M,1M,,FREE),ENVAR("_CEE_REALLOC_CONTROL=256K,25")
Thanks Mucho!
+++++NFOS0071 STARTED 04.15.00 267666-2-266040**
IEA995I SYMPTOM DUMP OUTPUT 913
USER COMPLETION CODE=4093 REASON CODE=0000001C
TIME=04.15.00 SEQ=11580 CPU=0000 ASID=0079
PSW AT TIME OF ERROR 078D1000 8B560BB6 ILC 2 INTC 0D
NO ACTIVE MODULE FOUND
NAME=UNKNOWN
DATA AT PSW 0B560BB0 - 00181610 0A0DA7F4 001C1811
AR/GR 0: 00000000/84000000 1: 00000000/84000FFD
2: 00000000/0000001C 3: 00000000/00000001
4: 00000000/0001D41C 5: 00000000/0001CF585
6: 00000000/0001C358 7: 00000000/0B560B7A
8: 00000000/80000000 9: 00000000/00005000
A: 00000000/00000001 B: 00000000/8B560AD8
C: 00000000/10019018 D: 00000000/10015538
E: 00000000/8001404E F: 00000001/0000001C
END OF SYMPTOM DUMP
BPXP018I THREAD 1088780000000001, IN PROCESS 66097, ENDED 914
WITHOUT BEING UNDUBBED WITH COMPLETION CODE 84000FFD
, AND REASON CODE 0000001C.
IEF450I NFOS0071 *OMVSEX - ABEND=S000 U4093 REASON=0000001C 915
Was there any critical 'show stopper' upgrade changes or DOC reviewed that might need to be applied for V2.3?
The SSHD REGION size is set to REGION=0M on the jobcard.
I cant find anywhere that the cozsftp_config has this coded - should it be added?
ssh-le-options=HEAP(12M,1M,,FREE),ENVAR("_CEE_REALLOC_CONTROL=256K,25")
Thanks Mucho!