ABEND S80A R=00000010

[Shady]

We installed Ported Tools 1.2.
In this version we had some errors and abends with SFTP (Ported Tools SFTP and Co:Z SFTP):

Code: Select all

fatal: FOTS2050 xrealloc: out of memory (new_size 1212416 bytes)

We found it when a college tried to upload some GBs of data into the ZFSes.
We could reproduce the error by generating arround 3000 empty files and transmit these files.

We found that it has something to do with the REGION of the STC for the SSHD. So we set REGION=0K in the STC and we haven't got the error again.
We testet Co:Z SFTP and Ported Tools SFTP with the 3k files again...
Now we get an other error...:

Code: Select all

CEE3250C The system or user abend S80A  R=00000010 was issued.                                                         
         From entry point ZosExitInterface::checkCommandWithFileArg(const char*,const char*,const char*,const char**)  
         at compile unit offset +00000F7E at entry offset +00000F7E at address 1745C526.                  

/edit: BTW...: Using Co:Z 1.7.0.
Is this may fixed in 1.7.2...?

Made the Co:Z toolkit compatible with IBM Ported Tools for z/OS version 1.2. Specifically, components in the toolkit that spawn the ssh client have been changed to spawn with a default setting of _BPX_SHAREAS=YES (the old default value was MUST). This accomodates IBM's change to ssh that requires it to run in a separate address space.

[dovetail]

This is a known problem with IBM Ported Tools.
see: http://www-01.ibm.com/support/docview.w ... sg1OA34819

Our recommendation is to change your SSHD Job:

1) Set the following LE option: HEAP(12M,1M)
2) make REGION=32M (at least)

Setting the HEAP initial size will prevent "xrealloc" from fragmenting storage, which otherwise will cause you to use MUCH more virtual storage.

If you are running z/OS 1.12, then setting _CEE_REALLOC_CONTROL as suggested in the APAR is also recommended.

[Shady]

We already set REGION=0K and the xrealloc error didn't come neigther for Ported Tools SFTP Server nor for Co:Z SFTP Server. So I mean we solved that problem and this above is an other one...?

We get this message...:

Code: Select all

CEE3250C The system or user abend S80A  R=00000010 was issued.                                                          
         From entry point ZosExitInterface::checkCommandWithFileArg(const char*,const char*,const char*,const char**)  
         at compile unit offset +00000F7E at entry offset +00000F7E at address 1745C526.                 

That just comes with the Co:Z SFTP Server. The Ported Tools SFTP Server works after changing the REGION parm.
I wanted to transmit arround 3000files and it abend after arround 1800-2200

[dovetail]

I'm sorry, I didn't read your note closely enough.

1) Even though you set the REGION=0K, without following the recommendations in IBM's APAR for SSH you will be using MUCH more virtual storage than you need to. My suggestion is that you also set HEAP=(12M,1M) on the job.

2) Just to be clear - are you getting the S80A as a separate problem? We will look into this, but may need more information. Did you get a CEEDUMP? Did the job continue?

[dovetail]

We suspect that the S80A that you are seeing is also a problem related to a storage shortage, but in the Co:Z sftp-server address space.

Even though you are asking for REGION=0M, you may not get unlimited storage if you have an installation IEFUSI or IEALIMIT exit. Without either HEAP=(,,,FREE) or preferably HEAP=(12M,1M), storage may still be exhausted in the IBM SSHD process (address space) as described in OA34819.

Do you still get the S80A abend if you set REGION=32M and PARM='HEAP(12M,1M)/ ....' ?

If so, you will need to probably set a SLIP SVC dump on the S80A and then check the LDA to see what your region size really is.

[Shady]

I'm not at work at the moment so I can't say something to IEALIMIT but I know we haven't IEFUSI active...
We will try the HEAP settings and we'll see what happen.
To your former post...: We thougt this is a seperate problem, yes. We tried the transfer with a graphical client (WinSCP) and it cancels.
Before we set the REGION=0K also the SFTP Server hangs. It doesn't end when the transfer ends (only a "kill -9 <pid>"/"kill -K <pid>" helps. This don't happen now after changing REGION.

Thanks for your replies.
We will test a bit more and I will write back if I have some more informations.

[Shady]

I've got an update....

Both exits aren't active (IEALIMIT & IEFUSI).
We also set the HEAP(12M,1M) and REGION=32M and the SFTP Server hangs again.
Curious: Sometimes the SFTP Server just cancels the session and sometimes the session is still there aftere the error. I can see it with ps -el from USS shell and it just ends with a kill -k <pid>.
Firstly I thought we haven't the hanging problem any more after changing REGION but sometimes it still hangs.
Note: Before changing the REGION from 512K to 0K/now 32M the Server hangs everytime and just a kill -K helped.

Shady

[dovetail]

Please restate your symptoms and environment.

- what version of z/OS, Ported Tools, and Co:Z SFTP are you using?
- please describe what is required to cause the hang
- what messages do you see?
- on the client
- in the z/OS syslog
- in z/OS syslogd
- do you still see the S80A error?
- if so, check the LDA in a dump to verify your ACTUAL region size

- Can you reproduce the problem with IBM's sftp-server?
- if so, the hang is more likely in SSHD. Please open a PMR with IBM
- if not, please obtain a trace of Co:Z sftp-server as described here:
http://dovetail.com/docs/sftp/sftp-server-rc.html

[Shady]

- what version of z/OS, Ported Tools, and Co:Z SFTP are you using?

Ported Tools 1.2
Co:Z SFTP 1.7.0 --> That's why I asked if may 1.7.2 helps...

- please describe what is required to cause the hang

OK.... Firstly we got the xrealloc error and we found the APAR OA34819.
We wanted to reproduce the xrealloc error and we got it with Co:Z SFTP and with Ported Tools SFTP.
To reproduce the error...:
On my client PC I created a empty file called "TEST". I marked it, copied it and pasted it so that I got a "Copy of Test". Then again: Mark all - strg+a - copy all - strg+c - paste all - strg+v
I repeated that process untill I got arround 3000 files. (During creating the files I got an error on the PC because the length of the file name exceeds so that I created TEST2 and do the mark - copy - paste thing with TEST2.) Now we testet the files with SFTP... We got the xrealloc error with Co:Z SFTP but not with Ported Tools SFTP.
Then we changed the REGION=0k.
I tested again with the 3000 created empty files...: Ported Tools SFTP works fine but with Co:Z SFTP we got the S80A error. Also the transfer of arrund 20k files worked with Ported Tools SFTP (they weren't empty).

I hope you understand what I mean...?

- what messages do you see?
- on the client

We use WinSCP and we got the following message...:
"Connection has been unexpectedly closed. Server sent command exit status 146
Copying files to remote side failed."

- in the z/OS syslog

Nothing spectaculere

- in z/OS syslogd

Just...:

Code: Select all

Connection closed by <IP>
Closing connection to <IP>

- do you still see the S80A error?

Yes.

- Can you reproduce the problem with IBM's sftp-server?

No.

[dovetail]

Thanks for this information. We will try to reproduce the problem.

One more thing: What version of WinSCP are you using?

[Shady]

I'm at home so I can't say for sure, but I think it is 4.1.3.
I'll check it tomorrow and correct it if it is wrong

[dovetail]

We were able to reproduce the problem, and found that it was a memory leak. The fix is very simple, so we plan on releasing this in our upcoming 1.7.5 release, which we expect very shortly.

Note: with REGION=0M, we were not able to reproduce the S80A when transferring 3000 empty files, so we suspect that your sftp-server is actually getting something less. You would have to get an SLIP/SVC dump and look at the LDA to confirm your actual region size.

We want to thank you very much for reporting this problem.

[Shady]

The fix is very simple, so we plan on releasing this in our upcoming 1.7.5 release, which we expect very shortly.

Looking forward to.

Note: with REGION=0M, we were not able to reproduce the S80A when transferring 3000 empty files, so we suspect that your sftp-server is actually getting something less. You would have to get an SLIP/SVC dump and look at the LDA to confirm your actual region size.

I think we will wait and test it with your new release...

We want to thank you very much for reporting this problem.

No problem. Helped everybody.

[Shady]

Hi...

We installed 1.7.5 (running zOS V1R11, Ported Tools 1.2),

I tested the file transfer with the files I allready described and we get the same error:

Code: Select all

CEE3250C The system or user abend S80A  R=00000010 was issued.
         From entry point ZosExitInterface::checkCommandWithFileArg(const char*,const char*,const char*,const char**)
         at compile unit offset +00000F9A at entry offset +00000F9A at address 17559A62.

The only thing is: We can transmit some files more than before, it'll cancel later.
If you want I can send you these files and you can test these in your environment?

/edit: Is there may a BPX address space option I don't consider...?


Greetings

[dovetail]

These are still 3000 empty files? We did test with 3000 empy files and verified that the memory leak was fixed, but there may be something else in your environment that is different.

In any case, you are welcome to email a zip file to info@dovetail.com and we will try to reproduce.

If you can obtain a dump of the failing address space, we might want to look at it also.