RC=1808 when using FTP-SSH Proxy.

ebbie · Post by **ebbie** » Mon Aug 02, 2010 4:10 am

After 3 days of successful transfers using batch, i suddenly get jobs failing with a rc=1808. Within the STC i see the following:
07.01.02 STC16334 IGD101I SMS ALLOCATED TO DDNAME (SYS00005) 803
803 DSN (AUTO.JVM.TDUMP.TCPSOCK5.D100801.T070020 )
803 STORCLAS (WORK) MGMTCLAS (WORK) DATACLAS ( )
803 VOL SER NOS= WRS235
07.01.40 STC16334 IGD104I AUTO.JVM.TDUMP.TCPSOCK5.D100801.T070020 RETAINED

I also see the messages below in the log:
JVMST109: Insufficient space in Javaheap to satisfy allocation request
JVMDG217: Dump Handler is Processing OutOfMemory - Please Wait.
JVMHP002: JVM requesting System Transaction Dump
JVMHP012: System Transaction Dump written to AUTO.JVM.TDUMP.TCPSOCK5.D100801.T07

Looks like a JAVA type problem. Assistance will be appreciated.

Post by **dovetail** » Mon Aug 02, 2010 10:05 am

This indicates that Java ran out of memory.

If you run the JZOS batch launcher with +D and ad "-d" to the ARGS (either before or after the -h switch), you can get trace information that might help us to see what is happening.

ebbie · Post by **ebbie** » Thu Aug 12, 2010 7:23 am

Im not sure where to specify the +D on the JZOS batch launcher. I am using FTP-SSH Proxy in batch mode to send out files from the mainframe.

Post by **dovetail** » Thu Aug 12, 2010 9:12 am

(starting from the sample JCL)

Code: Select all

//JAVA EXEC PROC=JVMPRCxx,   <-- the IBM SDK JZOS proc 
// JAVACLS='com.dovetail.ftpsshproxy.ProxyServer',
// LOGLVL='+D', 
// ARGS='-d -h 127.0.0.1:1080' <- listen on loopback interface, port 1080

LOGLVL='+D' is the log setting for the JZOS Batch launcher proc.
The above also adds the "-d" argument to the ProxyServer, which will turn on debug logging in the Java code.

ebbie · Post by **ebbie** » Wed Oct 13, 2010 9:43 am

Below is a portion of the trace and log output. Let me know if you need more.

2010-10-13 13:11:29,699 INFO ProxyConnection[/10.33.255.36:47192] - control connection closed
2010-10-13 13:47:09,547 INFO ProxyServer - shutting down...
2010-10-13 13:47:09,653 INFO ProxyServer - completed; server socket closed

Exception in thread "Thread-4837" java.lang.OutOfMemoryError
at com.jcraft.jsch.Buffer.<init>(Buffer.java:38)
at com.jcraft.jsch.Channel$1.init(Channel.java:273)
at com.jcraft.jsch.Channel$1.write(Channel.java:291)
at sun.nio.cs.StreamEncoder$ConverterSE.implFlushBuffer(StreamEncoder.java:285)
at sun.nio.cs.StreamEncoder$ConverterSE.implFlush(StreamEncoder.java:291)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:202)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:236)
at com.dovetail.ftpsshproxy.FtpControlSession.serverWriteCmd(FtpControlSession.java:398)
at com.dovetail.ftpsshproxy.FtpControlSession.processServerFtpLogin(FtpControlSession.java:196)
at com.dovetail.ftpsshproxy.FtpControlSession.startConnection(FtpControlSession.java:131)
at com.dovetail.ftpsshproxy.FtpControlSession.run(FtpControlSession.java:76)
at com.dovetail.ftpsshproxy.ProxyConnection.runProxySession(ProxyConnection.java:135)
at com.dovetail.ftpsshproxy.ProxyConnection.doRun(ProxyConnection.java:91)
at com.dovetail.ftpsshproxy.ProxyConnection.run(ProxyConnection.java:72)
at java.lang.Thread.run(Thread.java:811)
Exception in thread "Thread-4840" java.lang.OutOfMemoryError
at java.lang.StringBuffer.setLength(StringBuffer.java:834)
at org.apache.log4j.spi.LocationInfo.<init>(LocationInfo.java:107)
at org.apache.log4j.spi.LoggingEvent.getLocationInformation(LoggingEvent.java:191)
at org.apache.log4j.helpers.PatternParser$ClassNamePatternConverter.getFullyQualifiedName(PatternParser.java:538)
at org.apache.log4j.helpers.PatternParser$NamedPatternConverter.convert(PatternParser.java:511)
at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:64)
at org.apache.log4j.PatternLayout.format(PatternLayout.java:503)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:301)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:159)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
at org.apache.log4j.Category.callAppenders(Category.java:203)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.debug(Category.java:257)
at com.dovetail.ftpsshproxy.FtpControlSession.serverReadResponse(FtpControlSession.java:408)
at com.dovetail.ftpsshproxy.FtpControlSession.processServerFtpLogin(FtpControlSession.java:195)
at com.dovetail.ftpsshproxy.FtpControlSession.startConnection(FtpControlSession.java:131)
at com.dovetail.ftpsshproxy.FtpControlSession.run(FtpControlSession.java:76)
at com.dovetail.ftpsshproxy.ProxyConnection.runProxySession(ProxyConnection.java:135)
at com.dovetail.ftpsshproxy.ProxyConnection.doRun(ProxyConnection.java:91)
at com.dovetail.ftpsshproxy.ProxyConnection.run(ProxyConnection.java:72)
at java.lang.Thread.run(Thread.java:811)
Exception in thread "Connect thread 10.33.0.40 session" java.lang.OutOfMemoryError
at com.ibm.crypto.provider.AESCipher.engineUpdate(Unknown Source)
at javax.crypto.Cipher.update(Unknown Source)
at com.jcraft.jsch.jce.AES128CBC.update(AES128CBC.java:69)
at com.jcraft.jsch.Session.read(Session.java:812)
at com.jcraft.jsch.Session.run(Session.java:1177)
at java.lang.Thread.run(Thread.java:811)
Exception in thread "Thread-4843" java.lang.OutOfMemoryError
at java.lang.StringBuffer.<init>(StringBuffer.java:65)
at java.lang.StackTraceElement.toString(StackTraceElement.java:131)
at java.lang.String.valueOf(String.java:1505)
at java.lang.StringBuilder.append(StringBuilder.java:194)
at java.lang.Throwable.printStackTrace(Throwable.java:248)
at org.apache.log4j.spi.LocationInfo.<init>(LocationInfo.java:105)
at org.apache.log4j.spi.LoggingEvent.getLocationInformation(LoggingEvent.java:191)
at org.apache.log4j.helpers.PatternParser$ClassNamePatternConverter.getFullyQualifiedName(PatternParser.java:538)
at org.apache.log4j.helpers.PatternParser$NamedPatternConverter.convert(PatternParser.java:511)
at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:64)
at org.apache.log4j.PatternLayout.format(PatternLayout.java:503)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:301)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:159)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
at org.apache.log4j.Category.callAppenders(Category.java:203)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.info(Category.java:663)
at com.dovetail.ftpsshproxy.FtpControlSession.startSshTunnel(FtpControlSession.java:242)
at com.dovetail.ftpsshproxy.FtpControlSession.startConnection(FtpControlSession.java:129)
at com.dovetail.ftpsshproxy.FtpControlSession.run(FtpControlSession.java:76)
at com.dovetail.ftpsshproxy.ProxyConnection.runProxySession(ProxyConnection.java:135)
at com.dovetail.ftpsshproxy.ProxyConnection.doRun(ProxyConnection.java:91)
at com.dovetail.ftpsshproxy.ProxyConnection.run(ProxyConnection.java:72)
at java.lang.Thread.run(Thread.java:811)
Exception in thread "Connect thread 10.33.0.40 session" java.lang.OutOfMemoryError
at com.ibm.crypto.provider.AESCipher.engineUpdate(Unknown Source)
at javax.crypto.Cipher.update(Unknown Source)
at com.jcraft.jsch.jce.AES128CBC.update(AES128CBC.java:69)
at com.jcraft.jsch.Session.read(Session.java:812)
at com.jcraft.jsch.Session.run(Session.java:1177)
at java.lang.Thread.run(Thread.java:811)
Exception in thread "Thread-4846" java.lang.OutOfMemoryError
at java.lang.J9VMInternals.getStackTrace(Native Method)
at java.lang.Throwable.getInternalStackTrace(Throwable.java:198)
at java.lang.Throwable.printStackTrace(Throwable.java:246)
at org.apache.log4j.spi.LocationInfo.<init>(LocationInfo.java:105)
at org.apache.log4j.spi.LoggingEvent.getLocationInformation(LoggingEvent.java:191)
at org.apache.log4j.helpers.PatternParser$ClassNamePatternConverter.getFullyQualifiedName(PatternParser.java:538)
at org.apache.log4j.helpers.PatternParser$NamedPatternConverter.convert(PatternParser.java:511)
at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:64)
at org.apache.log4j.PatternLayout.format(PatternLayout.java:503)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:301)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:159)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
at org.apache.log4j.Category.callAppenders(Category.java:203)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.info(Category.java:663)
at com.dovetail.ftpsshproxy.FtpControlSession.startSshTunnel(FtpControlSession.java:242)
at com.dovetail.ftpsshproxy.FtpControlSession.startConnection(FtpControlSession.java:129)
at com.dovetail.ftpsshproxy.FtpControlSession.run(FtpControlSession.java:76)
at com.dovetail.ftpsshproxy.ProxyConnection.runProxySession(ProxyConnection.java:135)
at com.dovetail.ftpsshproxy.ProxyConnection.doRun(ProxyConnection.java:91)
at com.dovetail.ftpsshproxy.ProxyConnection.run(ProxyConnection.java:72)
at java.lang.Thread.run(Thread.java:811)

ebbie · Post by **ebbie** » Fri Dec 03, 2010 4:37 am

Hi

Can anyone help me with the trace output i placed in my previous email? The STC continually goes into this state every 3 days roughly. Help will be appreciated.

Post by **coz** » Fri Dec 03, 2010 2:42 pm

In order to help, we will need to see the javacore and heapdump that get generated when this occurs.

By default, both of these files will be in the current working directory of the JVM process and have the following names:

javacore.%Y%m%d.%H%M%S.%pid.txt
heapdump.%Y%m%d.%H%M%S.%pid.phd

In addition, please run the launcher with the +T log option (rather than the +D described earlier).

Once you have this information collected, please send an email to info@dovetail.com with the launcher trace output attached (We'll need the entire trace).

Once we receive that, we'll give you instructions to upload the two dumps to our FTP server.

Post by **dovetail** » Tue Dec 21, 2010 10:28 am

We received the dumps that you sent, and we have found that there was a memory leak, so this would explain how you were running out of memory after running for several days with many sessions.

Please download and verify that the new release (1.0.1) fixes your problem:
http://dovetail.com/downloads/coz/index.html
(see the right hand side of the page, near the top)

ebbie · Post by **ebbie** » Tue Jun 07, 2011 4:29 am

All went well since i installed as you directed.
Lately this issue has reared it s head again. I have the java and heapdumps if you need to check them.
We are running COZ 175 on our live systems.

Post by **coz** » Fri Jun 10, 2011 8:49 am

Yes, please send the new dump files to info@dovetail.com. How is this related to Co:Z 1.7.5? The FTP-SSH-Proxy is a separate download.

We are swamped right now, but will get this as time permits.

ebbie · Post by **ebbie** » Fri Jun 10, 2011 9:00 am

Sorry. We using SSHProxy version 1.0.1. Will email the dumps through to you.