ICSF and COZSFTP

[njd]

We want to enable the option of using ICSF with COZSFTP. From the manual I can see that we have to add the following lines
to /etc/ssh/zos_ssh_config and /etc/ssh/zos_sshd_config:

CiphersSource any
MACsSource any

and we should also define the following RACF resources/access (assuming these aren't already in place)

RDEFINE CSFIQA CLASS(CSFSERV) UACC(NONE)
RDEFINE CSF1TRC CLASS(CSFSERV) UACC(NONE)
RDEFINE CSF1TRD CLASS(CSFSERV) UACC(NONE)
RDEFINE CSF1SKE CLASS(CSFSERV) UACC(NONE)
RDEFINE CSF1SKD CLASS(CSFSERV) UACC(NONE)
RDEFINE CSFOWH CLASS(CSFSERV) UACC(NONE)
PERMIT CSFIQA CLASS(CSFSERV) ID(*) ACCESS(READ)
PERMIT CSF1TRC CLASS(CSFSERV) ID(*) ACCESS(READ)
PERMIT CSF1TRD CLASS(CSFSERV) ID(*) ACCESS(READ)
PERMIT CSF1SKE CLASS(CSFSERV) ID(*) ACCESS(READ)
PERMIT CSF1SKD CLASS(CSFSERV) ID(*) ACCESS(READ)
PERMIT CSFOWH CLASS(CSFSERV) ID(*) ACCESS(READ)
SETROPTS CLASSACT(CSFSERV)
SETROPTS RACLIST(CSFSERV) REFRESH

Is that all that is required? Could someone also clarify what ICSF would be used for ? Is it the encryption of the file being transferred?
I assume without ICSF fully configured the encryption would take place in the SSHD software?

[dovetail]

There are setup guides for Ported Tools OpenSSH that cover everything here:

http://dovetail.com/docs/pt-quick-inst-12/index.html (P.T. 1.2)
http://dovetail.com/docs/pt-quick-inst/index.html (P.T. 1.3)

[njd]

I assume you are referring to sections 1.6 and 2.1 - 2.3?

We have been running some tests using "export COZ_LOG=D" to compare the cpu and initially this was showing double the cpu usage under ICSF as it was without ICSF.

After adding the following to /etc/ssh/sshd_config

# Only support ICSF/CPACF SHA-1 MACs:
MACs hmac-sha1,hmac-sha1-96

and the following to /etc/ssh/ssh_config

# 2) Prefer AES ICSF/CPACF Ciphers, but fallback to others
Ciphers aes128-cbc,aes192-cbc,aes256-cbc,aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,
arcfour128,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,rijndael-cbc@lysator.liu.se

we see the CPU come down to just below what was used when not using ICSF. Should we not see this being 50% less CPU than when we run without ICSF? Also, by limiting the MACs to only hmac-sha1 and hmac-sha1-96 and the Ciphers as above are we likely to break anything?

[dovetail]

Most customers do see around a 50% reduction in CPU when switching to ICSF ciphers and macs.
I can't really tell from your post what you are comparing (which Ciphers and Macs you are actually using in the ICSF vs. non-ICSF CPU comparison).

You may want to consult IBM if you are having CPU utilization issues with ICSF use with IBM Ported Tools OpenSSH.
We have heard of overhead problems having to do with ICSF calls to RACF (or other security product) that I believe were found to be a RACF problem.

One thing to try (if you are at the A1 version of ICSF) is to disable individual RACF checks for MACs. This is covered in our Quick Start guides.

RDEFINE CSF.CSFSERV.AUTH.CSFOWH.DISABLE
CLASS(XFACILIT) UACC(READ)
RDEFINE CSF.CSFSERV.AUTH.CSFRNG.DISABLE
CLASS(XFACILIT) UACC(READ)
SETROPTS CLASSACT(XFACILIT)
SETROPTS RACLIST(XFACILIT) REFRESH

I can't say whether you will break anything by only using SHA-1 MACs. I would think that it is extremely unlikely, since SHA-1 is the most common and required in a SSH implementation. Nevertheless, it is theoretically possible that one of your connection partners doesn't support SHA-1.

[dovetail]

I forgot to point something out when looking at Co:Z SFTP CPU costs.

When running Co:Z SFTP, there is a separate OMVS address space that runs either ssh or sshd for the connection. This address space is where all of the encryption occurs and if ICSF is not used it will generally account for over 90% of the overall CPU costs. You would need to look at the SMF30 records to see the account for this, since it will not show up in the batch job account for running the Co:Z SFTP client.

[njd]

Can I ask what I would have to define in the Ciphers and Macs parameters to run in the most CPU efficient way and not disable anything?
I assume there is a complete list of Ciphers and Macs that COZSFTP supports and a preferred order. I assume by just coding "MACs hmac-sha1,hmac-sha1-96 " as recommended in the manual certain MACs would not be available for use.

[dovetail]

In our quickstart guide, I refer you to this:

http://dovetail.com/docs/pt-quick-inst- ... -icsf.html

Cipher and MAC negotiation rule
The first algorithm in the client list that appears anywhere in the server list will be selected.

Therefore, in the ssh client's Ciphers and Macs list, you can simply move the ICSF enabled ciphers (SHA-1) to the front of the list. These (the ICSF enabled Ciphers and Macs) will then be selected as long as the server supports them.

There is no perfect solution for configuring the SSHD server. Here are your options from the QuickStart guide:

Configuring SSHD server Ciphers and MACs

The negotiation rule implies that your have fewer choices for selecting Ciphers and MACs in your SSHD server configuration. Generally, there are three strategies:

1. Only allow ICSF/CPACF supported Ciphers and fail otherwise
2. Allow any Cipher; try to ask your clients to prefer ICSF/CPACF supported Ciphers
3. Allow a subset of the supported Ciphers as required by your clients

For complete reference information. see the IBM z/OS OpenSSH User's Guide.

[Rock73]

we see the CPU come down to just below what was used when not using ICSF. Should we not see this being 50% less CPU than when we run without ICSF? Also, by limiting the MACs to only hmac-sha1 and hmac-sha1-96 and the Ciphers as above are we likely to break anything?

[dovetail]

Take a look at "Verifying ICSF Usage" here: https://dovetail.com/docs/pt-quick-inst ... erify-icsf

1) check that you are using an ICSF enabled Cipher and Mac (which ones?)

2) You can compare quickly by using these options on your cozsftp command line:

-oCiphersSource=OpenSSL -oMacsSource=OpenSSL

-oCiphersSource=ICSF -oMacsSource=ICSF

3) add this to your script before running the cozsftp command line:

export COZ_LOG=D

this will enable Debug level messages, and most importantly you should see a message like this:

... SSH process times: elapsed=... secs, user cpu=... secs, sys cpu=... secs

[njd]

<t>We've started to revisit this to see if we can save some CPU.<br/>
<br/>
However, we appear to be seeing some odd results. I am wondering if this is what others are seeing?<br/>
<br/>
When we run a large COZSFTP loop back transfer job with the following options and "export COZ_LOG=D" <br/>
<br/>
-oCiphersSource=OpenSSL<br/>
-oMacsSource=OpenSSL<br/>
<br/>
We see this:<br/>
<br/>
SSH process times: elapsed=268 secs, user cpu=47.210000 secs, sys cpu=15.730000<br/>
<br/>
Rerunning exactly the same job, replacing the above options with these:<br/>
<br/>
-oCiphersSource=ICSF<br/>
-oMacsSource=ICSF <br/>
<br/>
We see this:<br/>
<br/>
SSH process times: elapsed=540 secs, user cpu=59.160000 secs, sys cpu=19.720000<br/>
<br/>
e.g. over a 20% increase in CPU by offloading to ICSF. We don't understand how this can be happening. Any thoughts?<br/>
<br/>
I see in the IBM Ported Tools manual, it claims that up to a 50% saving in CPU can be accomplished by offloading to CPACF. <br/>
Any ideas how we can ensure we are using CPACF? Is it possible that CPACF is being used by default anyway?</t>

[njd]

Reposted as format seems to have been corrupted by saving as draft initially.

We've started to revisit this to see if we can save some CPU.

However, we appear to be seeing some odd results. I am wondering if this is what others are seeing?

When we run a large COZSFTP loop back transfer job with the following options and "export COZ_LOG=D"

-oCiphersSource=OpenSSL
-oMacsSource=OpenSSL

We see this:

SSH process times: elapsed=268 secs, user cpu=47.210000 secs, sys cpu=15.730000
Rerunning exactly the same job, replacing the above options with these:

-oCiphersSource=ICSF
-oMacsSource=ICSF

We see this:

SSH process times: elapsed=540 secs, user cpu=59.160000 secs, sys cpu=19.720000

e.g. over a 20% increase in CPU by offloading to ICSF. We don't understand how this can be happening. Any thoughts?

I see in the IBM Ported Tools manual, it claims that up to a 50% saving in CPU can be accomplished by offloading to CPACF.

Any ideas how we can ensure we are using CPACF? Is it possible that CPACF is being used by default anyway?

[dovetail]

What you are seeing is not typical, but having said that we have seen a couple of reports of performance problems like this with ICSF in IBM z/OS OpenSSH.
You should open a PMR with IBM, since this is happening in IBM z/OS OpenSSH, and has nothing to do with Co:Z SFTP.

Note:
In one case, a user reported relieve by disabling RACF checks for hash functions, using:

CSF.CSFSERV.AUTH.CSFOWH.DISABLE

This would seem to point to a system overhead/performance problem with ICSF SAF/RACF calls or RACF itself (or another ESM).

[njd]

Adding RACF profile CSF.CSFSERV.AUTH.CSFOWH.DISABLE appears to have made a huge difference. We now seem to be seeing at least a 30% drop in CPU when using CPACF with this profile defined.

Any ideas how I can ensure that we have all ciphers and macs defined in /etc/ssh/ssh_config and /etc/ssh/sshd_config

Also do you have the complete list of all cipers and macs that can run in CPACF ? Or should I ask IBM?

[njd]

We are seeing some odd results with ICSF acceleration. In our test systems we do see a noticeable drop in CPU usage. However, in development on a different LPAR we don't, in fact the CPU usage seems to be 10% higher using ICSF compared to OpenSSL.
Other than the RACF profiles you mention any thoughts on what could be causing this? Should we be expecting a 50% decrease when using ICSF CPACF acceleration compared to using OpenSSL defaults?