IBM z/OS OpenSSH V2R2/V2R3 & APAR OA54299

[njd]

The following looks very interesting:
• With the PTF for APAR OA54299 installed, IBM z/OS OpenSSH V2R2/V2R3 can directly use CPACF instructions, when present, to implement cipher and MAC algorithms. This can significantly reduce CPU usage compared to CPACF via ICSF.

Would this work for any release of COZSFTP? We’ve enabled CPACF via ICSF and not really seen any real reduction in CPU. So I’m wondering whether the above could make it viable for us. Do you have idea how significant a CPU reduction this would be?

[dovetail]

Previously, a large majority of customers that used ICSF+CPACF for supported algorithms (aes, sha, etc) see a 50% or so reduction in CPU when compared to OpenSSL (pure software) for the common aes*-cbc + hmac-sha1 combination. However, we have encountered several customers that reported no benefit. In some cases we have found that the problem seemed to be related to the *many* SAF/RACF check calls made by ICSF. I don't believe that IBM knows why this performance problem occurs in some environments and not in others.

With this new support for using CPACF instructions directly from OpenSSH (without ICSF), we have seen a 15% or more reduction in CPU on top of the 50% reduction seen by the customers who were successful with ICSF+CPACF in the past. Also, aes*-ctr mode is now just as fast as cbc mode, where it was significantly slower using ICSF+CPAF.

For the customers who had performance issues with ICSF+CPACF - you should see a big improvement with this new PTF.
For those where ICSF+CPACF worked OK, this is even better; especially for aes*-ctr mode.

Please note that the crypto algorithms are done in IBM OpenSSH code and not Co:Z, so please consider contacting IBM for z/OS OpenSSH support. If you are a Co:Z support customer, we encourage you to open a support ticket with us and we will be happy to assist with z/OS OpenSSH questions and troubleshooting.

[njd]

Hi, many thanks for the information. I can confirm that we are now seeing considerable CPU reduction with APAR OA54299 applied. I see in the manual that it is no longer necessary to code CiphersSource and MACsSource in /etc/ssh/zos_ssh_config and /etc/ssh/zos_sshd_config . That certainly seems to match what I see in my testing. CPACF appears to be always chosen for Ciphers.

I'm less clear whether we need to code Macs and Ciphers in /etc/ssh/ssh_config and /etc/ssh/sshd_config. The only difference I can see is that the default Mac would be taken from OpenSSL rather than CPACF if I don't code a Mac statement with CPACF eligible Macs at the beginning of the statement. However, I don't see much difference in CPU usage between using Mac hmac-sha1-etm@openssh.com from CPACF or hmac-md5-etm@openssh.com from OpenSSL. Any thoughts on the best configuration for reduced CPU usage when coding /etc/ssh/ssh_config and /etc/ssh/sshd_config would certainly be appreciated.

[dovetail]

Take a look at CH 2 in our new Quick Install Guide for OpenSSH V2R2+ with this PTF.

https://dovetail.com/docs/pt-quick-inst ... cpacf.html

Please let me know if this answers your questions or is confusing since I would like to get this guide right.

BTW: the "-etm" MAC variants only affect the order that MACs/Ciphers are done in the packets, not the algorithm itself.
Also, MD5 is not one that you want to use since it is much weaker than sha-1 and not CPACF enabled and therefore much slower.
Some people are concerned about recent vulnerabilities in sha-1, but the hmac-sha1 used by OpenSSH is more than fine IMO for the purpose of a packet checksum. Using plain sha-1 in a long-lasting certificate is where one might be worried about cracks.

[njd]

I think the bit that is slightly confusing is this "In addition, configuration changes and ICSF RACF/SAF rules are not necessary. We recommend that most users will want to use this approach, which is now the default. "

I guess that refers to files /etc/ssh/zos_ssh_config and /etc/ssh/zos_sshd_config and not /etc/ssh/ssh_config and /etc/ssh/sshd_config .

So in summary I think you are saying that we no longer need the CiphersSource and MACsSource in /etc/ssh/zos_ssh_config and /etc/ssh/zos_sshd_config , but it's still a good idea to code Cipher and Mac statements in /etc/ssh/ssh_config and /etc/ssh/sshd_config .
Particularly in the case of Macs as MD5, the first default Mac, is not a good choice because it's not CPACF eligible and not very secure.

[dovetail]

Remember that there are new "default" configuration files shipped by IBM in /samples with this new PTF.
If you look at these, I think you will find that MD5 MACs are not at the beginning of ssh_config.

Here is what you should see in the sample ssh_config:

# The defaults above are reordered below to prefer ICSF HW accelerated algorithms, AES over 3DES, and SHA over MD5
Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,3des-cbc,aes256-gcm@openssh.com,aes128-gcm@openssh.com,arcfour128,arcfour256,blowfish-cbc,cast128-cbc,arcfour
MACs hmac-sha1-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-sha1,hmac-sha2-256,hmac-sha2-512,hmac-sha1-96,hmac-md5-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-md5-96,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com

[njd]

Hi, I hadn't realised the sample had changed. I don't see any Macs or Cipher statements in /samples/sshd_config though. Wouldn't I need to code something in there? Or will both client and server operations default to using ssh_config?

[dovetail]

The Ciphers and MACs in the samples/sshd_config is not explicitly specified, so it uses the list that you see commented out.
These are all of the supported Ciphers and MACs.
Remember that with the server, the order doesn't matter.

[njd]

Hi, many thanks for your advice on this, it's appreciated.

[njd]

Hi Dovetail,

To confirm after applying this APAR and changing the configuration to use CPACF ciphers and macs we are seeing considerable drops in CPU usage.
So I'd say this is an important APAR for users of COZSFTP.

[dovetail]

We did send out a notification in January on this APAR to our customers, but thanks for the feedback and confirmation.