Usually we launch hundreds of instances in AWS for day to day work. As long as they are accessible from our hosts (probably a RHEL or Ubuntu or your own mac), we are good to go. But there are some instances where you might get a patch from IT for your host. Once you apply the patch, you realize that you are unable to access your AWS instances anymore. And your IT team doesn't have any clue on what happened. You contact AWS support, and they say it all looks good. So how do you proceed from this scenario? Where to start and what to do. This talk goes through all the steps starting with most basic checks all the way to updating the crypto key exchange algorithms on your host.

1. Who broke my crypto
Nikhil Prathapani
Enterprise Routing – SDWAN group
Cisco Systems Inc

2. Things you WON’T learn in this talk
• Crypto currency
• Bitcoin
• Blockchain

3. Agenda
• Chapter 0 – The Problem
• Chapter 1 – The Puzzle
• Chapter 2 – The Chase
• Chapter 3 – The Eureka
• Chapter 4 – The End

4. On a fine Monday morning:
I tried to SSH to my EC2 instance, but it kept bailing out on me.
Chapter 0 – The Problem

5. • Why am I unable to SSH to an instance which worked fine until Friday.
• I listed out the things changed from my end:
• EC2 Instance type: unchanged, not even touched since Friday
• Host machine : Same host machine - RHEL instance
Chapter 1 – The Puzzle

6. Oh I know how to debug this.
Its simple:
• Instead of SSH, just add –vvv for further debug.
• ssh -v will tell you what is happening mostly on your end
• ssh -vv will tell you low level on both ends
• ssh -vvv will tell you almost everything from both ends.
Chapter 2 – The Chase

7. Contacted AWS support.
A very patient support rep helped me debug the issue further
Step 0: SSH with "-vvv" flag for verbosity
I did that, didn’t help. Still lost connection.

8. Step 1: Create packet capture
$ sudo -i
# tcpdump -i any -w /tmp/$(hostname)_capturefile.cap -s256 port 22 &
# ssh -vvv user@<elastic_ip_ec2_instance>
# killall tcpdump ; pkill tcpdump
# zip -9 /tmp/$(hostname)_capturefile.cap.zip /tmp/$(hostname)_capturefile.cap

9. Step 2: Perform TCP traceroute over different ports, such as 22 and 443
$ mtr -c 50 --no-dns --show-ips --report-wide --report --tcp --port 443 <elastic_Ip>
$ mtr -c 50 --no-dns --show-ips --report-wide --report --tcp --port 22 <elastic_Ip>
$ tcptraceroute <elastic_Ip> 22
$ traceroute -T -p 22 –n <elastic_Ip>
To install tcptraceroute:
# yum -y install --enablerepo='*' tcptraceroute telnet
# apt install tcptraceroute # On Ubuntu

10. And that didn’t help either.
Okay, Let’s take a step back and check my email.
“IT has upgraded your VM from RHEL6 to RHEL8 over the weekend.
Please open a support case with us in case you are facing issues”.
Check the host machine:
Vm>lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterprise
Description: Red Hat Enterprise Linux release 8.1 (Ootpa)
Release: 8.1
Codename: Ootpa
"ootpa" is IRC nick of Larry
Troan, who was a Red Hat
engineer and who died in
2016.
RHEL 8 "ootpa" codename
was chosen as a tribute to
Larry Troan.

11. Great, something has changed wrt host machine, but what exactly.
<Few days pass by>
How does SSH work behind the scenes?
<opens textbook>
Information Security: Principles and Practice, Mark Stamp

13. <Search google for Red Hat documentation>
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/securing_networks/using-secure-
communications-between-two-systems-with-openssh_securing-networks
• Two versions of SSH currently exist: version 1, and the newer version 2.
• The OpenSSH suite in Red Hat Enterprise Linux 8 supports only SSH version
2, which has an enhanced key-exchange algorithm not vulnerable to known
exploits in version 1.
• OpenSSH is a program depending on OpenSSL the library, specifically
OpenSSH uses the libcrypto part of OpenSSL.
Chapter 3 – The Eureka

14. man ssh_config: (on RHEL8)
The supported ciphers are:
• 3des-cbc
• aes128-cbc
• aes192-cbc
• aes256-cbc
• aes128-ctr
• aes192-ctr
• aes256-ctr
• aes128-gcm@openssh.com
• aes256-gcm@openssh.com
• chacha20-poly1305@openssh.com

15. <deep google search for redhat issues>
• “GCM ciphers are not available in SSH on RHEL 7.4 in FIPS mode”
https://github.com/ComplianceAsCode/content/issues/1613
• GCM ciphers used to be allowed in FIPS mode, but it seems that was a
bug.
https://bugzilla.redhat.com/show_bug.cgi?id=1420910
• FIPS guide (Federal Information Processing Standards)
https://wiki.openssl.org/index.php/FIPS_mode_and_TLS

16. Go back to my host machine and look at logs:
• debug1: SSH2_MSG_KEXINIT sent
• debug1: SSH2_MSG_KEXINIT received
• debug1: kex: algorithm: ecdh-sha2-nistp256
• debug1: kex: host key algorithm: ecdsa-sha2-nistp256
• debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit>
compression: none
• debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit>
compression: none
• debug1: sending SSH2_MSG_KEX_ECDH_INIT
• debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
• Connection closed by <elastic-ip> port 22
https://www.cryptosys.net/pki/manpki/pki_aesgcmauthencryption.html

17. Go to my EC2 instance and take a look:
ec2:/etc/ssh# cat ssh_config
# Cipher 3des
# Port 22
# Protocol 2
# Cipher 3des
Ciphers aes256-gcm@openssh.com,aes128-
gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-
ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc

18. Lessons learned:
1. Issue in EC2 instance code where its defaulting to GCM ciphers.
Real bug- filed and fixed
2. Genuine Red Hat bug which accidentally blocks GCM ciphers, which
kept me hanging (still not fixed yet)
3. Simple workaround:
1. Look for any common cipher in host and EC2 instance:
For example: “AES256-CTR” is there in both places
2. Use it to SSH to the instance:
Example usage: ssh - c “AES256-CTR” user@<elastic_ip_ec2_instance>
Chapter 4 – The End

19. Happy Ending after all.

20. Questions?