SwiftOnSecurity
==Training Lesson==
INVESTIGATION NARRATIVE: SSH Kill la Killed 🧵
My job is to solve the Weird Problems as the Final escalation tier. I do this with generalist knowledge and practical experience.
New InfoSec/IT entrants often ask what this looks like in practice. Follow below.
Rich Felker
@davesomebody @SwiftOnSecurity Back when I had DSL I was trying to figure out Linux networking hacks to intentionally make not just 2 but 5+ copies of each outgoing packet to ensure they got thru, since the DSL/ATM layers had no way to notify of lost packets or retransmit and no way to force more conservative handshake in upstream direction, and packet loss was over 75%. 🤬 Never found it tho.
SwiftOnSecurity
You finally get access to firewall. In the logs you find the security blocks.
A rule named "ET SCAN Potential SSH Scan OUTBOUND" which triggers on:
-- 5 outbound SSH connections
-- from the same internal IP address
-- within 120 seconds
had been set to Block by an admin trying to do right thing.
Dave C.
@SwiftOnSecurity reminds me of a time troubleshooting a slow network connection at a client. They had really outdone themselves setting up a redundant, round robin network. It had two of everything and the routers at the edges were cross strapped. Something got misconfigured and the cross strap was creating duplicate packets on the network causing everything to come crashing down when really using the connection. They kept blaming the software for this mess. Good times!
SwiftOnSecurity
(There are MULTIPLE ways to get to this eventual understanding)
For some reason the 5th SSH session is dying early. Let's filter on the stream ID to isolate this individual TCP conversation. Yes there's a SYN then two repeats. Nothing is getting established from get-go. But what about the others?
Going back to our original black lines in Wireshark, those are the 4 different existing connections trying to re-transmit an ACK saying please continue sending. But they are never hearing back. So the APPLICATION is seemingly not dying, it is somewhere else. And no TCP tear-down.
SwiftOnSecurity
What do we know so far?
1.) Initial connectivity works.
2.) Connectivity with up to 4 sessions seems to work.
3.) The 5th connection doesn't work.
4.) All connectivity to server stops functioning at this point.
This smells like an IPS firewall hitting a signature and dropping the traffic. It's probably not the remote server since no FIN but not sure how reliable that is.
SwiftOnSecurity
At this point we have acted on knowing _LITERALLY NOTHING_ except "the application stopped working." We didn't know WinSCP, we didn't know SSH protocol internals, or have any logs or server access.
This is not where you WANT to start, but sometimes starting at bare-metal when you're stuck gets you a new insight.
SwiftOnSecurity
Okay, we know these TCP sessions are simulatanously getting
KILL LA KILLED
But why?
SwiftOnSecurity
I get WinSCP to right before the problem occurs, start recording, in one window, and start the transfers in the other. 1, 2, 3, 4... Something flashes by in Wireshark scrolling packets. 5 is stuck... Timeout.
Well that temporal clue is unusually handy, isn't it. Could be nothing. Let's go look.
!!! NOTE !!! This Wireshark also shows the computer talking with the server over TLS/443, ignore those they're irrelevant to this scenaro.
SwiftOnSecurity
Before jumping to conclusions at red text, let's look around. I familiarize myself with the flow of the traffic and patterns. I've never troubleshooted SSH traffic before, but I'm going to list you some of what I start to observe. Importantly, I find when it was normal.
SwiftOnSecurity
DISCLAIMER: I am not a network professional and there's more Wireshark stuff I could have done this is just for approachability.
Okay, we've found a boundary layer. This is the last SSH-tagged packet and the last from the remote server, plus an ACK from client. After that it's internal noise. (you could filter more but don't have to).
SwiftOnSecurity
Learn to use the Statistics tools in Wireshark.
Okay, so in networking mutliple sessions can occur by opening new source port, connecting to the server again, and then doing business that way. That appears to be how SSH works. Look! We see 4 connections with lots of traffic, then
SwiftOnSecurity
Okay WinSCP starts opening more (we're going to call them multithreaded) connections to transfer 6 files at once. In some protocols, you can have multiple connections for the same file.
It starts opening them... and then the last one is stuck at connecting. And the others die?
SwiftOnSecurity
In this non-enterprise scenario there are basically 6 broad layers this problem could be. Application, Security, OS, Network device, or Server. Many paths. It can be paralyzing, this is where experience comes in.
In large incidents you may encounter varying levels of "hot potato" between departments where no path is selected because there is no incident command. You can learn to be that person.
SwiftOnSecurity
To emulate often limited information and collection abilities early in an incident, before doing ANY other basic troubleshooting we're going to launch Wireshark to inspect the network. Start somewhere you can!
"But SSH is encrypted."
Troubleshooting is often connecting metadata.
SwiftOnSecurity
"I don't know what I'm looking for, but I'll know it when I see it."
I have never troubleshooted WinSCP or SSH packets before. Wireshark I've used for years as basic user but not often, and my networking is really fundamentals and "weird stuff I've run into."
Importantly, I kind of know what Normal looks like.
SwiftOnSecurity
NOTE: You can mute this thread if not interested it will be long.
I have a seedbox in Europe to coalesse torrent downloads from other servers at 10gbe uplink to many other similar colocated servers hosting the content. I then collect finished over SSH file copy at my leisure.
SwiftOnSecurity
In some scenarios you can increase overall transfer speeds by running multiple sessions simultaneously, like a multi-lane highway. This can help saturate your connection, which I was not getting.
I go into WinSCP and turn this on, 6 sounds good.
Rich Felker
@SwiftOnSecurity LOL'd that in the end it was your own BOFH inspection rule doing it. I kinda guessed this halfway thru tho. 😁
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.