Notices by SwiftOnSecurity (swiftonsecurity@infosec.exchange)

Oops you tried to open an unsupported port I just crashed your industrial press :(

*opens the breeder reactor after hearing weird sounds*

*a bunch of fucking furries fall out*

I was there Gandalf. I was there 3000 years ago...

15 years ago: The Guild - Do You Want To Date My Avatar
https://youtu.be/urNyg1ftMIU

AI nudifier makes a mockery of Meta's moderation https://fakedup.substack.com/p/ai-nudifiers-make-a-mockery-of-metas-moderation

In some scenarios you can increase overall transfer speeds by running multiple sessions simultaneously, like a multi-lane highway. This can help saturate your connection, which I was not getting.

I go into WinSCP and turn this on, 6 sounds good.

NOTE: You can mute this thread if not interested it will be long.

I have a seedbox in Europe to coalesse torrent downloads from other servers at 10gbe uplink to many other similar colocated servers hosting the content. I then collect finished over SSH file copy at my leisure.

==Training Lesson==
INVESTIGATION NARRATIVE: SSH Kill la Killed 🧵

My job is to solve the Weird Problems as the Final escalation tier. I do this with generalist knowledge and practical experience.

New InfoSec/IT entrants often ask what this looks like in practice. Follow below.

"I don't know what I'm looking for, but I'll know it when I see it."

I have never troubleshooted WinSCP or SSH packets before. Wireshark I've used for years as basic user but not often, and my networking is really fundamentals and "weird stuff I've run into."

Importantly, I kind of know what Normal looks like.

To emulate often limited information and collection abilities early in an incident, before doing ANY other basic troubleshooting we're going to launch Wireshark to inspect the network. Start somewhere you can!

"But SSH is encrypted."

Troubleshooting is often connecting metadata.

In this non-enterprise scenario there are basically 6 broad layers this problem could be. Application, Security, OS, Network device, or Server. Many paths. It can be paralyzing, this is where experience comes in.

In large incidents you may encounter varying levels of "hot potato" between departments where no path is selected because there is no incident command. You can learn to be that person.

Okay WinSCP starts opening more (we're going to call them multithreaded) connections to transfer 6 files at once. In some protocols, you can have multiple connections for the same file.

It starts opening them... and then the last one is stuck at connecting. And the others die?

Learn to use the Statistics tools in Wireshark.
Okay, so in networking mutliple sessions can occur by opening new source port, connecting to the server again, and then doing business that way. That appears to be how SSH works. Look! We see 4 connections with lots of traffic, then

DISCLAIMER: I am not a network professional and there's more Wireshark stuff I could have done this is just for approachability.

Okay, we've found a boundary layer. This is the last SSH-tagged packet and the last from the remote server, plus an ACK from client. After that it's internal noise. (you could filter more but don't have to).

Before jumping to conclusions at red text, let's look around. I familiarize myself with the flow of the traffic and patterns. I've never troubleshooted SSH traffic before, but I'm going to list you some of what I start to observe. Importantly, I find when it was normal.

I get WinSCP to right before the problem occurs, start recording, in one window, and start the transfers in the other. 1, 2, 3, 4... Something flashes by in Wireshark scrolling packets. 5 is stuck... Timeout.
Well that temporal clue is unusually handy, isn't it. Could be nothing. Let's go look.

!!! NOTE !!! This Wireshark also shows the computer talking with the server over TLS/443, ignore those they're irrelevant to this scenaro.

Okay, we know these TCP sessions are simulatanously getting

KILL LA KILLED

But why?

At this point we have acted on knowing _LITERALLY NOTHING_ except "the application stopped working." We didn't know WinSCP, we didn't know SSH protocol internals, or have any logs or server access.

This is not where you WANT to start, but sometimes starting at bare-metal when you're stuck gets you a new insight.

What do we know so far?

1.) Initial connectivity works.
2.) Connectivity with up to 4 sessions seems to work.
3.) The 5th connection doesn't work.
4.) All connectivity to server stops functioning at this point.

This smells like an IPS firewall hitting a signature and dropping the traffic. It's probably not the remote server since no FIN but not sure how reliable that is.

(There are MULTIPLE ways to get to this eventual understanding)

For some reason the 5th SSH session is dying early. Let's filter on the stream ID to isolate this individual TCP conversation. Yes there's a SYN then two repeats. Nothing is getting established from get-go. But what about the others?

Going back to our original black lines in Wireshark, those are the 4 different existing connections trying to re-transmit an ACK saying please continue sending. But they are never hearing back. So the APPLICATION is seemingly not dying, it is somewhere else. And no TCP tear-down.