Embed Notice

Got an "emergency" helpdesk ticket today, a customer lost communication between all six sites, which is critical for their operation.

Logged into the main firewall that all the other five sites connect to via VPN, as they have two internet connections with public IPs.

Found that all VPNs were down on both Internet connections, failover not working, and that no traffic was coming in on UDP 500/4500 on the wan side.

I suspected their CPE routers were the culprit, and wouldn't you know it, someone created a remote access IPSec connection on both routers, using the ports required by the firewall behind them.

Called the customer and they confirmed that some MSP that shall not be named had set up their own VPN access yesterday without telling anyone, nor testing if anything's still running. 😑

Scenarios like this make up most of my work: Fixing problems caused by other people who don't know what they're doing, but are smart enough to be dangerous.