Conversation

Notices

1. Embed this notice
   Stefano Marinelli (stefano@mastodon.bsd.cafe)'s status on Wednesday, 11-Sep-2024 04:19:54 JST Stefano Marinelli

   This morning, I took my wife to the hospital for routine blood tests that had been scheduled for some time. Everything was going smoothly: check-in, number, waiting room. Suddenly, everything came to a halt and shut down. I was connected to the hospital’s public Wi-Fi and noticed that my connection also went down.

   Having managed a couple of similar facilities, I immediately understood what had happened. I saw the staff panicking and calling the technicians, but they quickly reorganized within 10 minutes. They managed to process everyone who already had a number and then proceeded with the others in the order of their arrival. Despite the ten-minute delay (even though people started complaining right away), they were extremely efficient.

   I later confirmed that the entire booking, check-in, and queue system is “in the cloud.” The hospital experienced a connectivity interruption, and all related services stopped. The staff no longer had access to anything, so a technician sent the lists to a manager via another channel, and everything resumed manually.

   For years, I’ve insisted that certain things MUST be local. The healthcare facilities I manage have all the necessary systems for the operation of the facility internally, including patient records. External services like websites, emails, etc., are secondary.

   Everything essential must always be accessible locally and, in special cases, it should be possible to physically access the servers and connect directly to them, bypassing any network/switch failures.

   There has been only one interruption in the past, due to human error. Today, we have redundant servers (not HA on virtualizers, but two machines running the same software with replicated databases - on separate power lines) so such an issue shouldn’t happen anymore.

   Not everything can be anticipated, but history is a great teacher. The Internet connection will eventually be interrupted :-)

   When it comes to the health and survival of people, there are no compromises.

   #IT #Internet #Networking #Outage #Health #HA #Cloud #CloudComputing #OwnYourData

   • clacke likes this.
   • Embed this notice
     Jess👾 (jesstheunstill@infosec.exchange)'s status on Thursday, 12-Sep-2024 14:10:55 JST Jess👾

     in reply to

     • Kevin Russell
     • TomAoki

     I can see both sides of the issue. Fact of the matter is that local IT at small hospitals and doctor's offices simply are not capable of maintaining a high availability and secure EMR system until and unless you get it all the way to "here's your half rack of server appliances - plug it in, done". Losing network is a major problem in other ways as well, of course. But either and both are cases that need to be carefully thought through as part of the facilities' BCP. A cyber attack taking down your local servers can kill your availability same as a network outage.

     @kevinrns
     @stefano @TomAoki

     clacke and Fish of Rage like this.
     clacke repeated this.
   • Embed this notice
     Kevin Russell (kevinrns@mstdn.social)'s status on Thursday, 12-Sep-2024 14:10:57 JST Kevin Russell

     in reply to

     • TomAoki

     @stefano @TomAoki

     My main point is the complex protections and needs have NOT been accounted for, the continuation of providing service despite computational sctewups has not been met, or studied.

     Its a complete dogs breakfast.

   • Embed this notice
     Stefano Marinelli (stefano@mastodon.bsd.cafe)'s status on Thursday, 12-Sep-2024 14:10:58 JST Stefano Marinelli

     in reply to

     • Kevin Russell
     • TomAoki

     @kevinrns @TomAoki of course, there's no need to keep the records for all the people. But at least the ones being treated by that specific ward

   • Embed this notice
     Kevin Russell (kevinrns@mstdn.social)'s status on Thursday, 12-Sep-2024 14:10:59 JST Kevin Russell

     in reply to

     • TomAoki

     @TomAoki @stefano

     There are rules that can be followed, but medical records for 51 million people do not need to be stored at every hospital.

     Your point is a start.

   • Embed this notice
     TomAoki (tomaoki@mastodon.bsd.cafe)'s status on Thursday, 12-Sep-2024 14:11:05 JST TomAoki

     in reply to

     @stefano
     Exactly.
     All critical data should be held locally as primary, and clouds should be for backups and shares. If making cloud as primary, multiple physical connections with different routes should be always kept, if the data there is "critical".

   • Embed this notice
     Demiurg (demiurg@fosstodon.org)'s status on Thursday, 12-Sep-2024 14:16:02 JST Demiurg

     in reply to

     • Jess👾
     • Kevin Russell
     • TomAoki

     @JessTheUnstill @kevinrns @stefano @TomAoki Yes, I second that. We have companies as customers and we had no one with a hack on cloud systems. We had several ones with incidents, who run their own servers 'due to security reasons'. You need a really capable team, a CISO and SIEM solutions to keep up with the threat actors. Most companies do not want to spend some budget on any of this.

     clacke likes this.
   • Embed this notice
     Xenotar (xenotar@mastodon.social)'s status on Wednesday, 23-Apr-2025 14:04:58 JST Xenotar

     in reply to

     @stefano Agree