Conversation

Notices

1. Embed this notice
   Pete Wright (pete_wright@nlogic.systems)'s status on Tuesday, 01-Aug-2023 23:39:19 JST Pete Wright
   Some dude calling himself Chief Delivery Officer at my job, because he must have a C level job title, telling me how to hire syseng people.
   “Software devs make better devops engineers in my experience”
   
   Ok, first of all DevOps is not a job it’s a process.
   
   Also wtf planet are you from - have you ever seen what devs do to their local computers trying to get basic shit working 😅
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Tuesday, 01-Aug-2023 23:39:18 JST feld

     in reply to
     most software devs don't understand the OS and network which is the most important part
     
     You need sysadmins who learn to program not an application developer who tries to learn the OS/network stack
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Tuesday, 01-Aug-2023 23:49:51 JST feld

     in reply to
     pipe it through syslog hack:
     
     /var/service/foo/run:
     
     #!/bin/sh
     
     exec 2>&1
     
     whatever else to start the service
     
     
     /var/service/foo/log/run:
     
     #!/bin/sh
     
     exec /usr/bin/logger -t foo -p daemon.info
   • Embed this notice
     Pete Wright (pete_wright@nlogic.systems)'s status on Tuesday, 01-Aug-2023 23:49:52 JST Pete Wright

     in reply to

     • feld
     @feld yea that’s certainly one of the main challenges, that and logging imho should be taught with more emphasis.
     
     I’ve been happy moving most of my poorly maintained services to daemon(8). But I’ll agree there really isn’t one approach to rule them all.
     
     One thing I really hate about daemontools are the binary logs. I ran a large scale erlang cluster managed by daemontools and it was always so awkward to see what was going on, which was usually during an emergency. Interestingly that’s my least fav feature of systemd too.
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Tuesday, 01-Aug-2023 23:53:42 JST feld

     in reply to
     also 👋 I am the one that committed the ability to have a separate pidfile for supervisor and child
     
     I think I was involved in the syslog output too but it's been literally a decade or more
     
     This is one of my only C contributions to the open source world and it heavily relied on assistance of a friend who knew C much better than me.
   • Embed this notice
     feld (feld@bikeshed.party)'s status on Wednesday, 02-Aug-2023 07:51:11 JST feld

     in reply to

     • Charles Corbett
     Or all the kernel tuneables, the importance sometimes of CPU affinity to prevent cache thrashing, aligning storage block sizes to the data of the application... it goes on and on
     
     You could sit at the bar with another seasoned sysadmin and every time you think of a new important OS knob that needs to be considered for application performance at scale the other person has to drink and you'll both end up in the hospital
   • Embed this notice
     Charles Corbett (chas@tty0.social)'s status on Wednesday, 02-Aug-2023 07:51:12 JST Charles Corbett

     in reply to

     • feld

     @pete_wright @feld I've dealt with this regularly in my career. I've helped build infrastructure that served 150M end users with streaming media. Like, listen to me sometimes at least. I feel dismayed sometimes that there are developers who refuse to learn the first thing about linux when it is their target deployment platform. If you don't know what a ulimit is and you're trying to write software that handles massive concurrency you're gonna have problems

   • Embed this notice
     Pete Wright (pete_wright@nlogic.systems)'s status on Wednesday, 02-Aug-2023 07:51:13 JST Pete Wright

     in reply to

     • feld
     • Charles Corbett
     @chas @feld
     
     Thats awesome to know there are folks out there who've made that transition. I and I think it's great when it works out - lots of my mentors where sysadmins who were C programmers because that was so critical for supporting new hardware.
     
     But, to be honest I get tired of how quickly people discount the experience of people with decades of systems/network/support experience in favor of developers who speak the same language as senior management who are mostly people with business/softwaredev backgrounds.
     
     I see this pretty frequently when working in places trying to implement the DevOps philosophy, but maybe I'm just unlucky.
   • Embed this notice
     Charles Corbett (chas@tty0.social)'s status on Wednesday, 02-Aug-2023 07:51:14 JST Charles Corbett

     in reply to

     • feld

     @feld @pete_wright hi! former sysadmin who learned to code a decade ago and did devops/SRE for a long time. Now operate more like a software/infra architect turning eng manager. I've personally seen software engs make very good infra/ops people - but they were the ones who actually tried to learn their target system deeply.

   • Embed this notice
     feld (feld@bikeshed.party)'s status on Wednesday, 02-Aug-2023 08:10:11 JST feld

     in reply to

     • Charles Corbett
     Sounds almost exactly like a customer I had at a previous job...
     
     Remember stupid shit like "Apache htaccess files being enabled slows down the web cluster because every request causes a check for .htaccess file starting at /.htaccess and your website is 20 dirs deep so even your RAID runs out of IOPS once you're serving a couple hundred reqs/s" ?
     
     Or "your app is calling gettimeofday() every 30 nanoseconds"
     
     Stop abusing computers please 🙏 they have a family too.
   • Embed this notice
     Charles Corbett (chas@tty0.social)'s status on Wednesday, 02-Aug-2023 08:10:12 JST Charles Corbett

     in reply to

     • feld

     @feld @pete_wright actually funny story specific to this exact thing with CPU affinity. I worked at a place that had 20,000 iptables rules and didn't understand why connections were slow. They also didn't realize haproxy could split TLS offloading and traffic routing across CPUs. They had low-traffic but it operated like it was intensely under load all of the time because of this.

   • Embed this notice
     Pete Wright (pete_wright@nlogic.systems)'s status on Wednesday, 02-Aug-2023 08:11:35 JST Pete Wright

     in reply to

     • feld
     • Charles Corbett
     @chas @feld
     
     there is a flip side to this too. being on the support side of things gives you a unique perspective on the beauty of the KISS principle. i think people underestimate the power of a few well chosen Unix components. it may not get you famous on the conference circuit, but it also seems to help me sleep through the night more frequently.
     feld likes this.