Conversation

Notices

Embed this notice
EdBoatConnoisseur (edboatconnoisseur@poa.st)'s status on Thursday, 07-Nov-2024 13:32:46 JST EdBoatConnoisseur
i'm not surprised at all that it can be done but i guess that looking at a practical example is nice, i kinda want to build something using plain text files as databases but adding headers to the files so that instead of "blindly" using the column numbers being able to do get stuff from a column by the name it has on the header.
@karna @p @theorytoe @skylar @skylar
https://www.howtogeek.com/build-a-database-with-powerful-linux-built-in-tools/
In conversation about 15 days ago from poa.st permalink
Attachments
1. Untitled attachment
- Embed this notice
  pistolero (p@fsebugoutzone.org)'s status on Thursday, 07-Nov-2024 13:32:45 JST pistolero
  in reply to
  @EdBoatConnoisseur @karna @theorytoe @skylar It's good stuff but almost all of it goes back to v7, not just Linux stuff. Maybe worth noting, he keeps typing '{print $1":"$2":"$3":"$4}' and you can just '{OFS=":"; print $1, $2, $3, $4}' and it comes out a little more readable, and most of what he uses head/tail for you can just use sed to do, so his `head -n 20 | tail -n 10` to get lines 11-20 would just be `sed -n 11,20p`. I think it's interesting that people keep getting surprised by this stuff, you know? Like, this was the point of Unix: you could do this kind of thing ad hoc.
  
  In conversation about 15 days ago permalink
  
  ✙ dcc :pedomustdie: :phear_slackware: likes this.
- Embed this notice
  karna :flipflop: :buffsuki: (karna@poa.st)'s status on Thursday, 07-Nov-2024 13:33:24 JST karna :flipflop: :buffsuki:
  in reply to
  @EdBoatConnoisseur @p @theorytoe @skylar for a second I thought this was gonna about something more complicated than flat files based off the headline, but you can do a lot with just that too. While reading, a thought occurred to me though.. Depending on your use case, you may want to "transpose" the file so that each line represents a column (sort of like how columnar formats like parquet store data column-wise instead of row-wise) if you have a lot of columns and you dont always need to work with every column. Some of the commands would have to change but I wonder how each approach would scale
  
  In conversation about 15 days ago permalink
- Embed this notice
  pistolero (p@fsebugoutzone.org)'s status on Thursday, 07-Nov-2024 13:33:24 JST pistolero
  in reply to
  @karna @EdBoatConnoisseur @theorytoe @skylar If you're careful about how you do it, regular Unix utilities sometimes scale better than the tools that are designed to scale: https://livefreeordichotomize.com/posts/2019-06-04-using-awk-and-r-to-parse-25tb/ .
  In conversation about 15 days ago permalink
  Attachments
  1. Domain not in remote thumbnail source whitelist: livefreeordichotomize.com
    
    - Using AWK and R to parse 25tb
    
    from @LucyStats
    
    Recently I was tasked with parsing 25tb of raw genotype data. This is the story of how I brought the query time and cost down from 8 minutes and $20 to a tenth of a second and less than a penny, plus the lessons learned along the way.
  ✙ dcc :pedomustdie: :phear_slackware: likes this.
- Embed this notice
  pistolero (p@fsebugoutzone.org)'s status on Thursday, 07-Nov-2024 13:33:47 JST pistolero
  in reply to
  @karna @EdBoatConnoisseur @skylar @theorytoe
  
  > Also did you see the recent-ish guadec talk about all the edge cases that happen in developing a calendar. Was pretty funny what with the strange timezones that exist.
  
  I wrote a thing to allow lawyers to sell time; a client provided services to lawyers and they wanted to make it easy to do something like sell a monthly subscription, where they'd do document review and then you could get some number of 15-minute consultations per month. So the lawyers would set up a time, they'd describe their schedule like they would do Tuesday and Thursday afternoons.
  
  Here's something: the reason every startup in San Francisco has "a Delaware Limited Liability Corporation" on their "Legal" page is that they are all incorporated in Delaware, because VCs insist on it. Delaware is one of two states that doesn't require a corporation to disclose its owners and VCs won't invest if they can't lie about what they own and how much of it they own. Delaware, consequently, has a large industry of lawyers that just maintain P.O. boxes and act as the registered agent for a thousand shitty LLCs.
  
  So the situation you have to handle is a lawyer drags out "2 p.m. to 5 p.m." on the "Tuesday" and "Thursday" slots in a calendar, they're in one timezone, and a client in another timezone (possibly with different DST rules) wants to schedule a call, and you need to present them with five boxes to click on that have available time slots. This is a nightmare to actually build.
  global-DST.jpg
  In conversation about 15 days ago permalink
  Attachments
  1. global-DST.jpg
    https://fsebugoutzone.org/media/929bf77d-8f22-4312-8884-16a075a95c2d/global-DST.jpg?name=global-DST.jpg
  ✙ dcc :pedomustdie: :phear_slackware: likes this.
- Embed this notice
  karna :flipflop: :buffsuki: (karna@poa.st)'s status on Thursday, 07-Nov-2024 13:33:48 JST karna :flipflop: :buffsuki:
  in reply to
  @EdBoatConnoisseur @p @skylar @theorytoe That sounds pretty nice. Caldav is comfy too. Also did you see the recent-ish guadec talk about all the edge cases that happen in developing a calendar. Was pretty funny what with the strange timezones that exist. inb4 gnome..
  
  In conversation about 15 days ago permalink
- Embed this notice
  karna :flipflop: :buffsuki: (karna@poa.st)'s status on Thursday, 07-Nov-2024 13:33:49 JST karna :flipflop: :buffsuki:
  in reply to
  @EdBoatConnoisseur @p @theorytoe @skylar that sounds about right
  
  In conversation about 15 days ago permalink
- Embed this notice
  EdBoatConnoisseur (edboatconnoisseur@poa.st)'s status on Thursday, 07-Nov-2024 13:33:49 JST EdBoatConnoisseur
  in reply to
  @karna @p @skylar @theorytoe now i'm thinking of perhaps rolling out my own calendar thing, inspired by calcurse, storing the appointments in plain text, with a daemon to be able to sync with cloud calendar shit like google calendar and any caldav provider, then some frontends terminal and gtk frontends, buuuuuuut that would not be something i'd do this month.
  
  In conversation about 15 days ago permalink
- Embed this notice
  EdBoatConnoisseur (edboatconnoisseur@poa.st)'s status on Thursday, 07-Nov-2024 13:33:50 JST EdBoatConnoisseur
  in reply to
  @karna @p @theorytoe @skylar i think awk can still handle that without much issue but yeh you'd have to search by the first column to get the data, dunno why but i feel that "classic" row wise ordered data may be more efficient for the core utils as that is what they were usually designed around.
  
  In conversation about 15 days ago permalink
- Embed this notice
  pistolero (p@fsebugoutzone.org)'s status on Friday, 08-Nov-2024 04:27:23 JST pistolero
  in reply to
  @EdBoatConnoisseur @theorytoe @karna @skylar Ha, I can't remember; I think :bwk: said something like that in his talk but I never went looking.
  
  In conversation about 14 days ago permalink
  
  ✙ dcc :pedomustdie: :phear_slackware: likes this.
- Embed this notice
  EdBoatConnoisseur (edboatconnoisseur@poa.st)'s status on Friday, 08-Nov-2024 04:27:24 JST EdBoatConnoisseur
  in reply to
  @p @theorytoe @karna @skylar well, most of the unix tools were made to specifically process data, what was the story that aho started what would became awk because he needed to automate student grading and sed wasn't a thing yet right?
  
  In conversation about 14 days ago permalink
- Embed this notice
  Berk Berkman (berkberkman@shitposter.world)'s status on Friday, 08-Nov-2024 04:29:46 JST Berk Berkman
  in reply to
  @p @theorytoe @EdBoatConnoisseur @karna @skylar Not the first time awk was used in a similar way. No wonder the Suckless website had this related link under Philosophy:
  https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
  In conversation about 14 days ago permalink
  Attachments
  1. Domain not in remote thumbnail source whitelist: adamdrake.com
    
    Adam Drake
    
    Adam Drake is an advisor to scale-up tech companies. He writes about ML/AI/crypto/data, leadership, and building tech teams.
  ✙ dcc :pedomustdie: :phear_slackware: likes this.
- Embed this notice
  pistolero (p@fsebugoutzone.org)'s status on Friday, 08-Nov-2024 06:23:04 JST pistolero
  in reply to
  @laurel @EdBoatConnoisseur @karna @skylar @theorytoe
  
  > I thought that maybe you could use unix time for everything. Then the problem becomes only a matter of presentation. Hours charged and scheduling will always be correct,
  
  Nope. How do you apply DST to Unix time? So what I did was to keep the original representation around, calculate the times per day because a page never had to display more than thirty days. You can actually lean on Postgres for this but the queries come out extremely long so it was easier to grab a month, calculate when the slots would be. Not the most beautiful solution, but zero bugs and the more edge cases we started testing, the more skeptical I got that there could be a beautiful solution.
  
  > you'll just have to remind the user
  
  Lawyers are no good at this.
  
  > having one hour charged while the local clock would go forward by two
  
  Oh, if we're calculating the difference between two times, absolutely you just use Unix time, but this is different, this is basically drawing a calendar widget and letting a lawyer say when they are willing to block off time to talk to people on the phone, then presenting available slots to their clients to schedule a 15-minute call. Nothing to do with billing on that end, but the clients would pay, whatever, $100-200 per month flat and then they'd have a lawyer to call.
  
  In conversation about 14 days ago permalink
  
  ✙ dcc :pedomustdie: :phear_slackware: likes this.
- Embed this notice
  laurel (laurel@fsebugoutzone.org)'s status on Friday, 08-Nov-2024 06:23:05 JST laurel
  in reply to
  @p @karna @EdBoatConnoisseur @skylar @theorytoe
  
  How did you solve it?
  I thought that maybe you could use unix time for everything. Then the problem becomes only a matter of presentation. Hours charged and scheduling will always be correct, you'll just have to remind the user for the various quirks, like having one hour charged while the local clock would go forward by two. But you don't have to program any logic for this.
  
  In conversation about 14 days ago permalink