Conversation

Notices

1. Embed this notice
   Royce Williams (tychotithonus@infosec.exchange)'s status on Saturday, 24-Aug-2024 01:52:20 JST Royce Williams

   I know this dates me, but ... 80% of the problems I'm solving with jq are caused by using JSON at all ... when a simpler format would have been fine.

   Repeating every verbose field name in each record, when the schema is flat, is often premature "schema might need to be variable someday" optimization.

   When the Rapid7 DNS data was freely available, it was distributed as a one-line-per-stanza JSON file. The first thing I'd do after downloading it was convert it to CSV ... which cut its size by 60%.

   It's like buying a ten-pound box of individually wrapped grains of rice.

   • Embed this notice
     Ryan Castellucci :nonbinary_flag: (ryanc@infosec.exchange)'s status on Saturday, 24-Aug-2024 01:52:19 JST Ryan Castellucci :nonbinary_flag:

     in reply to

     @tychotithonus This matters a lot less if you compress it. Overhead matters a lot less at that point.

     I have a bunch of tooling that automatically detects (via the first few bytes) compressed data and handles it. Mostly using zstandard.

     Also, CSV is bad and I hate it. TSV is acceptable

   • Embed this notice
     Royce Williams (tychotithonus@infosec.exchange)'s status on Saturday, 24-Aug-2024 02:08:42 JST Royce Williams

     in reply to

     • Ryan Castellucci :nonbinary_flag:

     @ryanc Yeah, Definitely pro TSV! When I say CSV out loud, I actually mean TSV in my head. I need to watch that ...

     I'll also have to dig up the post where I grieve for the alternate future where we actually used the actual dedicated field and record separator characters built into ASCII. So much avoidable pain.

   • Embed this notice
     Ryan Castellucci :nonbinary_flag: (ryanc@infosec.exchange)'s status on Saturday, 24-Aug-2024 05:10:10 JST Ryan Castellucci :nonbinary_flag:

     in reply to

     • Aaron Toponce ⚛️:debian:

     @tychotithonus @atoponce I used to regularly work with TSV where the values were JSON. :lolsob:

   • Embed this notice
     Royce Williams (tychotithonus@infosec.exchange)'s status on Saturday, 24-Aug-2024 05:10:11 JST Royce Williams

     in reply to

     • Aaron Toponce ⚛️:debian:
     • Ryan Castellucci :nonbinary_flag:

     @atoponce Yes, exactly! ...for sufficiently obscure and esoteric values of "solve".😅 Most people have never even heard of them.

     If they had been universally used from the beginning, CSV wouldn't even be a thing, and plenty of things we do to avoid CSV would also not be things ...

     @ryanc

   • Embed this notice
     Aaron Toponce ⚛️:debian: (atoponce@fosstodon.org)'s status on Saturday, 24-Aug-2024 05:10:12 JST Aaron Toponce ⚛️:debian:

     in reply to

     • Ryan Castellucci :nonbinary_flag:

     @tychotithonus @ryanc Doesn't 0x1E solve this problem as a record separator, or am I misunderstanding?

   • Embed this notice
     Chris Herdt (cherdt@infosec.exchange)'s status on Sunday, 29-Dec-2024 03:01:30 JST Chris Herdt

     in reply to

     • Ryan Castellucci :nonbinary_flag:

     @ryanc @tychotithonus The place it makes the biggest difference for me is Splunk ingest volume. Switching Zeek logs to TSV is going to cut 500GB or more from my daily ingest!