GNU social JP
  • FAQ
  • Login
GNU social JPは日本のGNU socialサーバーです。
Usage/ToS/admin/test/Pleroma FE
  • Public

    • Public
    • Network
    • Groups
    • Featured
    • Popular
    • People

Conversation

Notices

  1. Embed this notice
    Royce Williams (tychotithonus@infosec.exchange)'s status on Saturday, 24-Aug-2024 01:52:20 JST Royce Williams Royce Williams

    I know this dates me, but ... 80% of the problems I'm solving with jq are caused by using JSON at all ... when a simpler format would have been fine.

    Repeating every verbose field name in each record, when the schema is flat, is often premature "schema might need to be variable someday" optimization.

    When the Rapid7 DNS data was freely available, it was distributed as a one-line-per-stanza JSON file. The first thing I'd do after downloading it was convert it to CSV ... which cut its size by 60%.

    It's like buying a ten-pound box of individually wrapped grains of rice.

    In conversation about 10 months ago from infosec.exchange permalink
    • Embed this notice
      Ryan Castellucci :nonbinary_flag: (ryanc@infosec.exchange)'s status on Saturday, 24-Aug-2024 01:52:19 JST Ryan Castellucci :nonbinary_flag: Ryan Castellucci :nonbinary_flag:
      in reply to

      @tychotithonus This matters a lot less if you compress it. Overhead matters a lot less at that point.

      I have a bunch of tooling that automatically detects (via the first few bytes) compressed data and handles it. Mostly using zstandard.

      Also, CSV is bad and I hate it. TSV is acceptable

      In conversation about 10 months ago permalink
    • Embed this notice
      Royce Williams (tychotithonus@infosec.exchange)'s status on Saturday, 24-Aug-2024 02:08:42 JST Royce Williams Royce Williams
      in reply to
      • Ryan Castellucci :nonbinary_flag:

      @ryanc Yeah, Definitely pro TSV! When I say CSV out loud, I actually mean TSV in my head. I need to watch that ...

      I'll also have to dig up the post where I grieve for the alternate future where we actually used the actual dedicated field and record separator characters built into ASCII. So much avoidable pain.

      In conversation about 10 months ago permalink
    • Embed this notice
      Ryan Castellucci :nonbinary_flag: (ryanc@infosec.exchange)'s status on Saturday, 24-Aug-2024 05:10:10 JST Ryan Castellucci :nonbinary_flag: Ryan Castellucci :nonbinary_flag:
      in reply to
      • Aaron Toponce ⚛️:debian:

      @tychotithonus @atoponce I used to regularly work with TSV where the values were JSON. :lolsob:

      In conversation about 10 months ago permalink
    • Embed this notice
      Royce Williams (tychotithonus@infosec.exchange)'s status on Saturday, 24-Aug-2024 05:10:11 JST Royce Williams Royce Williams
      in reply to
      • Aaron Toponce ⚛️:debian:
      • Ryan Castellucci :nonbinary_flag:

      @atoponce Yes, exactly! ...for sufficiently obscure and esoteric values of "solve".😅 Most people have never even heard of them.

      If they had been universally used from the beginning, CSV wouldn't even be a thing, and plenty of things we do to avoid CSV would also not be things ...

      @ryanc

      In conversation about 10 months ago permalink
    • Embed this notice
      Aaron Toponce ⚛️:debian: (atoponce@fosstodon.org)'s status on Saturday, 24-Aug-2024 05:10:12 JST Aaron Toponce ⚛️:debian: Aaron Toponce ⚛️:debian:
      in reply to
      • Ryan Castellucci :nonbinary_flag:

      @tychotithonus @ryanc Doesn't 0x1E solve this problem as a record separator, or am I misunderstanding?

      In conversation about 10 months ago permalink
    • Embed this notice
      Chris Herdt (cherdt@infosec.exchange)'s status on Sunday, 29-Dec-2024 03:01:30 JST Chris Herdt Chris Herdt
      in reply to
      • Ryan Castellucci :nonbinary_flag:

      @ryanc @tychotithonus The place it makes the biggest difference for me is Splunk ingest volume. Switching Zeek logs to TSV is going to cut 500GB or more from my daily ingest!

      In conversation about 6 months ago permalink

Feeds

  • Activity Streams
  • RSS 2.0
  • Atom
  • Help
  • About
  • FAQ
  • TOS
  • Privacy
  • Source
  • Version
  • Contact

GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.

Creative Commons Attribution 3.0 All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.