Embed Notice

HTML Code

<blockquote style="position: relative; padding-left: 55px;"><section><a href="https://fire.asta.lgbt/notes/a64gmvtnxhl5034p">Asta [AMP] (aud@fire.asta.lgbt)'s status on Thursday, 03-Apr-2025 14:10:37 JST</a><a href="https://fire.asta.lgbt/@aud" title="aud@fire.asta.lgbt"><img src="https://gnusocial.jp/theme/gnusocialjp/default-avatar-stream.png" width="48" height="48" alt="Asta [AMP]" style="position: absolute; left: 0; top: 0;">Asta [AMP]</a></section><article><p>heeeeeey <a href="https://fire.asta.lgbt/tags/python" rel="tag">#python</a> cats!<br><br>anyone know of a decent multi-language text tokenizer?<br><br>To be clear: I am explicitly looking to use it for non-generative-AI and other [slop/scab/labor theft] purposes.   <br><br>Not sure of the specific terms I need to be looking up, frankly, since I'm mostly just finding Python's built in tokenize library which seems to be focused just on Python code.<br><br>Thank you!<br><br><a href="https://fire.asta.lgbt/tags/techPosting" rel="tag">#techPosting</a></p></article><footer><a rel="bookmark" href="https://gnusocial.jp/conversation/4837170#notice-9472780">In conversation</a><time datetime="2025-04-03T14:10:37+09:00" title="Thursday, 03-Apr-2025 14:10:37 JST">about 2 months ago</time> <span>from <span><a href="https://fire.asta.lgbt/notes/a64gmvtnxhl5034p" rel="external" title="Sent from fire.asta.lgbt via ActivityPub">fire.asta.lgbt</a></span></span><a href="https://fire.asta.lgbt/notes/a64gmvtnxhl5034p">permalink</a></footer></blockquote>

Corresponding Notice

Embed this notice
Asta [AMP] (aud@fire.asta.lgbt)'s status on Thursday, 03-Apr-2025 14:10:37 JST Asta [AMP]
heeeeeey #python cats!

anyone know of a decent multi-language text tokenizer?

To be clear: I am explicitly looking to use it for non-generative-AI and other [slop/scab/labor theft] purposes.

Not sure of the specific terms I need to be looking up, frankly, since I'm mostly just finding Python's built in tokenize library which seems to be focused just on Python code.

Thank you!

#techPosting
In conversationabout 2 months ago from fire.asta.lgbtpermalink

Public

Embed Notice

HTML Code

Corresponding Notice