Fedilab Apps
@yacc143
I'm not sure if I understand you correctly, but there is `Settings` --> `Compose` --> `Select languages in the picker`. If you select two languages, you can switch with one click. With more than two, a menu with the selected languages opens. Below that setting is the new language checker. It warns you if the language seems to be selected incorrectly and helps you correct it.
Marco
Andreas K
```
language = langdect(text)
if language == "UN": language = language_default
```
You might wonder what the language_default is:
- you either ask the user.
- you use the dominant language the user used to post in the past.
Basically we used this with the first option for datasets of texts (don't ask, many datasets, many texts, many countries, oddest quality standards), and it works surprisingly well on short texts (title/subjects).
Andreas K
Why does that detection work surprisingly well:
- by setting a set of allowed languages, you kick out weaker/wrong signals, e.g. if I post a German thread about literature, and have a couple of French articles cited in them, they will not sudden make the toots that have the French words from these title turn into a French language toot (as long I did not admit that I'm also a French speaker, nothing is perfect)
- and the unsolvable case "unknown" gets the statistically most likely answer.
Andreas K
Yes you can probably construct a predictor function that works for specific cases better, but in general that construct -> constrain to allowed set of languages + use the statistically most like solution in the cases where you can cannot cleanly predict -> should give something near the ideal predictor.
(Our production ETL raised also an operator warning if a language not in the allowed set popped up to often in a data set, so an human could look over it to check if a lang was overlooked.)
Andreas K
@apps but also @elk are even worse than the mastodon webui btw -> they literally do not allow you restrict the language settings so switching between my two primary languages that I use (DE/EN, I officially left school with 4 languages on my CV) is a PITA.
Andreas K
@FediTips Ah, that's where an automatic heuristic could probably help, e.g. for European users that statistically will be multilingual. But potentially favour one language, as most EU MS/regions have a dominant local language. Again, an assumption based on statistics.
You correctly mentioned that language detection is unreliable for short texts. Sigh. But if you ask the user what languages (s)he speaks, the following works surprisingly well for short texts (e.g. titles of documents):
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.