In greek it is also really bad (and makes rather obvious mistakes), in french it seems much better but makes some very obvious mistakes too. To make it interesting, I emphasise that I want nouns that refer to objects only (else it just spits out profession names and stuff like that, which is not interesting).
Also tbh, with all the hype of LLMs one would think that such a task would not be such a challenge.
>Also tbh, with all the hype of LLMs one would think that such a task would not be such a challenge.
The strange/sad thing is that despite being "large language models" they're often hypermyopic on English..
I've done some measurements comparing generation between various languages in the prompt and no matter what I do half the time i cannot get them to not include english text or comments in code unless the request is made in japanese, chinese, or a similar very-different-language.
It's long been observed (e.g. Emily Bender has written some articles to that effect) that NLP technology underperforms on languages that aren't English, especially when they are significantly different structurally.
If you train and evaluate something mostly on a language loke English, you're going to end up with a model that thinks everything works like English, which means among other things very little morphological complexity.
It's actually quite amusing asking it to give you a list of consonants in which Greek words can end in, and then example words for each. It's pretty much all hallucination and then it tries to gaslight you.
Also tbh, with all the hype of LLMs one would think that such a task would not be such a challenge.