Greater than 2 minutes
The new Google Translate – a quick test case for the English – Dutch language pair I tested the new and improved version of Google Translate
Great news : Google announced Google Translate is now applying its new machine learning algorithms to the English into Dutch language pair. Neural machine learning will improve the quality of Google translations. We all applaud such improvement of useful technology!
Does that mean Google Translate has evolved from a rudimentary tool that helps people understand the essence of the message in a language they do not master, and the infamous source of jokes like Google Translate fails and Google Translate sings, to a translation service that is useful in a professional environment? Can you finally use it to translate your product brochures or website text into new languages to reach new markets all over the world? Can you address potential business partners in their own language when you write them an email?
Let’s do a quick check.
I tested ten Google Translate fails from the earlier version. I took ten short sentences that I found listed on the page Dat vertaal ik beter and checked what the new Google Translate made of them.
These are ten examples of sentences Google used to struggle with. They are all short, some are plain language, some are very common expressions. I did not filter these in any other way, they were simply the ten sentences that seemed the most “normal” to me. I skipped sentences that contained dialect words. I did not pick the ones I thought were going to fail. Neither are these conceived in such a way that we are trying to trick Google into producing a funny translation (like that infamous sentence that contains three animals: tits, cock and ass). Let’s be nice to Google Translate, now.
The old Google Translate translated expressions literally, left negative words out and thus produced quite the opposite of what was meant, and translated the sentences very literal overall – often word per word. The old GT did not score high on grammar either.
Let’s see what the new and improved version makes of them:
The conclusion? The new and improved version is definitely better with grammar and that seems a great step forward. The result is still poor, though. These ten old fails remain fails, although they are translated less literally. Google Translate is still struggling with expressions, no matter how common. And negative sentences are still being translated as positive ones, which is bad news for anyone who was hoping to use Google Translate to produce texts in a language they do not master themselves.
Google is promising improvement, though. The new Google Translate should be able to learn and get better over time. Shall we check again in a year or so?
Olesya Zaytseva 
The Open Mic Team 
Els, I’m monitoring as well how Google, Microsoft and SDL are doing with their new engines and I can confirm they are a bit better. In most cases they are more fluent, but rarely more accurate. If you’re the type of translator that prefers fluency, the new engines will make you more productive. If your source texts contain a lot of terminology, probably you will be less happy with the current engines.
Your test sets contain quite a lot of idioms and I don’t included those in my tests, as I think MT (any kind of MT) is not fit to deal with texts like these. I’m using test sets that contain sentences that MT systems should be able to translate correctly: sentences that contain all the context an engine needs (so no reference to other sentences), and that are not too short, not to long (like this sentence). My test sets also don’t include idioms of course. Only texts that one may find in rather technical documentation.
That being said: I think every translator should do the type of test you did for his/her own translation jobs. I know it is much easier to test if an engine does a good job on your own jobs than to test if an engine does a good job (or a better job) on generic texts.
Hence why I wanted to share this feedback.
What I liked about your test is that you compare an older engine to a new engine.
“DatVertaalIkBeter” contains (funny) MT fails. This site wants to show humans translate better than machines. That is a bit strange: we all know humans translate better. Words is what humans use to communicate. Computers use bits and bytes. There is little value in proving that humans do a better job than computers using a human communication tool.
So I liked your comparative test, but not your test set.
Hi Gert, thank you for taking the time to read my article.
I understand your concerns. I don’t pretend to have tested the new GT comprehensively, though. I did not check texts with more context either. I wanted to run a quick check and selected some sentences that are normal sentences and that were previously translated incorrectly by GT – some are indeed idioms, but very common ones. I did not try to trick GT into producing the wrong results.
“DatVertaalIkBeter” contains a lot of sentences that would mislead GT, for instance by using incorrect grammar or dialect words. I deliberately did not use those. I was interested in seeing how different GT’s approach is compared to before.
I myself do not use MT in my work so I cannot give you my opinion on usability – you will be better placed to do that. So please consider this a “quick test”, as stated in the title, based on curiosity, without any further judgement. I am interested in seeing the evolution of the system and I am quite serious about testing again in a year or so – but never pretending that this is a serious test of the functionality of the new GT.
Interesting test, definitely worth repeating every now and then 🙂 Another idea could be to test a MT system with different sets of sentences common in different text types and to gain conclusion as to which fields benefit most from MT improvements…
That would be interesting, Eleonora! This was just a quick test.