ChatGPT and DeepL ①: Comparing the Accuracy of Japanese to English Translation in Manufacturing and IT Fields

1. Test 1: Text with Simple Structure

First, let's check the translation results of a sentence with a simple grammar structure.
As a sidenote, for all of these verification tests, translations on ChatGPT were conducted with the basic prompt of "Translate this sentence into English," without any additional instructions.
The results of the first test sentence are as follows.

From these results, the points that require changes in post-editing are as follows. (Note: The level of post-editing described here follows the quality standards of "full editing," where the results should closely resemble the accuracy and natural flow human translation.)

These results indicate that DeepL's translations have an impressively low post-editing burden. In contrast, ChatGPT's translations seem to contain many unnecessary expressions, such as "purposes" and "please." Additionally, ChatGPT's translations tend towards using the plural forms of nouns, like "ports" and "purposes," which may require adjustments depending on the context.

If handled by a human translator, the results may resemble the following. For ease of comparison, I have also included the raw translation results from ChatGPT and DeepL, before the abovementioned post-editing.

It's worth mentioning that the addition of "any device" after "connect" is a thoughtful solution that only human translation can provide. If you simply translate the sentence exactly, using the information contained in the source, the result will always be a bit unnatural, as shown in both automatic translations above, because "connect" is a transitive verb without a direct object.

It is almost impossible for machine translation or AI translation to interpret and accurately introduce information that was omitted from the original text. In Japanese technical documents, subjects and direct objects are often omitted. This is important to keep this in mind when using a machine translation engine or AI to translate from Japanese to English.

2. Test 2: Text with Simple Structure

Here is another sample sentence with simple structure.

There are two points of interest that the translation results of ChatGPT and DeepL share. First, the original text uses "ください" ('kudasai'), but it was translated as a soft request, rather than an imperative command. (Of course, the imperative form is not always necessary when translating 'kudasai,' but it is most appropriate for this context.) The second point is that both translations use the phrase "please note," which is a loose interpretation of the content that is not actually present in the original Japanese text, so it is interesting that both systems generated the same result.

Neither composition meets the quality standards required for a full edit, and a complete rewrite is necessary. (*In the case of "light editing" (critically essential edits only), a complete rewrite is not required.)

If handled by a human translator, the results may resemble the following. Once again, for easy comparison, I have included the raw translation results from ChatGPT and DeepL before post-editing.

3. Test 3: Text with Complex Structure

Next, we will verify and compare the translations of text with a slightly more complex structure.

In the results of DeepL, there are mistranslations and ambiguous expressions, but there are no significant issues with the overall structure, so the post-editing workload is not especially large. On the other hand, the structure of the translation generated by ChatGPT is indeed advanced, but because the meaning of the original content is not accurately conveyed, a complete rewrite is necessary.

If handled by a person, the translation may resemble the following. I believe this approach to composition is unique to human translation, as I have improved the readability by splitting the original idea, which was expressed in one sentence, into two.

4. Results of Verification & Summary

Let's tally the results of this comparison.

According to this series of tests, it appears that, from the perspective of workload, DeepL tends to require slightly less effort in post-editing.

However, there are some additional points to consider. In these verification samples, we provided just one simple prompt to ChatGPT: "Translate this sentence into English." In reality, ChatGPT's greatest strengths lie in its ability to translate according to specific instructions regarding grammar, terminology, and sentence structure. By providing more detailed instructions, more accurate translations can be obtained.

Of course, it's worth noting that there are some drawbacks to this strength of ChatGPT. First, if you are not familiar with the grammar, structure, style conventions, and basic rules of technical writing for the target language, it can be difficult to provide appropriate instructions. Additionally, adding specific instructions for each translation task can be cumbersome.

Ultimately, the most important thing when undertaking translation work is understanding the strengths of each method and utilizing the system that is most suited to your needs.

For example, DeepL is well-equipped to process large amounts of text at once, especially with the addition of post-editing. Meanwhile, ChatGPT is better suited when you want to provide detailed instructions regarding terminology, grammar, and structure at the sentence or paragraph level.

Of course, when there are many regulations and reference materials regarding formats and terminology, and in cases where the text must be cross-referenced with other documents, human translation may turn out to be the most efficient and cost-effective.

We will continue to verify the accuracy of technical documents translated using ChatGPT and DeepL and will publish the results here in our company blog, so please check back for our latest findings.

Human Science offers human translation services as well as post-editing services. We are a translation company that handles translations in a wide range of fields including software, manufacturing, IT, automotive, and distribution. We have been assisting numerous companies with their translation needs since 1994. If your pursuit of translation is plagued by concerns like the ones below, please don't hesitate to contact us for a free consultation.

・Translation takes too much time!
・We get tons of complaints about poor quality!
・Translation is too expensive!
・The source manual is causing many issues!
・I don't know overseas laws and regulations!