Translating strings that have newlines produces unexpected output
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	abi_
	Jun 27 2023, 9:01 AM

Description

Steps to replicate the issue (include links if applicable):

Use MinT via CX API
Try to translate content that has newlines

What happens?:
Sometimes content after newline gets translated, sometimes it does not. See examples below.

What should have happened instead?:
Proper predictable translations.

Software version (skip for WMF-hosted wikis like Wikipedia):

MediaWiki master

Other information (browser name/version, screenshots, etc.):

Some test cases:

Case 1:
Input to the CX API:

The value for a parameter is required.\nTo get full help for the command, send `{1}help {0}`.

Output:

किसी पैरामीटर के लिए मान आवश्यक है ।\nकमांड के लिए पूरी मदद पाने के लिए भेजें ।

Output on https://translate.wmcloud.org/ :
एक पैरामीटर के लिए मान आवश्यक है ।
कमांड के लिए पूरी मदद पाने के लिए भेजें ।

Case 2:
Input to the CX API:

"Streams are available only for Wikimedia projects.\nList of allowed domains: {0}.",

Output:

धाराएँ केवल विकिमीडिया परियोजनाओं के लिए उपलब्ध हैं ।\nअनुमत डोमेन की सूचीः { 0 ]

Output on https://translate.wmcloud.org/ :
धाराएँ केवल विकिमीडिया परियोजनाओं के लिए उपलब्ध हैं ।
अनुमत डोमेन की सूचीः { 0 ]

Case 3:
Input to the CX API:

The stream goal (page title or namespace number) is required.\nTo get full help for the command, send `{1}help {0}`.

Output:

स्ट्रीम लक्ष्य (पृष्ठ शीर्षक या नेमस्पेस संख्या) की आवश्यकता है ।\nकमांड के लिए पूरी मदद पाने के लिए भेजें

Output on https://translate.wmcloud.org/ :
स्ट्रीम लक्ष्य (पृष्ठ शीर्षक या नेमस्पेस संख्या) की आवश्यकता है ।
कमांड के लिए पूरी मदद पाने के लिए भेजें ।

Details

	Subject	Repo	Branch	Lines +/-
	MachineTranslationAid: Avoid overriding variables	mediawiki/extensions/Translate	master	+7 -7

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		abi_	T340544 Enable MinT on translatewiki.net
		Resolved	BUG REPORT	abi_	T340526 Translating strings that have newlines produces unexpected output

Event Timeline

abi_ created this task.Jun 27 2023, 9:01 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 27 2023, 9:01 AM

abi_ added a project: ContentTranslation.Jun 27 2023, 9:02 AM

Pginer-WMF added a project: Language-Team (Language-2023-July-September).Jun 27 2023, 10:32 AM

abi_ added a parent task: T340544: Enable MinT on translatewiki.net.Jun 27 2023, 11:48 AM

Is this random or is there a case where this is consistantly happening?
I tried a few cases as below and output seems complete

curl -X 'POST' \
  'https://cxserver.wikimedia.org/v1/mt/en/hi/MinT' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "html": "The value for a parameter is required.\nTo get full help for the command, send {1}help {0}."
}'

Output: {"contents":"किसी पैरामीटर के लिए मान आवश्यक है ।\nकमांड के लिए पूरी मदद पाने के लिए { 1 ] हेल्प { 0 ] भेजें ।"}

curl -X 'POST' \
  'https://cxserver.wikimedia.org/v1/mt/en/hi/MinT' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "html": "The stream goal (page title or namespace number) is required.\nTo get full help for the command, send {1}help {0}."
}'

Output: {"contents":"स्ट्रीम लक्ष्य (पृष्ठ शीर्षक या नेमस्पेस संख्या) की आवश्यकता है ।\nकमांड के लिए पूरी मदद पाने के लिए { 1 ] हेल्प { 0 ] भेजें ।"}

curl -X 'POST' \
  'https://cxserver.wikimedia.org/v1/mt/en/hi/MinT' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "html": "Streams are available only for Wikimedia projects.\nList of allowed domains: {0}."
}'

Output: {"contents":"धाराएँ केवल विकिमीडिया परियोजनाओं के लिए उपलब्ध हैं ।\nअनुमत डोमेन की सूचीः{ 0 ]"}

Pginer-WMF moved this task from Quarter Backlog to Priority: Translation on the Language-Team (Language-2023-July-September) board.Jun 30 2023, 11:40 AM

Pginer-WMF moved this task from Backlog to General translation functionality on the MinT board.Jul 3 2023, 11:16 AM

Nikerabbit triaged this task as Medium priority.Jul 3 2023, 11:17 AM

Nikerabbit removed a project: ContentTranslation.Jul 3 2023, 11:33 AM

abi_ updated the task description. (Show Details)Jul 3 2023, 6:41 PM

Change 935149 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/Translate@master] MinT: Stop pre/post processing input/output sent to API

https://gerrit.wikimedia.org/r/935149

Change 935150 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/Translate@master] MachineTranslationAid: Avoid overriding variables

https://gerrit.wikimedia.org/r/935150

Thanks, I went back to the code on the Translate end and found a few underlying issues that I've submitted patches to fix. This is what was causing the issues with newlines.

One issue that I do notice with MinT is handling of text within apostrophe. Eg:

`{1}help {0}`

In such cases it seems to remove the variables within the apostrophe. Not a huge deal but I've updated the task description.

Change 935150 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] MachineTranslationAid: Avoid overriding variables

https://gerrit.wikimedia.org/r/935150

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.18; 2023-07-18).Jul 11 2023, 3:00 PM

abi_ renamed this task from Translating strings that have newlines produces unexpected output to Translating strings that have apostrophe produces unexpected output.Sep 1 2023, 1:19 PM

abi_ updated the task description. (Show Details)

Pginer-WMF edited projects, added Language-Team (Language-2023-October-December); removed Language-Team (Language-2023-July-September).Oct 2 2023, 9:02 AM

Pginer-WMF moved this task from Quarter Backlog to Priority: Translation on the Language-Team (Language-2023-October-December) board.

Since the initial issue with newlines is solved, I'm marking the task as done. I'll create another task to deal with apostrophe.

Translating strings that have newlines produces unexpected outputClosed, ResolvedPublicBUG REPORTActions

Description

Details

Related ObjectsSearch...

Event Timeline

Translating strings that have newlines produces unexpected output
Closed, ResolvedPublicBUG REPORT
Actions

Related Objects
Search...