Plone's gettext approach and its impact on translation

[Roché Compaan]

Find below some feedback from Dwayne Bailey on Plone's gettext approach.
I've copied him again so he stays part of any discussion that might
follow.

--
Roché Compaan
Upfront Systems                   http://www.upfrontsystems.co.za

-------- Forwarded Message --------
From: Dwayne Bailey <[hidden email]>
To: Roché Compaan <[hidden email]>
Subject: Plone's gettext approach and its impact on translation
Date: Mon, 25 Oct 2010 11:40:41 +0200

Hi,

I'm doing some i18n work on a Plone application.  It is the very first
time that I look in detail at Plone localisation and I have some
concerns that I wanted to share with you.

Plone uses the msgid as key approach to Gettext instead of the msgid is
source text approach.

While this approach is not incorrect, the Gettext manual itself talks
about this approach.  The reality is that nobody really uses it in the
wild. So unless you are a Plone only translator this is a foreign
format.

The msgid is key approach usually creates issues in that the source text
is not defined, Plone works around the lack of source text by pushing
the source text into a developer comment.  But PO tools aren't really
designed to support this.  In most tools the comment will be placed in a
very different place to what translators are used to.  So you
essentially mess with a translators normal workflow.

Also, since the source text is now in a comment all of the support
infrastructure that a translator uses in their tools, Translation
Memory, Machine translation, terminology lookup, etc, are lost.  These
tools expect source text to be in the msgid so they are going to lookup
the key not the source text. The translator will not get any results
from these lookups. So essentially you put your translators at a
disadvantage, you make them work harder then they need to with the very
serious possibility of increasing translation error rates.

What makes it worse is that this key approach is not applied
consistently.  In cases where a key was not defined the text itself is
used. So now we need translators to be able to determine when they must
translate the key and when they must translated the comment!  This just
makes them slower as they need to interpret the text by looking at both
areas then decide what action to take.

From a translators perspective there are no advantages to the
monolingual Gettext approach.  None of our tools support this. If I told
a programmer to code in Notepad would they be happy?  No code
completion, no lookups, no syntax highlighting, nothing. It's
essentially what you are asking Plone localiser to do, "Just use a text
editor and drop your fancy PO editor".

So maybe I can pose some questions to the i18n developers.

1) Is there a plan to move to a more standard Gettext approach?
2) Are there plans to eliminate the confusing mix of key and source?
3) Has anyone written a fixplone script that converts Plone monolingual
PO files to a proper bilingual PO file?

Thanks for your help

--
Dwayne Bailey
Associate             Research Director        [phone] (w)
Translate.org.za      ANLoc                    [phone] (c)

Recent blog posts:
* Localizing Mac OS X strings files using open source PO editors
http://www.translate.org.za/blogs/dwayne/en/content/localizing-mac-os-x-strings-files-using-open-source-po-editors
* What's new in Virtaal 0.6.1
* Localisation: How we guess the target translation language in Virtaal

Firefox web browser in Afrikaans - http://af.www.mozilla.com/af/
African Network for Localisation (ANLoc) - http://africanlocalisation.net/






------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[Hanno Schlichting-4]

On Mon, Oct 25, 2010 at 11:54 AM, Roché Compaan
<[hidden email]> wrote:
> -------- Forwarded Message --------
> From: Dwayne Bailey <[hidden email]>

[...] snip, we are aware of all these concerns

> So maybe I can pose some questions to the i18n developers.
>
> 1) Is there a plan to move to a more standard Gettext approach?

Plans, sure, for years. Somebody stepping up and actually doing it, not so far.

We track this under the topic of supporting the "new" msgctxt that was
added to gettext to solve the use-cases that lead us to use our own
msgid as key approach. Adding support for this requires some API
changes to various parts of our i18n stack, so it's not trivial and
needs to deal with backwards compatibility concerns. I have started
some code for this with zope.i18nmessage and zope.babel, but these are
early prototypes.

> 2) Are there plans to eliminate the confusing mix of key and source?

Yes. We have started to introduce new messages generally without the
explicit id and use the standard text as the msgid. We haven't done
this for all existing messages as this would be a major undertaking.
There are places where we do need to differentiate words on the
context (and not just the domain), so we cannot go all the way without
msgctxt support.

> 3) Has anyone written a fixplone script that converts Plone monolingual
> PO files to a proper bilingual PO file?

I believe such scripts where contributed and are part of plone.app.locales.

Hanno

------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[Maurits van Rees-3]

Op 25-10-10 11:54, Roché Compaan schreef:
> Find below some feedback from Dwayne Bailey on Plone's gettext approach.
> I've copied him again so he stays part of any discussion that might
> follow.

Hi Dwayne,

Thanks for sharing your thoughts and concerns with us.  I will give my
perspective as programmer.

I am under the impression, that using msgids is preferable to using the
source text directly, at least when this source text is a bit long.
That way, a restructuring of the original text, or simply fixing a typo,
does not lead to fuzzy translations.

When you have this:

# Default: Hello Worlt
msgid "msg_hello_world"
msgstr "Hallo wereld"

and someone fixes the typo in the original English and regenerates the
po files, you get this:

# Default: Hello World
#, fuzzy
msgid "msg_hello_world"
msgstr "Hallo wereld"

Now the translator only needs to look at the new default, conclude that
the existing translaton is still fine and remove the fuzzy remark.
Compare that to this:

msgid "Hello Worlt"
msgstr "Hallo wereld"

After fixing the typo in the Plone code and syncing with the po file,
this would become:

msgid "Hello World"
msgstr ""

and the translator needs to translate the string completely new.  Not
hard for these two words, but it gets annoying when you have long
sentences to translate.

(Okay, if I notice this when fixing the typo, I try to make sure I
simply fix the typo in the po files as well, so that neither a fuzzy
remark gets added, nor msgstrings removed.)


As for special po-editors: I am fine with using a normal text editor
---as far as you can call Emacs normal ;-)--- but then I am a programmer.


I won't defend how we are doing things, but as a programmer who only has
experience with how translations are done in Plone, I am fine with the
current situation.

That said, it might be that we are inadvertently scaring away competent
translators by the way we are handling things now, so that translations
are left to programmers, who might use language that scares away end
users or might not translate at all.


--
Maurits van Rees
Programmer, Zest Software


------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n

[Hanno Schlichting-4]

On Tue, Oct 26, 2010 at 11:10 AM, Maurits van Rees
<[hidden email]> wrote:
> I am under the impression, that using msgids is preferable to using the
> source text directly, at least when this source text is a bit long.
> That way, a restructuring of the original text, or simply fixing a typo,
> does not lead to fuzzy translations.

This specific problem is one of the two reasons we adopted for our
msgid approach. It solves a problem you get if you don't use any
sophisticated tools for doing translations, like it is the case for
most developers.

But it actually does hurt all translators using real tools, which have
translation memories to solve this. If you have a translation memory,
that's the place where you keep the old translation. If you give a
translator any new or changed string, they'll ask their translation
memory to pull out any fuzzy matches from it and then adjust any
mismatching bits. These translation memories assume that the msgid is
the real text and do the matching based on it.

I think we should change the approach to maybe preserve the old text
in an extra comment, so developers using no tools still see it, but we
don't confuse the real tools.

Hanno

------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
Plone-i18n mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-i18n