View source for Automated localization build tool

== Project Name ==
''automated localization tool'' - This tool, given a locale and a set of rules, create a build of the same language but different region (e.g. en-IN from en-GB)

== Project Description ==
* '''THE BUG''' - [https://bugzilla.mozilla.org/show_bug.cgi?id=399014 Bug 399014 – we need an l10n-merge tool]
* Learn python
* Understand the scripts from the test l10n tools
* Understand the l10n build system
* Reproduce en-IN from en-GB
* Determine what our Python based system will "do" in 0.1 release

== Tool's 0.2 Release Functionality & Features ==
* '''INSTRUCTIONS''': ''TBD''
* '''Tool - 0.2 release ZIP file''': ''TBD''

#Given any directory as a starting point, should walk through all sub-directories and files and make changes based on translation rules ('''Done''')
#Add ability to update Properties files ('''Done''')
#Fix key changing problem
#Add more "rules" (for now we are testing by changing ''color'' --> ''colour'')

== Tool's 0.1 Release Functionality & Features ==
* '''INSTRUCTIONS''': [http://matrix.senecac.on.ca/~azambran/mozilla/instructions0.1.txt TXT file]
* '''Tool - 0.1 release ZIP file''': [https://bugzilla.mozilla.org/attachment.cgi?id=284789 The Zip file filed on the bug]

#Should be able to accept a localization ('''Done''')
#Should be able to accept a Firefox build (eg; en-GB or en-US) ('''Done''')
#Read through every DTD and Properties file in the current directory with the "Parser.py" file ('''Done''')
#Changes the word "color" to "colour" in every DTD file and have it saved ('''Done''')

== Want to help??? ==

* Our project can benefit from anyone who is knowledgeable in Python regular expressions. Feel free to contact us if you would like to contribute to the project in this way.

* We have a problem when matching a pattern inside of the DTDs, look below:
 re.sub(r'([Cc])olor', r'\1olour', instring)
 Changes every appearance of 'Color' or 'color' with 'Colour' and 'colour' (the Canadian wa)
 Therefore a line like:
 * <!ENTITY  colorsDialog.title              "Colors">
 becomes like this:
 * <!ENTITY  coloursDialog.title              "Colours">
 WE DON'T WANT colo'''u'''rsDialog.title!!!
 A better regular expression could avoid this but we are not good yet with RegExps (check anlaysis on RegExprs on next section)

* We would like to walk into subfolders and make changes in each subfolder; any good python example?

== Project Leader(s) ==
*[[User:Rueen|Rueen Fiez]] (rueen) [http://rueenfiez.wordpress.com Rueen's blog]
*[[User:Vlam6|Vincent Lam]] (vlam6) [http://vlam6.wordpress.com Vince's blog]
*[[User:Armenzg|Armen Zambrano]] (armenzg) [http://armenzg.blogspot.com Armen's blog]

== Project Details ==
* Our script for [http://docs.google.com/Doc?id=dnkkbhp_65p4whj5 now]; We will be also posting on the bug 399014
* We are also awaiting for some code that dynamis has been working on in Japan
* [http://docs.google.com/Doc?id=dnkkbhp_36gps298 Notes from Axel(pike)] about the project
* [http://docs.google.com/Doc?id=dnkkbhp_35gscqgh Team notes] - we collect notes related to the project
* [http://docs.google.com/Doc?id=dnkkbhp_32ds8tz7 Armen's MozDev process] - diary - You can read notes of what Armen has been trying
* The l10n tools are in [http://mxr.mozilla.org/mozilla/source/testing/tests/l10n/ mxr.m.o/mozilla/source/testing/l10n]
* The file we have been using: [http://mxr.mozilla.org/mozilla/source/testing/tests/l10n/lib/Mozilla/Parser.py Parser.py] using the DTDParser class and the PropertiesParser class. These both are derived from the Parser class.
* [http://mxr.mozilla.org/mozilla/source/tools/l10n/l10n.py mozilla/tools/l10n/l10n.py] might give you an idea of what it takes to '''copy existing data over to a new location'''
* To get the l10n tools type: '''$> cvs -z3 -d:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/tools/l10n mozilla/testing/tests/l10n'''


* Some notes from trying to get en-GB ([http://armenzg.blogspot.com/2007/10/session-6-trying-to-build-en-gb.html read more]):
 * '''make -f client.mk l10n-checkout MOZ_CO_PROJECT=browser MOZ_CO_LOCALES=en-GB LOCALES_CO_TAG=HEAD'''
 * An option for the .mozconfig: '''mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../en-GB'''
* To get en-US:
 * 
* To check the completness of your localization //takes long
 * make -f [http://mxr.mozilla.org/mozilla/source/tools/l10n/l10n.mk tools/l10n/l10n.mk] '''check-l10n'''
* To fill out what is missing in the source of your localization
 * MOZ_CO_PROJECT=browser make -f tools/l10n/l10n.mk '''create-en-GB'''

=== Regular expressions ===
* [http://docs.python.org/lib/re-syntax.html http://docs.python.org/lib/re-syntax.html]
* [http://www.amk.ca/python/howto/regex/ http://www.amk.ca/python/howto/regex/]
* Change we should make to a localization source code(I'm not sure about the regular expressions):
 >> color(s) -> colour(s) -- re.sub(r'([Cc])olor', r'\1olour', instring)
 >> dialogue  -> dialog -- re.sub(r'([Dd])ialogue', r'\1ialog', instring)
 >> Go forward -> Go forwards -- 
 >> Minimize -> Minimise
 >> Center -> Centre
 >> Organize -> Organise
 >> Customize -> Customise

=== DTD regular expression analysis ===
* An analysis of regular expression, specific to DTDs (from Parser.py)
 self.key = re.compile('<!ENTITY\s+([\w\.]+)\s+(\"(?:[^\"]*\")|(?:\'[^\']*)\')\s*>', re.S)

   '''\s+''' - one or more (??) blank spaces, tabs, end of line, and others whitespace elements
   '''([\w\.]+)''' - one or more alphanumeric characters and/or(??) a dot
   '''\s+''' - more of the same
   '''(\"(?:[^\"]*\")|(?:\'[^\']*)\')''' - difficult part
         if the left of '|' matches,the right part doesn't get analyzed
     ''(\"(?:[^\"]*\")'' - matching something in between " and "
          (?:[^\"]*\") - everything after "?:" if matched cannot be referenced
              not a backslash or a " - the * indicate zero or more - RIGHT???
     ''(?:\'[^\']*)\')'' - the right side of the difficult reg. expression
 
   '''\s*''' - none or more white characters
   '''re.S''' - makes the dot to match even new lines - it is like raising DOTALL  flag

 Example of matching line:
   "<!ENTITY  colorsDialog.title              "Colors">"

=== Related regular expressions theory ===
 * (...) what is inside the parentheses are a group - the contents of a group can be retrieved after a match has been performed, and can be matched later in the string with the ''\number'' special sequence
 * (?...) - This is an '''extension notation''' - Extensions usually do not create a new group; (?P<name>...) is the only exception to this rule. Following are the currently supported extensions.
 * List of supported extensions: (?iLmsux), (?:...), (?P<name>...), (?P=name), (?#...), (?=...), (?!...), (?<=...), (?<!...), (?(id/name)yes-pattern|no-pattern)
 * \number - Matches the contents of the group of the same number. Groups are numbered starting from 1.

== Project news ==
There are some common news from the collaborators that should be written here rather than splitting it between the collaborators:
* Sep. 24, 2007 - We are going to have a call conference with Michal from Toronto office
* Oct. 05, 2007 - Python will be our language of choice for this project which is a great opportunity to thoroughly learn it since it will be our first time using it. Determined some main tasks ahead of us before 0.1 release (tasks mentioned in [http://zenit.senecac.on.ca/wiki/index.php/Automated_localization_build_tool#Project_Description Product Description]).
* Oct. 07, 2007 - Added an 0.1 Release Functionality & Features section to the wiki so we have a clear description of what our project's 0.1 release should be able to do.
* Oct. 12, 2007 - Updated 0.1 Release Functionality & Features section.  A lot of the 0.1 code has been done.
* Oct. 23, 2007 - Added 0.2 Release Functionality & Features section. Team meeting held (6 hours) - our tool is now able to, given a localization directory (or any directory for that matter), walk through all sub-directories and files and make changes based on our translation rules. We've also determined a few other things to do before our 0.2 release and even brainstormed 0.3 ideas (incorporating l10n's setup.py script into our tool like Axel suggested).

== Bugs ==
- There are no bugs related specifically to the project, but the following were mentioned in our conversations
* * '''THE BUG''' - [https://bugzilla.mozilla.org/show_bug.cgi?id=399014 Bug 399014 – we need an l10n-merge tool]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=345039 Bug 345039 en-CA English (Canada): Firefox] - please read comment 16
* [https://bugzilla.mozilla.org/show_bug.cgi?id=392945#c8 Bug 392945 (fx20-en-IN) – add en-IN locale]

== Project Contributor(s) ==