Automated localization build tool

Project Name

Automatic Localization Fork Tool - This script, given a locale and a set of rules, will create an l10n source tree of the same language but different region (e.g. en-IN from en-GB)

• THE BUG - Bug 403215 – Python script that will help create from a translated locale a regional build
• THE BUG - Bug 399014 – we need an l10n-merge tool

How to Add to Tool & Pick It Up as a Project

For those wondering how they can pick this project up and run with it, please read the following tips:
The code is broken into three main components

• callback() is where you would add your new functionality and subsequently update the "instring" variable before closing the localization file. This method is the most important part of the program and is essentially the core from which all localization logic is derived from. I added much in-code documentation to guide newcomers along the way.
• translate() is where you tell the program what files to look for and subsequently what action Parser.py (the class supplied by Axel) should take, such as use a DTD Parser or use a Properties parser and so on...
• process() contains the file iteration logic whereby the program scans every file in every directory. There is potential for optomization here and Dynamis (Tomoya Asai, a Mozilla employee from Japan) can definately help steer you in the right direction as this method was inspired from reading the logic in Dynamis' l10n-merge script.
• main() is where you would alter the program's main interface, adding UI code in here is a sure possibility.
• The #l10n IRC Channel on Moznet has a wealth of people who are knowledgable in this area and can provide much information if asked
• Regular Expressions are you friend. Do what I did, go print out every copy of Regular Expression theory in Pyhon as you can, cause you will need it
• Mozilla MXR is your gateway to finding out where you will be looking to localize specific functionality or visual items

Project Plan & Description

Project Description

The l10n Auto Fork – formerly known as the Automated Localization Build Tool – is a Python-based tool that, given a locale and set of rules, creates an l10n source tree similar to the original as far as the language, yet modified to be used in a specific region. A practical example would be to use an en-US source tree and run the l10n auto fork tool to produce an en-CA source tree. The tool would, among other things, modify the default RSS feeds, dictionary, as well as the search engines to the target locale. Essentially, the auto fork tool will generate a localization template - which would be in the form of an XML, DTD, or Properties document - that “localizers” can use to enter the localized version of each string, date, currency, or other form of data. When this template is provided to the tool along with a source tree (such as en-GB), the tool outputs the desired localized source tree.

Project Plan

There are seven main points to address on the road that takes this project from its current 0.3 all the way to an 1.0 release. Along the way I plan to keep in touch with Mic and Axel as well as the rest of the localization community through IRC, email, phone, and Bugzilla. The table below outlines what my goals for each bi-weekly release are right up to the final 1.0 release.

Release	Download Release	Features/Points Addressed
0.4	Download	- All regular expressions, and related functions removed and replaced in existing code. Optimize existing code (the current directory “walk” methods can be re-factored). Add new command line options. For example, an option to generate a localization template (won't be fully functional yet, completed at future release). Another option would be to have verbose mode on/off for the tool so localizers can see what files are being localized in real-time. (DONE)
0.5	Download	- Complete fixing outstanding bugs. (DONE) Help command (will display all command line arguments and switches that can be used with tool). (DONE IN 0.4). Validation code. (DONE) Generate a template as well as settle on a specific file format (dtd, properties, or XML). (DONE)
0.6	Download	- Complete localization template functionality. Will provide “localizers” with a hassle free way of altering strings/dates/currency/default search engines/default RSS, to target locale.(DONE)
0.7.2	Download	- Add feature that will allow to modify default search engines to target locale.(DONE). Test on full source tree - worked, ScreenShots here (DONE)
0.8.1	Download	- Add feature that allows to change default dictionary (DONE).
0.9	Download	- Seperate browser Strings & Labels localization template from browser Search Defaults localization template(DONE) Fixed a bug (if the first char in KEY was different from first char in VALUE, localization changes would not take place)(DONE) Improved code's readability(DONE)
1.0	Download	- Find way to potentially optomize localization process. Also, add functionality so localization file for search engines is auto generated. Added documentation on this wiki on how students or community can pick up this project and continue it.(DONE)

Project Leader(s)

• Rueen Fiez (rueen) Rueen's blog - (Point of Contact)
• Vincent Lam (vlam6) Vince's blog
• Armen Zambrano (armenzg) Armen's blog about the project

Releases

Release notes

• 1.0 Release(DONE)
  • Find way to potentially optomize localization process. Also, add functionality so localization file for search engines is auto generated
• 0.9 Release
  • Seperate browser Strings & Labels localization template from browser Search Defaults localization template
  • Fixed a bug (if the first char in KEY was different from first char in VALUE, localization changes would not take place)
  • Improved code's readability
• 0.8 Release
  • Add feature that allows to change default dictionary
• 0.7 Release
  • Add feature that will allow to modify default search engines to target locale.
  • Test on full source tree - worked, ScreenShots here.
• 0.6 Release
  • Complete localization template functionality. Will provide “localizers” with a hassle free way of altering strings/dates/currency/default search engines/default RSS, to target locale.
• 0.5 Release
  • Complete fixing outstanding bugs
  • Validation code
  • If time permits, get the template generator functional. (That way its one less thing todo for 0.7 release)
• 0.4 release
  • Remove regex code and supporting methods
  • Optimize existing methods such as the directory walk method. (Small thing but necessary nonetheless)
  • Add new command line options such as an option to generate localization template and an option to turn on tool's verbose mode
• 0.3 release
  • Use of command line arguments rather than receiving user's input
  • Fix the key changing problem
  • Add more words to change in an l10n tree. But they are hard-coded :(
  • Stop using .bak file (and make the changes permanent)
  • Generate a second l10n tree out of original with the changes
• 0.2 release
  • Given any directory as a starting point, should walk through all sub-directories and files and make changes based on translation rules
  • Add ability to update Properties files
  • Fix key changing problem (In progress/May go to 0.3 release)
  • Add more "rules", for now we are testing by changing color --> colour (May go to 0.3 release)
  • Allow user to enter localization folder (eg; en-GB) as input

• 0.1 release
  • Should be able to accept a localization
  • Should be able to accept an l10n tree(eg; en-GB or en-US)
  • Read through every DTD and Properties file in the current directory with the "Parser.py" file
  • Changes the word "color" to "colour" in every DTD file and have it saved

Project news

There are some common news from the collaborators that should be written here rather than splitting it between the collaborators:

• Apr. 18, 2008 - Release 1.0 is now available, localizing process is now optomized to be completed faster. New functionality also added, ability to automatically generate a localization template for the default search engines with predefined keys
• Apr. 6, 2008 - Release 0.9 - Big change to the way localizer enters their changes. They must now enter String/Label localization changes in the "locale_template.properties" file and enter Search Default changes in the "locale_template_searches.properties" file. This seperation increases readability and avoids confusion when entering many localization values. Also fixed a critical bug found related to the first char of KEY being different than the first char of VALUE in properties files.
• Mar. 23, 2008 - Release 0.8 - tool will now alter values in Firefox's default dictionary (this means even the spellchecker in Firefox will localize accordingly)
• Mar. 9, 2008 - Updated 0.7 to 0.7.2 after running tests on a full source tree. Screenshots available here.
• Feb. 29, 2008 - Release 0.7 is done. I added the ability to change the locale of the default search engines - located in /browser/locales/en-US/searchplugins/.
• Feb. 20, 2008 - Release 0.6 is completed. The tool is now able to make changes dynamically based on the Key/Value pairs located in the Localization Template .properties file. This file must be located in the same directory as the tool itself for the tool to be able to output a new localized source tree.
• Jan. 29, 2008 - Updated Bug 403215 with information regarding latest release as well as plans for future releases. Updated release features for 0.5 and 0.7.
• Jan. 20, 2008 - Release 0.4 is completed. Program has now been turned into a real command line tool. Added several command line options. Can view them all by running "l10nFork.py -help". Fixed a MAJOR bug left over from 0.3.2, the localized changes were not taking place, bug was found in the callback() function. Removed/Commented-out all Regex, left one since need it for testing purposes right now. By 0.7 Release, 100% of regex in callback() function will be removed.
• Jan. 10, 2008 - Project's roadmap from 0.3 release to a 1.0 release posted. New releases will be available on a bi-weekly basis. Outline of new features for each new release available in the Project Plan & Description section. Many general Wiki page updates. The tool is now called the Automated Localization Fork tool, or l10n Auto Fork for short.
• Nov. 11, 2007 - Updated the contributions page.
• Nov. 10, 2007 - l10nMerge tool updated from 0.2.2 to 0.2.3 and available for download under the 0.2 Release Functionality & Features section. Added new feature (allow user to directly input localization [eg; en-GB] ), added lots of code documentation and debug sections.
• Nov. 9, 2007 - Created new bug (403215) that will guide the discussion about our tool
• Oct. 31, 2007 - It seems that a lot of things are going to be talked around l10n tools in general (" L10n tools talking" in Google Groups) and specifically in the bug (Bug 399014) "related" (between brackets) to our project. We will have to read dynamis' (he has finally appeared, he is called Asai Tomoya) code and see what the new direction of our script should take.

WE ARE A LITTLE CONFUSED BUT AFTER THE WEEKEND SOME LIGHT MIGHT DROP INTO THE PROJECT (Done)

• Oct. 30, 2007 - Axel has replied on the bug and it seems that our project shouldn't be called l10n merge, the bug seems to be for another reason. Some doubts arise concerning what our tool should really be doing.
• Oct. 25, 2007 - Meeting with Michal; She explained us the different tools that out there created by the community and by next week will gives us more information on which direction to follow
• Oct. 23, 2007 - Added 0.2 Release Functionality & Features section. Team meeting held (6 hours) - our tool is now able to, given a localization directory (or any directory for that matter), walk through all sub-directories and files and make changes based on our translation rules. We've also determined a few other things to do before our 0.2 release and even brainstormed 0.3 ideas (incorporating l10n's setup.py script into our tool like Axel suggested).
• Oct. 12, 2007 - Updated 0.1 Release Functionality & Features section. A lot of the 0.1 code has been done.
• Oct. 07, 2007 - Added an 0.1 Release Functionality & Features section to the wiki so we have a clear description of what our project's 0.1 release should be able to do.
• Oct. 05, 2007 - Python will be our language of choice for this project which is a great opportunity to thoroughly learn it since it will be our first time using it. Determined some main tasks ahead of us before 0.1 release (tasks mentioned in Product Description).
• Sep. 24, 2007 - We are going to have a call conference with Michal from Toronto office

Project Contributor(s)

The following people contributed to our project in some beneficial manner. We appreciate all advice received from the Mozilla l10n community, as well as Seneca students, and professors. If you would like to be contributor to our project, just get in touch with us through email, or seneca's IRC channel. The Contribution Info sub-section below outlines our current project needs. Potential contributors can pick one and we can talk about how to proceed.

Potential Ways of Contributing

Python regular expressions help Bug: When parsing files that use key/value pairs - DTD or Properties - our tool not only changes the value but the key as well. We don't want the key to be changed.

1. Testing our script when needed. (Usually right after a release)

• Timothy Joseph Duavis
  • Reviewed our Python script(s) and provided some ways to improve several methods. In particular, the process() and callback() methods.

• Vijey Balasundaram
  • Tested our tool and looked for bugs or any potential future pitfalls we might run into. Helped with some documentation for first time tool users.

• Mozilla l10n Community
  • Provided strong leadership and support. We wouldn't be able to get this project off the ground without them.

• Seneca Professors
  • Put us in contact with the Mozilla l10n community and helped us get this project off the ground through structure and guidance.

Contribution Info

• I would like to apply to a string multiple regular expressions substitutions. I have seen something like re.compile(bla bla bla). Could I collect a set of regular expressions in an array an iterate through it applying the substitutions to a string?? or something like that?
• Timothy Joseph Duavis (here to be of service!) (Done)

Other Project Details

• Our script for now; We will be also posting on the bug 399014 - Get our code from the latest release
• We are also awaiting for some code that dynamis has been working on in Japan - dynamys seems to be lost in combat - I have heard he might me moving to another location and that might be he has been difficult to contact him
• Notes from Axel(pike) about the project
• Team notes - we collect notes related to the project
• Armen's MozDev process - diary - You can read notes of what Armen has been trying
• The l10n tools are in mxr.m.o/mozilla/source/testing/l10n
• How to check out the l10n tools using CVS:

  cvs -z3 -d:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/tools/l10n mozilla/testing/tests/l10n

• The file we have been using: Parser.py using the DTDParser class and the PropertiesParser class. These both are derived from the Parser class.
• mozilla/tools/l10n/l10n.py might give you an idea of what it takes to copy existing data over to a new location
• To get the l10n tools type: $> cvs -z3 -d:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/tools/l10n mozilla/testing/tests/l10n

• Some notes from trying to get en-GB (read more):

* make -f client.mk l10n-checkout MOZ_CO_PROJECT=browser MOZ_CO_LOCALES=en-GB LOCALES_CO_TAG=HEAD
* An option for the .mozconfig: mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../en-GB

• To get en-US:

* 

• To check the completness of your localization //takes long

* make -f tools/l10n/l10n.mk check-l10n

• To fill out what is missing in the source of your localization

* MOZ_CO_PROJECT=browser make -f tools/l10n/l10n.mk create-en-GB

Regular expressions

• http://docs.python.org/lib/re-syntax.html
• http://www.amk.ca/python/howto/regex/
• Change we should make to a localization source code(I'm not sure about the regular expressions):

>> color(s) -> colour(s) -- re.sub(r'([Cc])olor', r'\1olour', instring)
>> dialogue  -> dialog -- re.sub(r'([Dd])ialogue', r'\1ialog', instring)
>> Go forward -> Go forwards -- 
>> Minimize -> Minimise
>> Center -> Centre
>> Organize -> Organise
>> Customize -> Customise

DTD regular expression analysis

• An analysis of regular expression, specific to DTDs (from Parser.py)

self.key = re.compile('<!ENTITY\s+([\w\.]+)\s+(\"(?:[^\"]*\")|(?:\'[^\']*)\')\s*>', re.S)

  \s+ - one or more (??) blank spaces, tabs, end of line, and others whitespace elements
  ([\w\.]+) - one or more alphanumeric characters and/or(??) a dot
  \s+ - more of the same
  (\"(?:[^\"]*\")|(?:\'[^\']*)\') - difficult part
        if the left of '|' matches,the right part doesn't get analyzed
    (\"(?:[^\"]*\") - matching something in between " and "
         (?:[^\"]*\") - everything after "?:" if matched cannot be referenced
             not a backslash or a " - the * indicate zero or more - RIGHT???
    (?:\'[^\']*)\') - the right side of the difficult reg. expression

  \s* - none or more white characters
  re.S - makes the dot to match even new lines - it is like raising DOTALL  flag

Example of matching line:
  "<!ENTITY  colorsDialog.title              "Colors">"

Related regular expressions theory

* (...) what is inside the parentheses are a group - the contents of a group can be retrieved after a match has been performed, 

and can be matched later in the string with the \number special sequence

* (?...) - This is an extension notation - Extensions usually do not create a new group; (?P<name>...) is the only 

exception to this rule. Following are the currently supported extensions.

* List of supported extensions: (?iLmsux), (?:...), (?P<name>...), (?P=name), (?#...), (?=...), (?!...), (?<=...), 

(?<!...), (?(id/name)yes-pattern|no-pattern)

* \number - Matches the contents of the group of the same number. Groups are numbered starting from 1.

Bugs

• THE BUG - Bug 403215 – Python script that will help create from a translated locale a regional build
• THE BUG - Bug 399014 – we need an l10n-merge tool
• Bug 398954 – l10n build automation doesn't pick the right en-US source for the build
• Bug 391680 – Make the python compare-locales work with suite and calendar
• Bug 345039 en-CA English (Canada): Firefox - please read comment 16
• Bug 392945 (fx20-en-IN) – add en-IN locale

Old want to help

• We have a problem when matching a pattern inside of the DTDs, look below:

re.sub(r'([Cc])olor', r'\1olour', instring) Changes every appearance of 'Color' or 'color' with 'Colour' and 'colour' (the Canadian wa) Therefore a line like: <!ENTITY colorsDialog.title "Colors"> becomes like this: <!ENTITY coloursDialog.title "Colours"> WE DON'T WANT coloursDialog.title!!! A better regular expression could avoid this but we are not good yet with RegExps (check anlaysis on RegExprs on next section) SOLVED: re.sub(r'(.*)([Cc])olor', r'\1\2olour', instring)

• We would like to walk into subfolders and make changes in each subfolder; any good python example?

SOLVED: Already works on 0.2 release

• Learn python
• Understand the scripts from the test l10n tools
• Understand the l10n build system
• Reproduce en-IN from en-GB
• Determine what our Python based system will "do" in 0.1 release