Difference between revisions of "Continuous Integration"

From CDOT Wiki
Jump to: navigation, search
 
(20 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
{{Admon/obsolete}}
 +
 
<big><big> Continuous Integration Project for NexJ Express Code</big></big>
 
<big><big> Continuous Integration Project for NexJ Express Code</big></big>
 
{{NexJ_Express Index}}
 
{{NexJ_Express Index}}
Line 4: Line 6:
  
 
==Project goal==
 
==Project goal==
−
Currently, NexJ has a Mercurial Repository (Internal) in which, the NexJ Express Server (Core) code is maintained with all its histories. NexJ is planning to share a Core Repository with the Open Source Community by creating a Repository (External) that will be kept in sync with the Internal Repository.
+
Currently, NexJ has a Mercurial Repository (Internal) in which, the NexJ Express Server (Core) code is maintained with all its histories. NexJ is planning to share a Core Repository with the Open Source Community by creating a Repository (External) that will be kept in sync with the Internal Repository.<br/>
−
== Current Status: Project Completed==
+
 
 +
=== Contributors ===
 +
* [[Minooz/Projects | Minoo Ziaei]]
 +
* [[User:Gbatumbya|Grace Batumbya]]
 +
 
 +
 
 +
== Current Status==
 +
<big><big>Project Completed</big></big>
 
<br/>
 
<br/>
 +
  
 
==Project Phases==
 
==Project Phases==
Line 12: Line 22:
 
* Get requirements from NexJ
 
* Get requirements from NexJ
 
* Incorporate requirements from NexJ into Design of algorithm for syncing repositories
 
* Incorporate requirements from NexJ into Design of algorithm for syncing repositories
−
* Provide the proposal for review and approved
+
* Provide the proposal for review and approved - [[Continuous_Integration/proposal | final version of proposal]] ,  [https://docs.google.com/document/d/17Fuj_9A7sBkzlbP0t0bOY3Qs1iiAhKAvfNTuUMnHUcg/edit?hl=en_US  Google Doc ]
−
[[Continuous_Integration/proposal | proposal]]
 
−
 
 
  
 
===Phase 2: Creating Similar Environment (Done) ===
 
===Phase 2: Creating Similar Environment (Done) ===
 
* Internal Repository
 
* Internal Repository
−
** Created a Mercurial repository and added some changes to it to grow the history of it.
+
** Creating a Mercurial repository and adding some changes to it to grow the history of it.
−
 
 
 
* Controller
 
* Controller
 
** This will be containing temporary repositories as the result of scripting.  
 
** This will be containing temporary repositories as the result of scripting.  
 +
* External Repository
 +
** Creating another Mercurial repository using 'hg' commands from the latest 'change-set' of Internal Repository.
  
 +
===Phase 3: Scripting (Done) ===
 +
* Ant
 +
** Doesn't support some of non-regular Mercurial commands that we need to play with ''change-sets''
 +
* Bash
 +
** Supports everything we need for the project. [[Continuous_Integration/script | script]]
  
−
* External
+
===Phase 4: Running the script on CI system (Done) ===
−
** Created a Mercurial repository using 'hg' commands from the latest 'change-set' of Internal Repository.
+
* Buildbot
 +
* Hudson
 +
** Simple to setup and configure.
 +
** Runs different types of scripts: ''bash'', ''ant'', ''batch''
 +
** Can be scheduled to run automatically or manually
 +
** [http://iran.proximity.on.ca:8080/ Test Project]
  
−
===Phase 2: Scripting (Done) ===
+
===Phase 5: Code Review (Done)===
−
* JSONMarshaller
+
* Execute the script on one of Nexj computers.
−
** Develop marshaller to transform NexJ Server Objects into JSON representation
 
  
−
* JSONUnmarshaller
+
===Phase 6: Make Changes as per Code Review (Done)===
−
** Develop an marshaller to transform JSON representation into NexJ Server Objects
+
* Define paths based on Nexj's requirements
 +
* Modify the script to be able to retrieve change-sets of a remote repository
 +
* Create specific error message for each problem during running the script
 +
 
 +
===Phase 7: Code Review (Done)===
 +
* Send code for review.
 +
* [http://mercurial.nexj.com/express.cgi/ NexJ Express Repository in use]
  
 
==Challenges==
 
==Challenges==
−
* 1- First challenge is how to trim history of a repository.
+
* [[Continuous_Integration/challenges | List of the challenges]]
−
: We have a Mercurial repository called A. We want to clone repo A and create a new repo called B.
 
−
: It's possible to use 'init' and create a new repo and just copy the files from repo A and add them.
 
−
: This way the revision number will reset. And we will start repo B from the tip of the repo A.
 
−
: The problem appears if we want to clone just the tip of the repo A and keep the same revision number of the 'tip'.
 
−
: Here are some links that we went through, but no success yet.
 
−
:[http://mercurial.selenic.com/wiki/EditingHistory Trimming History]
 
−
:[http://www.selenic.com/mercurial/hgrc.5.html .hgrc file]
 
−
:[http://mercurial.selenic.com/wiki/TipsAndTricks mq strip changeset]
 
−
:[http://mercurial.selenic.com/wiki/EditingHistory Editing History!!]
 
−
*2- I was receiving an error message while trying to do a new build on Hudson. It didn't let to clone the project on Hudson workspace. The error message is "Access is denied". So I had to delete the project and create a new one. Apparently one of the reasons is that Hudson doesn't let you queue the jobs. So, if you interrupt a job that is scheduled, you'll get the error message.
 
−
*3- To run a bash script to build the project, there are 2 ways:
 
−
:Execute Windows Batch Command:   
 
−
<source lang=bash>
 
−
D:\cygwin\bin\bash /home/HudsonPrac/buildHudson.sh
 
−
</source>
 
−
:Invoke an Ant Script
 
−
<source lang=java>
 
−
<project name="assign1" basedir=".">
 
−
<target name="myTarget" >
 
−
<exec executable="D:\cygwin\bin\bash" newenvironment="false">
 
−
<arg value="/home/HudsonPrac/buildHudson.sh"/>
 
−
</exec>
 
−
</target>
 
−
</project>
 
−
</source>
 
−
*3- Since the bash script is running in Windows platform through Cygwin, there are some conflicts,
 
−
so the PATH was changed to: PATH=/usr/bin:$PATH
 
−
Read more about some conflicts [http://cygwin.com/cygwin-ug-net/using.html#using-pathnames here]
 
−
*4- If there was an error applying a patch to a repository e.g. <code> hg import patch12.patch </code>, there might be a problem with line numbers merging issue. So, there will be an error created with this message: "Hunk #1 Failed". There are some ways to remove the hunks manually or just ignore that patch, if the later versions are fixed.
 
−
: By "hunk" I mean a "snippet of change", i.e. a part of the "diff". TortoiseHg uses this terminology and so does darcs. – Deniz Dogan
 
−
: For Hunk Failed message, first we need to make sure it's not applied before. Second thing to check is that bases are the same. All the revisions before that new changeset even with failed build should be applied before applying the successfully built change-set.
 
−
*5- If there is an uncommited message, the script can not work properly, e.g. Rev is 18+. So, error should be displayed for the uncommited change-sets.
 
−
*6- To run JUnit Tests from command line:
 
−
: add the JUnit installation directory and the junit.jar file and also QTjava to the CLASSPATH. e.g. <code>CLASSPATH=".;D:\cygwin\home\java\junit3.0.1\junit.jar;C:\Program Files\java\jre6\lib\ext\QTjava"</code>
 
−
: to run a sample AllTests class go to the installation directory and run: <code> java junit.textui.TestRunner junit.samples.AllTests </code>
 
−
: to run NexJ AllTests do:
 
−
:: build the model: go to ws/core/build/  run <code> ant </code> or <code> ant -f build_JUnitTest.xml</code>
 
−
:: from command line: go to ws/out/core/ run <code> java junit.textui.TestRunner nexj.core.AllTests</code> (getting error IOExcpetion) so we just ran one test file <code> java junit.textui.TestRunner nexj.core.util.MathUtilTest</code> (We did some changes to <code> core/test/nexj/core/util</code> to practice with it) Or we can run the ant target for test: <code> ant -f build_JUnitTest.xml test </code>
 
−
*7- Bugs after implementation at NexJ
 
−
 
 
−
: Working with remote repository; getting tip and log from remote repository is not as simple as moving to local repo and get all the information we need.
 
−
 
 
−
: NexJ Internal repo has several branches, and when cloning and pulling, we just needed to mention it as a default branch, to not to confuse revision numbers and latest changesets.
 
−
: Since the decimal part of the revision number will keep changing from computer to computer, it was preferred to use the global hex chagesetID (GUID).
 
−
: So, the first big change was adding this command to get the list of latest changesets up to the tip:
 
−
::<code>hg log ${IntDir} -r ${PrevRev}: -b default --template '{node}\n'</code>
 
−
: This number will also be added to the commit summary while importing the patch from Internal repo
 
  
 
==Resources==
 
==Resources==
−
===Continuous Integration System===
+
*[[Continuous_Integration/Resources | List of Resources]]
−
*'''Comparing Different CI Systems''' -> [http://confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix CI Feature Matrix]
+
:
−
#'''[[Hudson-nexj |Hudson]]'''
+
:
−
#'''[[CruiseControl-nexj | Cruise Control]]'''
 
−
#'''[[Buildbot-nexj | Buildbot]]'''
 
−
#'''[http://continuum.apache.org/ Apache Continuum]'''
 
−
#'''Maven'''
 
−
#'''[http://www.anthillpro.com/html/products/anthillos/default.html Anthill]'''
 
−
 
 
−
===Version Control Tool===
 
−
* Basic Tutorial @ http://hginit.com/
 
−
* '''comparing popular version control tools''' [http://martinfowler.com/bliki/VersionControlTools.html]
 
−
# '''[[Mercurial-nexj | Mercurial]]'''
 
−
#'''CVS'''
 
−
#'''SVN'''
 
−
#'''Git'''
 
−
 
 
−
===Scripting===
 
−
*1- '''[[Ant-nexj | Ant]]'''
 
−
:A new ant script is created e.g. ''buildHudson.xml'' that triggers the target(assign1.test) of main build file(build.xml) of the project. See below:
 
−
<source lang=java>
 
−
<project name="assign1" basedir="." default="myTarget">
 
−
  <target name="assign1.build.call">
 
−
    <!-- Call the target that does everything -->
 
−
    <ant antfile="build.xml" target="assign1.test"/>
 
−
  </target>
 
−
  <target name="myTarget.check" depends="assign1.build.call">
 
−
    <echo>The assign1.build was called!</echo>
 
−
  </target>
 
−
</project>
 
−
</source>
 
−
*2- '''[[Bash-nexj| Bash]]'''
 

Latest revision as of 19:29, 26 January 2014

Important.png
This page may be obsolete.
It contains historical information.

Continuous Integration Project for NexJ Express Code

Project goal

Currently, NexJ has a Mercurial Repository (Internal) in which, the NexJ Express Server (Core) code is maintained with all its histories. NexJ is planning to share a Core Repository with the Open Source Community by creating a Repository (External) that will be kept in sync with the Internal Repository.

Contributors


Current Status

Project Completed


Project Phases

Phase 1: Design Proposal (Done)

  • Get requirements from NexJ
  • Incorporate requirements from NexJ into Design of algorithm for syncing repositories
  • Provide the proposal for review and approved - final version of proposal , Google Doc

Phase 2: Creating Similar Environment (Done)

  • Internal Repository
    • Creating a Mercurial repository and adding some changes to it to grow the history of it.
  • Controller
    • This will be containing temporary repositories as the result of scripting.
  • External Repository
    • Creating another Mercurial repository using 'hg' commands from the latest 'change-set' of Internal Repository.

Phase 3: Scripting (Done)

  • Ant
    • Doesn't support some of non-regular Mercurial commands that we need to play with change-sets
  • Bash
    • Supports everything we need for the project. script

Phase 4: Running the script on CI system (Done)

  • Buildbot
  • Hudson
    • Simple to setup and configure.
    • Runs different types of scripts: bash, ant, batch
    • Can be scheduled to run automatically or manually
    • Test Project

Phase 5: Code Review (Done)

  • Execute the script on one of Nexj computers.

Phase 6: Make Changes as per Code Review (Done)

  • Define paths based on Nexj's requirements
  • Modify the script to be able to retrieve change-sets of a remote repository
  • Create specific error message for each problem during running the script

Phase 7: Code Review (Done)

Challenges

Resources

Retrieved from "https://wiki.cdot.senecapolytechnic.ca/w/index.php?title=Continuous_Integration&oldid=104069"