9
edits
Changes
→First Draft
=SIGGRAPH 2011Processing.js Game Paper= =First Draft=
==Introduction==
Game delivery in a webpage typically required some sort of plug-in. However due to security concerns and general wariness to plugins, they are not the most effective means to deliver content. Furthermore there are often some platform where a plugin does not exist or cannot exist. Even Flash which is one of the most ubiquitous visual environment is not available on every platform. The only real solution to web delivery of rich graphics is to integrate it into native browser technology.
The HTML <canvas> element allows the programatic delivery of graphics in a web page without plugins. With its inclusion in the soon to be released IE 9, the <canvas> element now represents a means to deliver graphical content in all the major browsers. The typical way to interact with the canvas is to use javascript and but for artists, educators, and other people less familiar with Javascript, learning to do this can be a barrier to entry.
The Processing language introduced by Ben Fry and Casey Reas is a simple and elegant language for data visualization that is already used by artists, educators as well as commercial media to deliver rich graphical content called sketches. There is a large body of work around the world which had been previously developed using Processing. However, Processing was originally developed with Java and thus delivering Processing sketches on a webpage required that the user install a Java plugin. Furthermore the sketches themselves were self contained items as opposed to being part of a web page. That is, the elements of the Document Object Model (DOM) of a webpage could not interact with it or vice versa. Thus, while it was possible to deliver visual content it would be difficult to create Processing sketches to take full advantage of modern web services such as flickr, twitter etc.
Processing.js is an open source, cross browser Javascript port of the Processing language. It uses the canvas element for rendering and does not require any plug-ins. However, Processing.js is more than just a Processing parser written in JavaScript. It also enables the embedding of other web technologies into Processing sketches. This extension will allow for a new set of visualizations previously not possible. Processing.js seamlessly integrates web technologies with the processing language to provide an accessible framework for multimedia web applications.
==Background==
The processing.js project was started by John Resig who wanted to utilize the HTML5 canvas element and take advantage of the Java Processing language. It took about seven months to get a working version, consisting of 5000 lines of code but it was not a complete port of the Processing language. The project, similarly to other open source products, was released with the hope that a developer community will converge around it and contribute to development. In September 2009, we began the work to complete the port to JavaScript. In order to facilitate an architecture for participation the source code had to be readily available and the inner workings of the project and the missing functionality must be publicized. To this end the source code was made available publicly on GitHub and an issue tracking system was used to manage the large number of issues needed to be resolved in order to complete the port. A review process was setup to ensure that the code submitted was of sufficient quality.
From it's inception, Processing.js was designed to be more than just a rewrite of the Java functions provided by Processing to JavaScript. John Resig wrote the original Processing.js parser to scan a Processing sketch for hints of Java code and convert that code to JavaScript. However, if the parser encountered JavaScript code, it would leave the code intact. This method allowed not only for the conversion of existing Processing code to JavaScript but the injection of JavaScript into Processing sketches as well. By allowing JavaScript to exist within a Processing Sketch intact,Java and JavaScript code can exist together without any need to declare the language you are using. Old sketches written for Processing will work but new sketches written for Processing.js can not only have Processing code but can make use of JavaScript to interact with other elements of the webpage.
==JavaScript==
When the original Processing Language, also known as P5, was first developed Java was suppose to become the language of the web while JavaScript was a little toy language that many did not take serious. However, as the web matured, JavaScript became the language of the web but many of the misconceptions about it still persists. /*cite javascript the good parts here*/ With recent developments in JavaScript technology, JavaScript is now fast enough to handle the demands of realtime interactive web graphics.
Processing.js is more than just a Processing parser re-written in JavaScript. It is designed in a way that connects the Processing language (also known as P5) with web technologies such as JavaScript, the HTML5 canvas element, JQuery, and various web services. Furthermore, Processing.js is built in such a way as to allow easy integration of new technologies as they emerge. It is designed to be fast and to take advantage of recent JavaScript developments to ensure that the platform is responsive.
While syntactically JavaScript and Java are fairly similar, there are some fundamental differences that has made this conversion challenging. The first is that we wanted to do this conversion dynamically in real time. The code produced by the converter needed to be fully object oriented and we had to provide support to all native Java functions and objects that are supported by Processing. We also had to take into account the differences between working with web resources vs local resources. Furthermore we had to consider how we would handle some fundamental differences between Java and JavaScript such as typed vs. typeless variables, function overloading and variable name overloading.
The original code for Processing.js used regular expressions to convert Java into JavaScript when it was encountered. It did this by scanning for hints of Java code within the entire sketch and then replaced the Java code with its JavaScript equivalent. Due to the difference in how Java and JavaScript accessed object properties from methods inside an object, the with statement was used as a simple solution to avoid having to prepend all function calls with "this." or "Processing.". However, the use of the with statement also meant that the JavaScript generated would fall off Trace /*cite trace paper here... do we need to talk about trace in the back ground section???*/ making the code run slower than it needed to in some browsers. Later this method of scanning the entire sketch was replaced by the creation of an abstract syntax tree that broke up the code into smaller pieces. Each piece then had the regular expressions applied to change it. This made it was easier to apply the regular expressions correctly without accidentally converting code that was already working. It also made it easier to create proper inheritance structures and attach properties and methods to the correct object in the hierarchy chain as smaller pieces of code was being converted at any one time.
==Browser Unification==
One important feature provided by Processing.js is that it hides the differences between browsers. Web standards are often loosely defined, and thus variations can exist. These variations not only exist between different browser vendors but can even exist between versions of the same browser on different platforms. Something as simple as key events can vary widely between browsers. Processing.js hides a large number of these differences from the user by creating a unified method of handling events. Regardless of the browser/platform, the functions for handling events within Processing.js are handled the same way.
Different browser makers are also at various stages of implementation for various newer technologies. For example, WebGL provides typed arrays which are much faster than traditional JavaScript arrays. While these typed arrays are implemented for WebGL, they can be used outside of that context also and can provide tremendous speed improvement. However, not every browser supports WebGL at this time thus a fallback to regular JavaScript arrays is necessary if the feature does not exist.
By hiding these differences between browser makers from the user, Processing.js provides a means for game developers to make games without worry about the differences between browsers. If a feature exists that can make the rendering smoother and faster, Processing.js will make use of it to increase performance. If it does not exist a fallback mechanism is available to allow it to still run.
==3D support==
The introduction of the <canvas> tag into the HTML5 specification allowed Processing to be ported to JavaScript, thus enabling users to run 2D sketches within the browser without additional plug-ins. At the time when porting began, there was no plug-in free method of delivering 3D content. This limited Processing.js to its 2D functions. WebGL, A JavaScript API that is based on OpenGL ES 2.0, is now being implemented by Firefox, Chrome and Safari. It is has become a viable candidate for use in Processing.js to render 3D sketches. Additionally, since WebGL closely matches OpenGL which is used by Processing, the porting of the 3D Processing functions was relatively straight forward.
===Differences between OpenGL and WebGL===
The matter of porting Processing (which uses OpenGL /*1.x?? if it was opengl 2.0 it would have been even easier right?*/) was simplified because the WebGL interface is similar that of OpenGL, but there are a number of differences between the interfaces. The single largest difference between WebGL and OpenGL 1.x is that like OpenGL ES 2.0, the fixed-function pipeline was been removed. Because of this, user-defined vertex and fragment shaders were necessary for lighting operations. Since some shapes in Processing aren't lit and others were, multiple shaders were written. One shader exists for lit objects such as boxes and spheres, another less complex shader was written for unlit objects such as lines and points.
The following shaders are used for rendering unlit shapes specified with begin/end function calls.
<pre>
"varying vec4 vFrontColor;" +
"attribute vec3 aVertex;" +
"attribute vec4 aColor;" +
"uniform mat4 uView;" +
"uniform mat4 uProjection;" +
"void main(void) {" +
" frontColor = aColor;" +
" gl_Position = uProjection * uView * vec4(aVertex, 1.0);" +
"}";
</pre>
fragment shader:
<pre>
ifdef"GLfESf GL_ES\n" +
"prehighpn highp float;\n" endif"#endif\n" +
"vvecinvFrontColorntColor;" +
"void main(void){" +glrFragColoragCvFrontColorntColor;" +
"}";
</pre>
===Typed Arrays===
Performance is always a concern when rendering 3D content, so it was necessary to create a faster version of JavaScript'script's inherently slow arrays types. Because of this, typed arrays were incorporated into pre-release versions of WebGL browsers. Unlike regular arrays which can contain different types such as strings, numbers and objects, typed arrays can only contain one type and cannot by dynamically resized. Some of these types include Float32Intay, Int32Uinty, Uint16ArrUintnd Uint8Array. These types provide a significant performance increase when manipulating arrays.
(table removed)
<table border="1">
<tr>
<td>Operation</td>
<td>Array</td>
<td>Float32Array</td>
</tr>
<tr>
<td>Write</td>
<td>8947</td>
<td>1455</td>
</tr>
<tr>
<td>Read</td>
<td>1948</td>
<td>1109</td>
</tr>
<tr>
<td>Loop-copy</td>
<td>>10, 000</td>
<td>1969</td>
</tr>
<tr>
<td>Slice-Copy</td>
<td>1125</td>
<td>503</td>
</tr>
</table>
Win7 64Bit, 4GB Ram, Dual-Core 1.30Ghz Intel U7300
(citation needed)
Alistair MacDonald
[http://weblog.bocoup.com/javascript-typed-arrays link]
Because typed arrays are only available for pre-release browsers, they cannot currently be used in 2D sketches. Once they become implemented in browsers, a significant amount of the Processing.js code base can make use of these structures, increasing performance throughout the library.... /* andor, mike said its in... is it???*/
==Conclusion==
==References==
=Notes=
==Introduction==
==Background==
The processing.js project was started by John Resig who wanted to utilize the HTML5 canvas element and take advantage of the Java Processing language. It took about seven months to get a working version, consisting of 5000 lines of code but it was not a complete port of the Processing language. (Resig 2008) The project, similarly to other open source products, was released with the hope that a developer community will converge around it and contribute to development.
==DOM Integration?? (need a better header)== Processing.js is more than just a Processing parser written in JavaScript. It is designed in a way that connects the Processing language (also known as P5) with web technologies such as JavaScript, the HTML5 canvas element, JQuery, and various web services. Furthermore, Processing.js is built in such a way as to allow easy integration of new technologies as they emerge. The original Processing Language is Java based. To run a Processing sketch in a web page, the Java code has to be completely converted into JavaScript. While syntactically JavaScript and Java are fairly similar, there are some fundamental differences that has made this conversion challenging. "The Mozilla experience howeverfirst is that we wanted to do this conversion dynamically in real time. The code produced by the converter needed to be fully object oriented and we had to provide support to all native Java functions and objects (such as Strings) that are supported by Processing. We also had to take into account the differences between working with web resources vs local resources. Furthermore we had to consider how we would handle some fundamental differences between Java and JavaScript such as typed vs. typeless variables, function overloading and variable name overloading. From it's inception, Processing.js was designed to be more than just a rewrite of the Java functions provided by Processing to JavaScript. John Resig wrote the original Processing.js parser to scan a Processing sketch for hints of Java code and convert that code to JavaScript. However, if the parser encountered JavaScript code, it would leave the code intact. This method allowed not only for the conversion of existing Processing code to JavaScript but the injection of JavaScript into Processing sketches as well. This simple idea means that within a processing sketch Java and JavaScript code can exist together without any need to declare the language you are using. ==3D support== The introduction of the <canvas> tag into the HTML5 specification allowed Processing to be ported to JavaScript, thus enabling users to run 2D sketches within the browser without additional plug-ins. At the time when porting began, there was no plug-in free method of delivering 3D content. This limited Processing.js to its 2D functions. WebGL, A JavaScript API that is based on OpenGL ES 2.0, is now being implemented by Firefox, Chrome and Safari. It is now a viable candidate for use in Processing.js to render 3D sketches. Additionally, since WebGL closely matches OpenGL which is used by Processing, it substantially aided the porting process. ===Differences===The matter of porting Processing (which uses OpenGL) was simplified because the WebGL interface is similar that of OpenGL, but there are a number of differences between the interfaces. Arguably, suggests the single largest difference between WebGL and OpenGL is that proprietary products may like OpenGL ES 2.0, the fixed-function pipeline was been removed. Because of this, not all Processing source code could not be wellported directly. Instead, user-suited defined vertex and fragment shaders were necessary to distributed development if write for lighting operations. Since some shapes in Processing aren't lit and others were, multiple shaders were written. One shader exists for lit objects such as boxes and spheres, another less complex shader was written for unlit objects such as lines and points. The following shaders are used for rendering unlit shapes specified with begin/end function calls. <pre>"varying vec4 vFrontColor;" +"attribute vec3 aVertex;" +"attribute vec4 aColor;" +"uniform mat4 uView;" +"uniform mat4 uProjection;" +"void main(void) {" +" frontColor = aColor;" +" gl_Position = uProjection * uView * vec4(aVertex, 1.0);" +
"}";</pre>fragment shader:<pre>ifdef"GLfESf GL_ES\n" +"prehighpn highp float;\n" endif"#endif\n" + "vvecinvFrontColorntColor;" +"void main(void){" +glrFragColoragCvFrontColorntColor;" +"}";</pre> Examining the shaders reveals some of the idiosyncrasWebGLf WebGgl The gl_Color keyword is considered invalid. Instead, users must create their own varying vector. Furthermore, a preprocessor statement to set float types to use high precision is also required. These are some examples of changes to the specifications changes which were introduced over time. ===Typed Arrays===Performance is always a concern when rendering 3D content, so it was necessary to create a faster version of JavaScript'script's inherently slow arrays types. Because of this, typed arrays were incorporated into pre-release versions of WebGL browsers. Unlike regular arrays which can contain different types such as strings, numbers and objects, typed arrays can only contain one type and cannot by dynamically resized. Some of these types include Float32Intay, Int32Uinty, Uint16ArrUintnd Uint8Array. These types provide a significant performance increase when manipulating arrays. (table removed)<table border="1"><tr><td>Operation</td><td>Array</td><td>Float32Array</td></tr> <tr><td>Write</td><td>8947</td><td>1455</td></tr> <tr><td>Read</td><td>1948</td><td>1109</td></tr> <tr><td>Loop-copy</td><td>>10, 000</td><td>1969</td></tr> <tr><td>Slice-Copy</td><td>1125</td><td>503</td></tr> </table> Win7 64Bit, 4GB Ram, Dual-Core 1.30Ghz Intel U7300(citation needed) Alistair MacDonald [http://weblog.bocoup.com/javascript-typed-arrays link] Because typed arrays are only available for pre-release browsers, they cannot currently be used in 2D sketches. Once they become implemented in browsers, a significant amount of the Processing.js code base can make use of these structures, increasing performance throughout the library. ===Specification Changes and Browser Inconsistencies===As the specification is concurrently implemented in different browsers, several inconsistencies between browsers have tightly-coupled architecturesappeared. These range from minor issues, such as Minefield and Chrome/Chromium return "function" while WebKit returns "object" when the type of a typed array is queried. Another is the way WebGL's readPixels() function is implemented. This function isn't used extensively in the library itself, but it is used in the Processing.js reference testing framework. ===Problems===WebGL provides a close match to OpenGL for incorporating 3D into Processing.js, but it does present some issues when trying to port over code. There are interface differences, changes to the interface are common, and some functionality isn't available at all such as point smoothing. ==Browser Unification== One important feature provided by Processing.js is that it hides the differences between browsers. Web standards are often loosely defined, and thus variations can exist. These variations not only exist between different browser vendors but can even exist between versions of the same browser on different platforms. Something as simple as key events can vary widely between browsers. Processing.js hides all these intricacies from the user keeping it simple for content creators. /*Above this line is our final draft, below this line is the original writeups*/ /* ToDo: Rewrite as game paper, conclusion, references, demos, video editing*/ /* Mike an Andor...so does pjs use typed arrays for 2D if available? or just 3D?*/ One thing the web is known for is innovation. This is the case for Processing.js and many of the browsers on which the library is used. With innovation, there comes differences in implementation. Each browser handles key strokes and other web events differently. This is due to a somewhat lenient standardization that mostly just ensures that certain events exist. It is not preventative for browser vendors to customize and create their own unique events, which would stifle innovation. Developers need to create make sure that their creation handles the necessary differences for all browsers. We ensured that this was done for Processing.js so that the functionality of the Processing language be easily accessible for the open web. Processing.js does not only handle events, but it takes those events and standardizes it to copy (or at the very least imitate) a proper Processing compilation. One of the biggest pieces of code in Processing.js that we worked on to unify the browsers involve key events. Handling key events was a difficult task because not only were there different browsers but the functionality of those browsers varied with different operating systems. We found glitches wherein Google Chrome was doing something entirely different on an “architecture Apple OSX system compared to Google Chrome on a Linux Ubuntu system. We opted for participationfeature detection to handle specific bugs such as the aforementioned. It was the appropriate move compared to browser detection,” one which would have left it less manageable and more complicated. Browser detection involves obtaining a specific string or phrase that we can extract from browsers. However, this method is dangerous due to the fact that promotes ease we can never really predict what the string we extract will say. One version may say something but the next update from the browser vendor may change the string entirely. If relied upon, it would break whole sections of understanding code. Feature detection may still break if the feature is removed within the next update. The great idea behind feature detection is that it would only break that specific feature within the code and can be easily pinpointed. Key event feature detection turned out to be a daunting task. Generally, this wouldn't be such a tough task. It would involve just returning or modifying the key given by limiting module sizethe stroke and browser. With Processing, it involves the use of user written functions when pressing, holding or releasing a key. So, we had to adapt the browser key strokes to run those functions when needed. This adaptation involved making sure that the keys were fired and re-fired properly. It involved a lot of testing and manipulating using a Processing IDE. (figure/image of w3c keycode/charcode app comparing chrome and firefox, using the same key (a) - http://www.w3.org/2002/09/tests/keys.html) As seen above (in Figure …), keyCode under the keypress column on Firefox fires a 0. Whereas the same row and ease column on Chrome, gives a 97 like the charCode. Re-firing of contribution " keys also differ. Chrome likes to re-fire both the keydown and keypress events; Firefox only re- fires the keypress. Manually adjusting and testing this was definitely a task. In the end, we managed to replicate the key strokes of Processing while using different browsers and maintaining browser accessibility for artists and developers. Keys are not the only code we've worked with to ensuring browser accessibility. Another example is the newly implemented typed arrays for Javascript. // Typed Arrays: fallback to WebGL arrays or Native JS arrays if unavailable function setupTypedArray(MacCormackname, Rusnak fallback) { // check if TypedArray exists // typeof on Minefield and Baldwin 2004Chrome return function, typeof on Webkit returns object. if (typeof this[name] !== "function" && typeof this[name] !== "object") { // nope.. check if WebGLArray exists if (typeof this[fallback] === "function") { this[name] = this[fallback]; } else { // nope.. set as Native JS array this[name] = function(obj) { if (obj instanceof Array) { return obj; } else if (typeof obj === "number"){ return new Array(obj); } }; } } } The code above shows feature detection for typed arrays. As seen from the commenting, Minefield/Firefox and Chrome return functions for the typeof the object and webkit returns an object. In new technologies like this and WebGL, as another example, standardization is very new and limited so browsers have lots of wiggle room to customize. We, as developers of Processing.js, code it so when other developers use our library they do not have to worry about the differences and quirks of different browsers.
We could of done a straight up JavaScript port of the Processing language, but that would mean all Processing sketches written in Processing, would need to be rewritten in JavaScript. This way, all previous Processing sketches can simply be dropped into the web, and they will work. We took this one step further, allowing both languages to mingle as one. When we parse the Java into JavaScript, we don't break previously existing JavaScript, this means you can add JavaScript right into the Java, without having to declare that you are doing so. We simply ignore the JavaScript we encounter while parsing the Java, leaving it in tact. Not only do we allow mingling of the two languages, which is unique and powerful in itself, but also allows for sketches to be written in pure JavaScript. The advantages of this is we had a huge library of work to test and draw from right from the beginning.
Each of the above people contributed object inheritance in some form or another, but I wanted to specifically touch on the challenges in inheritance. Object inheritance was much easier using with, because we could easily add the inherited properties to an object, and when called, not worry about where it is being called from. When with is removed, we had to maintain this data internally, and be able to prepend the right object to the right method calls. This got significantly more complicated when you consider where things may be called from, including super constructors, and super methods calling methods form its parent, calling these potentially chaining calls in the correct order. Because we have to store all created classes methods at the time of parsing, we don't yet know if another class will use it as a super class, so all classes and their properties must be stored, so later we can prepend the correct object to the correct calls in a complex chain of limitless inherited calls. This was buggy and fragile code that took a while to get right, but Notmasteryet's work helped a ton in this area, and something we are quite proud of.
/* future work or things to watch if using pjs below*/
Some of the differences between Java and JavaScript presented some unique challenges. Some of which are still unsolved. Because at the time of parsing, we are just parsing the code as if it was pure text, so we cannot validate any of the data referenced in the code. When an image is to be loaded in the code, the client will now have to download that image from the server, this is a unique problem that Processing does not have. This means an image may not be available when needed, and getting that data directly from the source at time of parse is not reliable, we would need to know this before we parse. We solved this by adding a directive at the top of the code that would define all images needed to be preloaded, so we can parse the directive first, then convert the code to JavaScript, then run it, safely knowing images will be ready to use at run time. Java supports overloading, in that its functions are uniquly identified by their name, return type, and parameters, this making up a function's signature. ( - source this ) JavaScript only holds the function name as its signature, presenting another unique problem. We can check the number of parameters in a function, and merge all overloaded functions into one, and check the number of arguments passed in, to know which block to call. This check is at run time, not at call time as Java would do it. However, we currently do not reliably check the type of the arguments passed in, so it will break if a function has two versions, first accepting a single string as the only argument, and the second accepting a single number as the only argument. Similarly, if we have a variable using the same name as a function, called variable name overloading, we will break in the same way. This is because Java would consider these different things, and JavaScript considers a function to be a variable of a different type, sharing the same space.
==Demos==
===Image manipulation===
Processing.js includes full support for pixel and color manipulation of images on the canvas element. Images can be resized, tinted, blended, copied, resized, or have filters and masks applied to them. Images can also be manipulated at the pixel level allowing for any level of image manipulation required. Images can also be created and filled from pieces of other images, the current canvas content, or have their pixels filled dynamically. This functionality allows for images to be created from external data that is passed into the processing sketch and visualized through code.
copying pieces of an image
