Processingjs paper

SIGGRAPH 2010

WebGL section

Andor Salga

[add 3D FFT visualizer]

WebGL Introduction

The introduction of the <canvas> tag into the HTML5 specification allowed Processing to be ported to JavaScript, thus enabling users to run 2D sketches within the browser without additional plug-ins. At the time when porting began, there still was no plug-in free method of delivering 3D content. This limited Processing.js to 2D until WebGL was introduced. Once WebGL was implemented on pre-release versions of Firefox, Safari and Chrome, it became a viable candidate for use in Processing.js to render 3D sketches. Additionally, since WebGL closely matches OpenGL which is used by Processing, it substantially aided the porting process.

WebGL first began as an experimental add-on for Firefox developed at Mozilla. It was later adopted by the Khronos group who manage the OpenGL specifications. It is a JavaScript API which provides a subset of the functionality of OpenGL ES 2.0. The interface is relatively simple, yet it still provides enough functionality to emulate almost all of Processing's 3D functions. WebGL continues go through interface changes and revisions.

Differences

The matter of porting Processing (which uses OpenGL) was simplified because the WebGL interface is similar that of OpenGL, but there are a number of differences between the interfaces. Arguably, the single largest difference between WebGL and OpenGL is that like OpenGL ES 2.0, the fixed-function pipeline was been removed. Because of this, not all Processing source code could not be ported directly. Instead, user-defined vertex and fragment shaders were necessary to write for lighting operations. Since some shapes in Processing aren't lit, a few shaders were written. One shader exists for lit objects such as boxes and spheres, another less complex shader was written for unlit objects such as lines and points.

The following shaders are used for rendering unlit shapes specified with begin/end function calls.

vertex shader:

"varying vec4 vFrontColor;" + "attribute vec3 aVertex;" + "attribute vec4 aColor;" + "uniform mat4 uView;" + "uniform mat4 uProjection;" + "void main(void) {" + " frontColor = aColor;" + " gl_Position = uProjection * uView * vec4(aVertex, 1.0);" +
 "}";

fragment shader:

ifdef"GLfESf GL_ES\n" + "prehighpn highp float;\n" endif"#endif\n" +

"vvecinvFrontColorntColor;" + "void main(void){" +glrFragColoragCvFrontColorntColor;" + "}";

Examinishadersshaders reveals some of the idiosyncrasWebGLf WebGgl The gl_Color keyword is considered invalid. Instead, users must create their own varying vector. Furthermore, a preprocessor statement to set float types to use high precision is also required. These are some examples of changes to the specifications changes which were introduced over time.

Typed Arrays

Performance is always a concern when rendering 3D content, so it was necessary to create a faster versJavaScript'script's inherently slow arrays types. Because of this, typed arrays were incorporated into pre-release versiWebGLf WebGL browsers. Unlike regular arrays which can contain different types such as strings, numbers and objects, typed arrays can only contain one type and cannot by dynamically resized. Some of these types include Float32Intay, Int32Uinty, Uint16ArrUintnd Uint8Array. These types provide a significant performance increase when manipulating arrays.

Operation

Array

Float32Array

Write

8947

1455

Read

1948

1109

Loop-Copy

> 10,000 ms

1969

Slice-Copy

1125

503

Win7 64Bit, 4GB Ram, Dual-Core 1.30Ghz Intel U7300

(citation needed)

Alistair MacDonald

[1]

Because typed arrays are only available for pre-release browsers, they cannot currently be used in 2D sketches. Once they become implemented in browsers, a significant amount of the Processing.js code base can make use of these structures, increasing performance throughout the library.

Specification Changes and Browser Inconsistencies

As the specification is concurrently implemented in different browsers, several inconsistencies between browsers have appeared. These range from minor issues, such as Minefield and Chrome/Chromium return "function" while WebKit returns "object" when the type of a typed array is queried. Another is the way WebGL's readPixels() function is implemented. This function isn't used extensively in the library itself, but it is used in the Processing.js reference testing framework.

Problems

WebGL provides a close match to OpenGL for incorporating 3D into Processing.js, but it does present some issues when trying to port over code. There are interface differences, changes to the interface are common, and some functionality isn't available at all such as point smoothing.

Js and processing integration

Processing is Java based, and in order to make it work in the web, it has to be completely converted into JavaScript. Syntactically JavaScript and Java are actually quite similar, and people have done work like this before (google, java nes emulator to js nes emulator). Our unique challenges were that we had to do this dynamically, be fully object oriented, support all native Java functions that are supported by Processing, and consider all web like differences, like images having to be pre loaded before we can start processing the code, casting typeless variables, function overloading, and variable name overloading.

We could of done a straight up JavaScript port of the Processing language, but that would mean all Processing sketches written in Processing, would need to be rewritten in JavaScript. This way, all previous Processing sketches can simply be dropped into the web, and they will work. We took this one step further, allowing both languages to mingle as one. When we parse the Java into JavaScript, we don't break previously existing JavaScript, this means you can add JavaScript right into the Java, without having to declare that you are doing so. We simply ignore the JavaScript we encounter while parsing the Java, leaving it in tact. Not only do we allow mingling of the two languages, which is unique and powerful in itself, but also allows for sketches to be written in pure JavaScript. The advantages of this is we had a huge library of work to test and draw from right from the beginning.

John Resig, the mastermind behind Jquery, is also the mastermind behind Processing.js. His initial work was to use regex to scan the sketch source code for hints of Java, replace it with JavaScript, and leave all JavaScript in tact. He started by taking a previously existing Processing sketch, adding functional support to make that one sketch work, and doing this one sketch at a time, creating missing functions as needed. He took advantage of the pre existing library of sketches, so for each sketch he explicitly supported, he would be that much closer to implicitly supporting other sketches.

“In development I worked in a backwards manner. Instead of building the API up from the ground - I worked from the top, down, implementing enough of the API to get individual demos working.” -http://ejohn.org/blog/processingjs/

Scott Downe's work was mostly related to fixing bugs, and removing the dangerous JavaScript function with. Fixing bugs was a good place to start learning the code, getting his feet wet. The first bug he fixed was to make sure potential code contained in strings were not parsed. This was initially accomplished by masking all strings with a key, and storing their values before the code was parsed, and later replacing the unchanged strings via their keys after parsing. Other, smaller bugs were fixed until it became apparent that the use of the with function meant we would fall off trace, and wouldn't reach our full speed potential. With was being used in two places, first being around all of the sketch, to load in the whole of the Processing library, and to load in method calls from internal function use. We have to do this, because of the differences in how Java and JavaScript call and access their object properties. JavaScript accesses all properties within the object itself separated with a dot from inside or outside the object, where as Java only needs a dot when accessed from outside the object. Using with meant we could contain all Processing functions inside an object, and not have to change how it is called inside the Java. This was the easiest and fastest way to do this, but needed to be changed. Removing with meant prepending the processing object to all calls to the API and internal object properties. So we needed to store a list of the existing properties for both the API and created objects, and when the parser finds a match, prepends itself, either being “Propcessing” or “this” to the property. This worked, but was fragile; we were still using regex's, and doing this to the whole of the source, meaning each new regex we called was a danger to parse code that is similar, but different, potentially breaking code we did not intend to that previously worked. Despite working, this was a hack and a maintenance nightmare. We needed something better.

Notmasteryet rewrote the parser to convert the sketch into an abstract syntax tree, which is an abstract tree representation of blocks of code. By doing this, blocks can be precisely parsed without the worry of breaking or parsing unintended things in an unexpected way. Regex is still used for each part, but is now contained to specifically targeted smaller chunks code, instead of the whole thing. This makes maintaining the code much easier, makes object inheritance easier, and makes JavaScript code included in the sketch more stable. In fact, since the abstract syntax tree's inclusion, we have found new bugs in the parser to be pretty much non existent.

Each of the above people contributed object inheritance in some form or another, but I wanted to specifically touch on the challenges in inheritance. Object inheritance was much easier using with, because we could easily add the inherited properties to an object, and when called, not worry about where it is being called from. When with is removed, we had to maintain this data internally, and be able to prepend the right object to the right method calls. This got significantly more complicated when you consider where things may be called from, including super constructors, and super methods calling methods form its parent, calling these potentially chaining calls in the correct order. Because we have to store all created classes methods at the time of parsing, we don't yet know if another class will use it as a super class, so all classes and their properties must be stored, so later we can prepend the correct object to the correct calls in a complex chain of limitless inherited calls. This was buggy and fragile code that took a while to get right, but Notmasteryet's work helped a ton in this area, and something we are quite proud of.

Some of the differences between Java and JavaScript presented some unique challenges. Some of which are still unsolved. Because at the time of parsing, we are just parsing the code as if it was pure text, so we cannot validate any of the data referenced in the code. When an image is to be loaded in the code, the client will now have to download that image from the server, this is a unique problem that Processing does not have. This means an image may not be available when needed, and getting that data directly from the source at time of parse is not reliable, we would need to know this before we parse. We solved this by adding a directive at the top of the code that would define all images needed to be preloaded, so we can parse the directive first, then convert the code to JavaScript, then run it, safely knowing images will be ready to use at run time. Java supports overloading, in that its functions are uniquly identified by their name, return type, and parameters, this making up a function's signature. ( - source this ) JavaScript only holds the function name as its signature, presenting another unique problem. We can check the number of parameters in a function, and merge all overloaded functions into one, and check the number of arguments passed in, to know which block to call. This check is at run time, not at call time as Java would do it. However, we currently do not reliably check the type of the arguments passed in, so it will break if a function has two versions, first accepting a single string as the only argument, and the second accepting a single number as the only argument. Similarly, if we have a variable using the same name as a function, called variable name overloading, we will break in the same way. This is because Java would consider these different things, and JavaScript considers a function to be a variable of a different type, sharing the same space.

“In order to support this there would have to be considerable overhead - and it's generally not a good practice to begin with.” -http://ejohn.org/blog/processingjs/

Another interesting difference stems from Java being a typed language, and JavaScript being typeless. Java would require casting in most cases, where as with javaScript we can simply throw the cast away for all literal variable types. The problem is if the type is something like a double, or a char, which in JavaScript is simply a string or int. ( source this? ) We solved this for chars with a custom char class, it solved a lot of issues we were having but it is not perfect, by not solving all issues in all cases. Some other types like double and byte will require more overhead and will not be possible without complete type tracking.