Chromium is moving to GPU hardware accelerate rendering all types of web content as much as possibly with their latest efforts.

For some time now, there’s been a lot of work going on to overhaul Chromium’s graphics system. New APIs and markup like WebGL and 3D CSS transforms are a major motivation for this work, but it also lets Chromium begin to take advantage of the GPU to speed up its entire drawing model, including many common 2D operations such as compositing and image scaling. As a lot of that work has been landing in tip-of-tree Chromium lately, we figured it was time for a primer.

The primer they are looking at is not just rendering the content made in WebGL, CSS3 3d transformations and more but the entire final pass of the output.  This leads to some very interesting years ahead in browsers.  With Chromium, IE9, Firefox and Safari all now with aspects of hardware rendering and acceleration via the GPU, anyone not doing GPU acceleration is seemingly behind the curve that seemed to start in 2007ish to a culmination of today’s latest browsers.

After these layers are rendered, there’s still a crucial last step to blend them all onto a single page as quickly as possible. Performing this last step on the CPU would have erased most of the performance gains achieved by accelerating individual layers, so Chromium now composites layers on the GPU when run with the –enable-accelerated-compositing flag.

Web content will get really interesting over the next couple years.  Even basic computers now have a GPU and bottom of 32MB video memory.  Why aren’t we using those GPUs as much as possible for web content and web games. The time of software rendering might be coming to an end now that processors seem to have topped out and the bottom level computer is capable of handling a decent amount of video memory. It will be easier to justify useful graphics acceleration with a better user experience when we can take advantage of all the computer/device has to offer.

• Chromium Blog post about this development
• Differences between Chrome and Chromium (the project behind Chrome)

If the question is if Unity can do interactives as smooth and stylish as Flash I think you may soon find out.  Carlos Ulloa of Papervision 3D fame has kicked it up a notch in Unity 3D with this interactive very reminiscent of the Ford Focus demo that helped bring in Papervision 3D for flash in style. Gotta say though a mini is much better than a Ford Focus.

Flash is still the leader in web interactives and even marketing interactive 3d, Unity largely replaced Director and tools like it and high-end hardware rendered required interactives and games. This interactive by HelloEnjoy has loads of polygons, unity physics system, lighting, environment mapping, showroom cameras, reflection, skid decals, highly detailed mesh and more.  Just take a peek inside the vehicle and at the rims for the detail that is impossible with the 2000 poly limit of Flash 3D software rendered engines.

Web interactives this heavy aren’t doable in a non hardware rendered player like Flash.  Unity is looking to pretty much own this level of quality in a browser.  I don’t think I have seen another interactive looking this good with Unity 3D.

Unity still is lacking many features that Flash has in support of making interactives for the web such as webcam support, mic support, better video support, better gui system, html support (although flash barely) and a larger install base but Unity could easily take the high-end advertising market in addition to owning highly immersive games that need hardware rendering which it is already doing for web gaming.  It is 2010 soon, most computers have a decent video card.  Put them to use!

• Blog Post on the project
• View the project

This is a very cool project called V8-GL.  It is an OpenGL engine with 80% of the API converted to run on the V8 Javascript engine, the same engine that runs Google Chrome.

This is exciting as more productive languages like Javascript get speed boosts from engines like V8 and are capable of manipulating more complex systems like OpenGL.  Google is also pursing this in the browser with O3D with javascript manipulation of hardware rendering.  Also, a Google funded project called Unladen Swallow is converting Python to the LLVM virtual machine, so that it can have increasing speeds to compete with gcc speeds.

Making things easier to produce and control with more simplified and minimal languages like Javascript, Python and Actionscript etc that control more complex systems, that typically you would need to invest more time in such as a platform on C++ is the goal. V8-GL has this goal in mind.

V8-GL from the author states:

V8-GL intends to provide a high-level JavaScript API for creating 2D/3D hardware accelerated desktop graphics.

In other words, you can hack some JavaScript code that opens a desktop window and renders some 3D hardware accelerated graphics. Bindings are made using the V8 JavaScript engine.

• V8-GL blog post
• V8-GL source on Github
• Also check out the SpiderMonkey jslibs project with other javascript wrappers for libraries like ode, sdl and more using the engine by Mozilla.

Flash 10 will be ready for mainstream hopefully by the end of this year, or early ’10 when the penetration numbers will be up in or around the 90% range via zeh fernando based on previous trajectories.  

With that, Flash 10 has many great new things such as the Vector structure that allows a collection of a certain type, which results in a faster collection because of the known type.  So anywhere where Arrays are used, that is a possible candidate for a performance increase within some code because you are asking the virtual machine to do less work on each loop (not having to dynamically find out the type).

ByteArray (Thibault Imbert) has demonstrated that for the JPEG encoding in corelib it is up to 2.5 times faster using Vectors than Arrays.  Your mileage may vary heavily but it is almost a guaranteed speed boost due to less work.  This obviously has great possibilities for speeding up code that uses lots of arrays.  

Due to the performance boost the Vector does have some constraints in the typical give and take of coder flexibility with compiler and virtual machine overhead.  Vectors are more explicit and strongly typed which is why they are fast, but this is also limiting.

In addition to the data type restriction, the Vector class has other restrictions that distinguish it from the Array class:

• A Vector is a dense array. Unlike an Array, which may have values in indices 0 and 7 even if there are no values in positions 1 through 6, a Vector must have a value (or null) in each index.
• A Vector can optionally be fixed-length, meaning the number of elements it contains can’t change.
• Access to a Vector’s elements is bounds-checked. You can never read a value from an index greater than the final element (length - 1). You can never set a value with an index more than one beyond the current final index (in other words, you can only set a value at an existing index or at index [length]).

 [ Vector docs  ]

ByteArray not only used Vectors heavily but did other optimizations that are always good to do, even though optimization is evil when you are working with precious client side resources ensuring an optimized base starting point can be a good thing.

So what did I do ?

• I used bitwise operators as much as possible.
• I replaced all Arrays with fixed length Vectors.
• I used pre-incrementation rather than post-incrementation (thanks Joa for this one ).
• I casted to int all my Vector indices access.
• Other minor stuff you always do to optimize your code 

Other sources as well for even more optimization or shall I say efficient AS3:

• Polygonal Labs Using Object Pools
• Joa Ebert Tweening and Object Pools
• Joa Ebert Optmization Wiki
• Sean Moore Efficient AS3 roundup (great list of resources for more at the bottom)
• Lost in Actionscript Efficient AS3 Tips
• Big SpaceShip Mouse Leave Tip (flash can hog resources so when it isn’t in focus then let is stop tasking the processor)
• Grant Skinner Idle CPU in AIR FPS Tip (set fps to a lower amount when it is out of focus)

@bartek from everydayflash.com is an amazing 3d flash designer and developer.  The latest from everydayflash is a sample using MouseConstraint in JiglibFlash the 3d physics engine for all major flash 3d engines.

It is easy to see how the latest version of JiglibFlash with MouseConstraint will be heavily influencing flash games and applications very soon. This is a very smooth and quick demo that feels very responsive on the controls.  There are so many possible uses for JiglibFlash now that the MouseConstraint is available.  It will evolve further but this version seems ready to start integrating into many flash game and interactive ideas and projects. Even though it is still alpha it has been heavily cleaned up and a plugin system added by bartek for pluggable 3d render engines.  That is a huge step for 3d pipelines in flash.

Great work JiglibFlash team!

A quick roadmap was posted by Unity3d.com blog on the immediate future of Unity iPhone.  Currently I am developing two games for the iPhone OS 3.0 and these are welcome updates.  We are really looking forward to items not in the hard version just yet but we are looking forward to terrain support and downloadable content support in iphone sdk 3.0.

Unity iPhone 1.0.2. Based on custom builds we’ve been sending to devs in need, this release will address engine memory leaks and fix other outstanding issues:

• Physics and audio related memory leaks
• Asset leaks while reloading scenes
• .NET sockets and threads
• Compressed audio related issues
• Stripping away too much of GUI components
• Occasional crashes in AOT compiler
• Support for both portrait and landscape splash screens 

Next will be Unity iPhone 1.1. Since the release of 1.0.1 we’ve been working on a number of performance and memory optimizations. Most of the work on 1.1 is finished already and we’re doing an internal bug fixing round before it goes to beta testers too. Along with optimizations this release will include number of important features such as:

• Binding custom ObjectiveC/C++ functions to C#/Javascript
• Native on-screen keyboard support and interoperability with Unity GUI
• Movie playback support
• Performance optimizations:
  • significant C#/Javascript performance improvements
  • general rendering loop optimizations resulting in less OpenGLES state changes and less CPU work per object
  • number of internal routines were rewritten using VFP coprocessor assembly
  • way much faster mesh skinning utilizing VFP
  • batching small objects, given that they share same material, into single draw call

• General distribution size optimizations which allows applications below 10Mb
• Number of significant memory footprint optimizations

 
We don’t have strict versioning past 1.1 yet. Some of the following features will end up in the next big release and some might find a way to sneak into 1.1:

• Compressed audio streaming directly from disk
• Support for 3.0 SDK
• 3.0 downloadable content
• 3.0 bluetooth networking
• GPS/Location support
• Vibration support
• Post-processing and render-targets support
• Terrain support
• Per-pixel DOT3 lighting support for skinned meshes
• Reduce load times
• Reduce distribution size even further
• Improve GarbageCollector collection patterns to reduce spikes

Torque3D seems to have full featured browser surfaces that you can use in the 3d engine seamlessly, see the video below (at :38). This is amazing stuff.

Making games that integrate content from the web is especially required these days.  It is a difficult thing to do within the 3d render because of all the plugins, styling etc that needs to be rendered on a 3d surface.  Well Torque3D has a killer feature in that it supports entirely full features browser render on a 3d surface.  So now you can integrate html content, flash video etc in your game easily.

You can play content in flash player content easily and have stripped down html but it is limited, you can play videos and have content in Unity3d but it is limited, even larger engines like Unreal 3 have difficulty handling flash and html content.  If this is a good implementation Torque3D has a killer feature on their hands! Flash is commonly used as user interface elements and content within games but it can be challenging.  This is pretty exciting if it works as advertised. Think of how cool all the little consoles, mini-games and controls in 3d games could be in flash easily.

See at :38 in the video…

GDC 2009 – Torque 3D Web Demo from GarageGames on Vimeo.

Away3D was updated to Flash 10 earlier this month.  Flooded with final semester, massive workload and all the conferences #swsx, #gdc, #mix09 etc I missed the announcement.  

It is looking pretty sweet with this mustang demo showing off the update.  Pixel bender has provided a performance update for effects that shows nicely here.

The above demo shows some of what is possible with the update: normalmapping with ambient, diffuse and correctly normalised specular shading, without the need for layers. As if that’s not enough, a further Pixel Bender shader is applied to the view to create a HDR (High Dynamic Range) effect on the highlights, something usually seen in much more 3d-rich console games. Special thanx go to Eddie Carbinfor donating an excellent normalmapped mustang model, and David Lenaerts for writing the HDR filter.

Airtight Interactive did a comparison a while back where FP10 rendering was faster against the olde version of Away3D, this update may change that will have to take a look.  All away3d needs is a designer

Here is a video with more information on Silverlight 3 Beta features that are matches of the latest Flash killer features in pixel shaders, 3d planes (ability to create pseudo-3d engines like papervision3d), local saving, pixel operations/bitmap handling, local messaging (silverlight to silverlight – like localconnection), out of browser desktop running ability of SL3, SEO and search indexing capabilities / deep linking navigation and more.

Video of the features of SL3 Beta, Demos, at a Slow Pace from #mix09

A few points after the video and taking a tour of the features.

The pixel shaders are written in HLSL (shader 2), however they are compiled to byte code and do not currently use the gpu for rendering. While the pixel shaders are very cool and the language to write them is standard pretty much for shaders in HLSL shader model 2 DirectX-based (the other is GLSL OpenGL based) they have not allowed this byte code to run on the GPU… yet. Here Flash and Pixel Bender actually are ahead there.

Although there are 3d planes which is very exciting, no good pseudo 3d engine exists yet matching the 3 in flash (papervision3d, away3d, sandy). When SL3 comes out I am sure we will see a few emerge or build them ourselves because this iteration of SL3 looks pretty fun.

Pixel based operations will be a huge advancement much like it was in earlier flash versions as it adds some demo scene type abilities and experiments with pixels that are fun.  This also lends to doing cool things like shaders, effects, AR, face recognition, motion detection etc.

Effects like Blur and Drop shadow are good and the ability to add custom ones, great. However currently they are pretty performance intensive.  They are also in Flash but there needs to be some refinement in SL3 effects before launch.

Desktop runnable apps in out of browser will be nice and this is a direct compete with Adobe AIR which was a surprise.

Local Communication supports desktop to browser communication.

Isolated storage (similar to shared objects) supports 1MB in browser, 25MB out of browser defaults.

This version of silverlight is really a 1.0 version as typical with most software.  Version 2 or version 3 is usually what the initial design goals pan out.  Much like the latest unity3d version (2.5) that has windows support and the iPhone SDK 3.0 that both came out this week, even actionscript 3 compared to earlier versions, these toolkits are finally iterated enough that they are really solid platforms for building cool stuff on and become platforms.  The next version of all these could be very, very dangerous.

Huge missing features:

Although there are some great features in SL3 beta, it is still not done and it is still missing some key components that Flash has which make it very attractive in the interactive space.

• Camera and Microphone support - Macromedia hired one of the smartest dudes around in Jeremy Allaire back in flash 6 days to help add support for Flash Communication Server (Flash Media Server now) Camer and Microphone support.  One of the best R&D periods at Macromedia. SL needs this soon.
• Printing support – what was long a problem in Flash is so in SL, there is no good printing support
• No GPU usage for Pixel Effects/Shaders – (neither flash nor silverlight support hardware accelerated shaders in PixelEffects/Pixelbender – Pixel Effects/Shaders need GPU support (see Kevin Goldsmith’s article on GPU mixed with CPU and how this may or may not be good. However processors are speeding up and multi-core helps software rendering, the quality of GPU is well beyond what software rendering can deliver for a few years to come at least while architecture advances, probably more like 5-10 years.
• No UDP plans yet - Adobe has RTMFP, SL sockets has no public plans for adding UDP that I have seen
• No Alpha Channel in Video - You can do this with a shader though but not supported by default.

Silverlight 3 Video

Flash has the upperhand in video and probably will still even though SL3 has H.264.  Flash added this at the same time and though they still have FLV which revolutionized web video they are now much broader in support in video than SL3.  Silverlight has H.264 and VC1 support (their own FLV like codec).  Still pretty cool a couple years ago there was no HD on the web now everyone has it in H.264 video support.

Currently nothing innovative, mainly catchup still, but here are some options

Silverlight 3 beta and the video below the features and highlights will look very similar to flash and flash community advancements over the last couple years.  There is no innovation just yet.

But where that could happen is in socket support with UDP. Flash has moved on this in RTMFP and the beginning of larger scale networking support with UDP with samples like stratus.  This is a huge differentiating feature for what I think will be game changer on the web (it already is on desktop mmos) in real-time or closer to real-time support for larger sets of users in online games like MMOs or virtual communities, even tools to make request based real-time sites like micro-blogging faster and able to handle more users (right now it is very linear if users get many followers, UDP will allow a better distributed framework for messaging).

Local Storage

Silverlight and Unity3D all need this, Flash could use better support for this.  Local saving of a files for cache beyond the internet cache and greater than the 1MB/25MB limits of SL3 IsolatedStorage.  This is an issue when you are making large scale games in that you need to save lots of assets to a client but to make it economical you want ot save more than the default internet cache amount. Flash Shared Object (Local) allow you to do this somewhat but it would be great to have a way to just download files for cache (upon user agreement) to store assets in bulk of allowable types (images, video, models, bundles) to the file system.

Hardware rendering for 3d support and UDP support will put Flash and SL3 on par with the killer Unity3D kit for making online web games and other activex/plugins like instantaction that allow you to do these things already.

The one thing SL has over Flash

Flash and Flex are great. But there is this massive division in the community and marketing of Flash. Silverlight is entirely unified and this has much to do with starting clean at a time that interactive development is heading more into a technology and developers control. Flash and Flex need to bring it together. AS3 has been out long enough that the people with skills have hopped on and taken it to a new level, mainly from programmers. If Adobe created a version of Flash that was a new IDE and consolidated Flex and Flash into just Flash, made the IDE as powerful as FDT or FlashDevelop3 there could be hope to bring the platform together. I understand they had to work it in slowly because it was a designers platform really (even though coders still pushed the limits in games and apps built on it) so they had to tip toe carefully on this to not alienate people. But now I think the division is a serious problem with the platform and must be addressed, noone expected Silverlight to be this quick on at least SL3 features. And even though the initial approach might have been bad as SL1 was a huge letdown, Microsoft does not give up and you can see in the XBOX360 and DirectX that they are very pursuant. DirectX really didn’t become huge until version 7 so these guys won’t relent.

I am not a huge fan of using the proprietary tools. Even in Flash I use as much open source as I can even though the player is locked, but Moonlight is something that trails Silverlight development and is a very unique thing in both open source and cross platform/multiplatform development. It is a clear relationship and aims to make Silverlight run on multiplatform mono including Linux. This could win out in the end who knows.

Futures

Great iterations of software happened this week in the latest unity3d version (2.5) that has windows support and the iPhone SDK 3.0 and now SL3 is quite a surprise in feature set.

I have been really busy this week just delving into all them and hope to start making more cool and useful projects in them.  The best part is right now is great to be an interactive or game developer as all major software companies and markets are focused on retaining good developers.  I don’t’ recall a time other than the beginning the the web virtual land rush that has so many options and markets that skilled developers and designers can choose from.  Good times.

Libspark from Japan is a treasure trove of great flash advancements, they seem to realize the great things that can come from porting in existing solid libraries from C/C++ etc into flash and have been scoring lately including augmented reality in flash porting the ARToolkit to FLARToolkit.  Recently a port of openCV for as3 called Marilena was found and it is for object detection and decent facial recognition (it is a computer vision library from intel) considering the processing power needed to do this.

Face Detection: Here is the sample included with Marilena showing facial detection on an image.

Lots of recent action has blown up on this front from Mr doob, quasimondo (optimizing the Marilena classes for better performance) and Boffwswana. Also there is a kit called deface by sshipman that is the first foray into this a year ago doing similar things but it was just a bit before it’s time and a bit slow in previous versions of flash, it performs decent now in this sample. Flash 10 performance of the AVM2 and future directions with Alchemy will lead to more interesting stuff just like this.

Mr. doob head tracking sample, be sure to check lots of other examples there

Boffswana example of head tracking Johnny Lee Wii style with only a webcam and flash, no wiimote needed since it uses facial detection to check where you are and how close you are in the screen and then moves accordingly.

This is stemming from the recent explosion of the FLARToolkit and augmented reality in flash as well as the gimmicks used by Nintendo with the wii and Johnny Lee’s great head tracking advancements. Porting great libraries to flash seems to be the phase we are entering now judging by the recent excitement around Adobe Alchemy and the LLVM along with the lead from the libspark.org contributors. We have also seen this heavily last year in ports of Box2D for 2d physics and other toolkits using established working code and porting that to flash now that is is mostly capable of handling the performance.

OpenCV (Open Computer Vision Library by Intel) is quite a powerful platform that allows you to do all this and now it is available in flash. There are other great libraries for nearly all platforms now. I have done some previous work with Aforge which is also a port of OpenCV mainly for motion detection. This was always around but not until the recent performance updates and the innovation that has come with Alchemy and the thinking that goes along with that (porting in libraries to flash from C/C++ etc), has allowed this to flourish in flash and thus the web.

The amazing new things we can do with flash by porting in existing libraries is only going to get more intense as alchemy and flash 10 are even more mainstream.  It is almost as if Flash will eventually just become a web renderer and simplified front end to many great toolkits that exist in more native environments like C/C++ but with the speed and distribution access of the web with Flash.  Exciting times ahead.