Google is pushing Web Sockets

Starting in the Google Chrome developer channel release 4.0.249.0, Web Sockets are available and enabled by default. Web Sockets are “TCP for the Web,” a next-generation bidirectional communication technology for web applications being standardized in part of Web Applications 1.0. We’ve implemented this feature as described in our design docs for WebKit and Chromium.

Sample Code

if ("WebSocket" in window) {
  var ws = new WebSocket("ws://example.com/service");
  ws.onopen = function() {
    // Web Socket is connected. You can send data by send() method.
    ws.send("message to send"); ....
  };
  ws.onmessage = function (evt) { var received_msg = evt.data; ... };
  ws.onclose = function() { // websocket is closed. };
} else {
  // the browser doesn't support WebSocket.
}

Socket Advantage

Flash has long been the answer for sockets for web applications and once sockets were added to Flash it instantly became a better interactive and gaming platform for multi-user applications and multiplayer games. They started with XmlSocket then recently added Socket for raw binary data in as3.  Silverlight and Java also have this feature but having this in script is pretty significant because many applications could really use a browser supported bi-directional communication link.

What is Missing

The biggie missing from Flash, Silverlight, etc and Web Sockets is UDP and preferably RUDP or Reliable UDP which allows UDP datagrams to be sent back and forth either verified delivery or broadcast. Unity does support UDP.  The best socket layers are reliable UDP based because mixing TCP and UDP can lead to queuing and not all messages are critical so having just UDP isn’t enough, having TCP is too much.  Reliable UDP is the way to go but so far no web layers are doing it well except Unity on that one (you still have to make your own RUDP implementation – libraries like Raknet or enet in C/C++ give you this but you can’t use that in Unity client only on the server). (Edit: Flash does have RTMFP which is based on UDP and uses FMS for nat for p2p but it is still not a true low level UDP socket just yet as it supports more features. A low-level UDP socket would also be nice in flash.)

Web Communication Evolving

I am a big Flash fan and have been developing it since 1999 among other platforms, I have recently watched other technologies nearly match the features and some go beyond it.  The interesting thing about Web Sockets is that it does go after a core feature of flash; Canvas and WebGL or O3D also do. Flash still has the webcam, mic, sound mixers/tranform, and for now sockets which put it at an advantage in gaming and interactive. Flash used to  be the sole greatest video player but Silverlight is doing a pretty good job of that as well so that is still an advantage but others are entering including possibly browser support in html5. I still think it is the best video but they would need to keep innovating.

Another interesting point about this is XMLHttpRequest objects.  Originally “AJAX” was created by Microsoft for IE, pushing new features and innovating back when IE was a good browser and ahead in IE4. Mozilla and others adopted this feature (as well as editable text areas for html) because they were great features for web applications to evolve to.  now Google is pushing with Chrome and Web Sockets is the next step that should be in web browsers even if it is only TCP based for now.  This will add great capabilities and will probably be preferred over AJAX/XMLHttpRequest for really interactive and real-time tools/games should it take hold.  Ian Hickson is running the table on the standards with this effort and it is a good one to get behind.

• Blog post about Web Sockets
• W3C Standard proposed by Google (Ian Hickson)
• WHATWG Standard proposed by Google (Ian Hickson)
• IETF RFC ‘web socket protocol’ by hixie (Ian Hickson)
• pywebsocket for Apache to test with Web Sockets

Flash Professional CS5 will enable you to build applications for iPhone and iPod touch using ActionScript 3. These applications can be delivered to iPhone and iPod touch users through the Apple App Store.*

A public beta of Flash Professional CS5 with prerelease support for building applications for iPhone is planned for later this year. Sign up to be notified when the beta starts.

I have been questioning why they have not moved to this model before when others are doing so such as haXe, Unity3D and MonoTouch.  Getting Flash on the web browsers on a mobile is hard because Flash is pretty CPU intensive on embedded devices which is really where computers were in the late 90’s and close to 400-600 MHz processors.  Today these machines wouldn’t be able to run Flash very well and that is the same effect you get on a mobile phone.  But cross-compiling to native, similar to how Unity 3D does it or other solutions like MonoTouch and XNATouch, this is the best solution until mobile/embedded devices have 1GHz processors and more than 500MB of memory. Adobe is using LLVM, much like the Alchemy model, to achieve getting AS3 content onto an iPhone/Touch with AOT or Ahead of Time compilation rather than JIT compilation.

So how do you build an application for the iPhone? It’s simple, really. The forthcoming beta of Adobe Flash Professional CS5 incorporates the ability to create an iPhone application. You have access to nearly all the AIR 2.0 and Flash Player 10.1 APIs. For example, you can use APIs such as RTMP, Remote Shared Objects, and AMF as well as AIR APIs like SQLite and filesystem access. For more information see the developer FAQ on Adobe Labs.

I am glad to see Adobe finally moving on mobile platforms beyond Flashlite.  Flashlite is a poor solution in most cases on embedded devices because they really need native apps to perform, again due to the hardware limitations and it is a whole new platform to learn. Adobe is doing the hard work to make it easy to get developers content on the new embedded devices that are storming the world such as the iPhone and Touch.

• Info on the CS5 Beta
• Sign up for the beta of CS5
• FAQ on Flash iPhone Applications
• Developer article on how the LLVM cross compiling will work
• Flash 10 full version on all phones by Q2 2010 except iPhone

Microsoft has released Silverlight 3 to the web a day earlier than expected. You can go ahead and grab Silverlight 3 RTW build 3.0.40624.0 (4.69MB) from Microsoft.com/Silverlight. Version 3 supports Windows Internet Explorer 6/7/8, Firefox 2/3, and Safari 3/4. In addition, the Silverlight 3 SDK (9.5MB) and the Silverlight 3 Tools (32.2MB) have been posted on the Microsoft Download Center. (arstechnica)

As Silverlight versions go it is quite impressive and pretty much a complete solution now including desktop save support and full set of tools for RIA development (early versions were only Javascript or had limited controls libraries).

Granted there are lots of years of gain that Flash has on Silverlight but the path that Silverlight it following leads right to Flash.  Hopefully this will lead to more innovation on both sides (they might need it with Google Wave pushing html5 <canvas>).

One very nice element of Silverlight since it has been released at verison 1.0 is the HD video support.  It has gotten better with each release.  This release has smooth streaming support that is pretty impressive for web video.

Interestingly they chose the open source Blender Foundation project Bug Buck Bunny to demonstrate the smooth streaming feature. Ryan Rea has a bit of analysis on how well the video plays across a quad core and memory compared to flash hd video.

More on the Silverlight 3 new features here and here.

• Install Silverlight 3
• Silverlight 3 SDK (9.5MB)
• Silverlight 3 Tools (32.2MB)

Roy Schestowitz plays the flip side and calls this a ’silver-lie’ released and has lots to say about the Silverlight 3 release and even using Big Buck Bunny to demo it. He states a true fact that up til now lots of companies have abandoned Silverlight in favor of Flash (mlb, nyt etc).  It is still used at Netflix but that has an XBOX deal to play Netflixon xbox360.

Flash didn’t really get good until version 4.  Typically software is a real version at version 3, that is when most software has the goals and ambitions of 1.0 fully complete and integrated.  We shall see how things play out but I still think Silverlight has a long way to go in winning over developers, myself included, but competition is never a bad thing when you are wanting to see innovation.

A very nice feature of this library, in addition to being coded cleanly and as3 style, is the ability to have the Twitter OAuth page render inside of Flash.

Instead of opening the Twitter authorization page in the browser the library also contains OAuthLoader which is a wrapper around HTMLLoader which enables to directly show the authorization page within an AIR window:

// use this in the requestTokenHandler instead of navigateToURL
var loader:OAuthLoader = new OAuthLoader();
loader.load(request);
loader.percentWidth = 100;
loader.percentHeight = 100;
var w:Window = new Window();
w.width = 800;
w.height = 400;
w.title = req.url;
w.addChild(loader);
w.open();

• Shannon Hicks core oauth as3 library
• High level AS3/Flex library for OAuth
• oauth-as3 Google Code page
• OAuth
• Twitter OAuth FAQ
• Twitter API Documentation
• Getting Started with OAuth

This is exciting as more productive languages like Javascript get speed boosts from engines like V8 and are capable of manipulating more complex systems like OpenGL.  Google is also pursing this in the browser with O3D with javascript manipulation of hardware rendering.  Also, a Google funded project called Unladen Swallow is converting Python to the LLVM virtual machine, so that it can have increasing speeds to compete with gcc speeds.

Making things easier to produce and control with more simplified and minimal languages like Javascript, Python and Actionscript etc that control more complex systems, that typically you would need to invest more time in such as a platform on C++ is the goal. V8-GL has this goal in mind.

V8-GL from the author states:

V8-GL intends to provide a high-level JavaScript API for creating 2D/3D hardware accelerated desktop graphics.

In other words, you can hack some JavaScript code that opens a desktop window and renders some 3D hardware accelerated graphics. Bindings are made using the V8 JavaScript engine.

• V8-GL blog post
• V8-GL source on Github
• Also check out the SpiderMonkey jslibs project with other javascript wrappers for libraries like ode, sdl and more using the engine by Mozilla.

With that, Flash 10 has many great new things such as the Vector structure that allows a collection of a certain type, which results in a faster collection because of the known type.  So anywhere where Arrays are used, that is a possible candidate for a performance increase within some code because you are asking the virtual machine to do less work on each loop (not having to dynamically find out the type).

ByteArray (Thibault Imbert) has demonstrated that for the JPEG encoding in corelib it is up to 2.5 times faster using Vectors than Arrays.  Your mileage may vary heavily but it is almost a guaranteed speed boost due to less work.  This obviously has great possibilities for speeding up code that uses lots of arrays.  

Due to the performance boost the Vector does have some constraints in the typical give and take of coder flexibility with compiler and virtual machine overhead.  Vectors are more explicit and strongly typed which is why they are fast, but this is also limiting.

In addition to the data type restriction, the Vector class has other restrictions that distinguish it from the Array class:

• A Vector is a dense array. Unlike an Array, which may have values in indices 0 and 7 even if there are no values in positions 1 through 6, a Vector must have a value (or null) in each index.
• A Vector can optionally be fixed-length, meaning the number of elements it contains can’t change.
• Access to a Vector’s elements is bounds-checked. You can never read a value from an index greater than the final element (length - 1). You can never set a value with an index more than one beyond the current final index (in other words, you can only set a value at an existing index or at index [length]).

 [ Vector docs  ]

ByteArray not only used Vectors heavily but did other optimizations that are always good to do, even though optimization is evil when you are working with precious client side resources ensuring an optimized base starting point can be a good thing.

So what did I do ?

• I used bitwise operators as much as possible.
• I replaced all Arrays with fixed length Vectors.
• I used pre-incrementation rather than post-incrementation (thanks Joa for this one ).
• I casted to int all my Vector indices access.
• Other minor stuff you always do to optimize your code 

Other sources as well for even more optimization or shall I say efficient AS3:

• Polygonal Labs Using Object Pools
• Joa Ebert Tweening and Object Pools
• Joa Ebert Optmization Wiki
• Sean Moore Efficient AS3 roundup (great list of resources for more at the bottom)
• Lost in Actionscript Efficient AS3 Tips
• Big SpaceShip Mouse Leave Tip (flash can hog resources so when it isn’t in focus then let is stop tasking the processor)
• Grant Skinner Idle CPU in AIR FPS Tip (set fps to a lower amount when it is out of focus)

amfast is a new remoting library  that looks to be as sweet as pyamf (where sweet == fast and useful).  I am checking out amfast now but the speed boost alone might be worth it.  For instance, working with real-time games, when you need static content you need to grab that quickly sometimes via a content service.  The faster that link the better. It also has Twisted integration which is great for networking and SQLAlchemy integration which is in my opinion the best ORM for python (pyamf has twisted, django, pylons, sqlalchemy as well)

amfast is well documented and has some great examples.  If you have the Python addiction, check it.

Description

• AmFast is a Flash remoting framework for Python.
• AmFast can use AMF to communicate between Python and Flash, Flex, and any other system that supports AMF.
• AMF is a binary object serialization protocol used by Actionscript based applications.

Server Features

• Support for NetConnection and RemoteObject RPC.
• Support for Producer/Consumer ‘push’ messaging with HTTP polling, HTTP long-polling, and real-time HTTP streaming channels.
• Support for authentication with NetConnection and RemoteObject.
• Flexible Target mapping system to map message destinations to invokable Target objects.
• Support for ChannelSets with multiple Channels to expose resources in different ways.
• Built in Channels for CherryPy, Twisted Web, and plain WSGI.
• Support for configurable Endpoints. Use AmFast’s built-in AMF encoder/decoder C-extension, or use an external AMF encoder/decoder, such as PyAmf for a pure-Python implementation.

AMF Encoder/Decoder Features

• AMF0/AMF3 encoder/decoder written in C as a Python extension for speed.
• More than 10x faster than the PyAmf encoder/decoder (even when using PyAmf’s optional C-extension).
• Map custom classes with ClassDef objects for complete control over serialization/de-serialization.
• Full support for IExternalizable objects.
• Data persistence with SqlAlchemy including remotely-loadable lazy-loaded attributes.
• Actionscript code generation from ClassDef objects.

The site mentions that OpenGL source can be converted to run in the engine.  You can do this now with Alchemy although it is in very early stages.  It is not clear if it is an automatic conversion or if it simply means it is similar in syntax and method signatures, objects etc.

I definitely will be watching and see how it progresses, there isn’t much other than a single post about the engine so far and no info on the api or sample code.  Looking forward to seeing more, the z-sorting is quite nice.  Doesn’t appear like collisions are there yet but it has a nice look.

Sometimes excellent toolkits come out of the blue like this such as Ffilmation (isometric flash engine) or Alternativa (flash 3d engine flash 10 focused) so you never know.

Now they are also making and releasing an O3D plugin that looks to be another way to do web 3d scenes and games although it is a very early stage. They appear to want to have an open discussion about how best to add hardware rendering to the web.  Their approach uses a javascript api to control the browser plugin and the O3D control is essentially just a renderer.

This won’t change anything now as Unity3D, Flash 3D pseudo engines, even Director 3D still are the top choices for games, apps, and interactives that need effects and possibly hardware rendering. But it is interesting that Google is essentially re-entering this debate after ditching on Lively and they must see some benefit to having a discussion about 3d on the web and 3d standards in general.  I know they have lots of models and tools with SketchUp and Google 3D warehouse so who knows maybe they will take it over by being standards, open and information based.

What is O3D?

O3D is an open-source web API for creating rich, interactive 3D applications in the browser. This API is shared at an early stage as part of a conversation with the broader developer community about establishing an open web standard for 3D graphics.

Get involved

• Download the plug-in (Windows and Mac) and explore the samples to see O3D’s capabilities. Linux users, see these instructions.
• Read the Technical Overview and Developer’s Guide to learn how to get started.
• Join the O3D developer groups to provide feedback.

One thing is for sure, 3d development is still old school proprietary lock in in most cases.  Working with 3d and tools like Maya, 3dsmax and others they have always been very non standard.  From file formats to interfaces to even basic movements, all different.  The general maths of 3d are the same and so should 3d pipelines.  Formats like COLLADA are nice because they are starting to open up 3d pipelines and content creation but COLLADA still has many porting issues.  FBX file format is another that is really useful and common making pipelines in Unity 3D, for instance, very nice. But it is owned and run by Autodesk who owns all the 3d apps (Maya, 3dsmax, SGI) and I am a bit leary of that method.  But in the end 3d pipelines and rendering will be somewhat standardized and maybe the web will be hardware rendered one day.  In most cases it is not needed, but for gaming, immersion, demos and other entertainment it could benefit heavily from a more standardized 3d pipeline and methods.

Video of the features of SL3 Beta, Demos, at a Slow Pace from #mix09

A few points after the video and taking a tour of the features.

The pixel shaders are written in HLSL (shader 2), however they are compiled to byte code and do not currently use the gpu for rendering. While the pixel shaders are very cool and the language to write them is standard pretty much for shaders in HLSL shader model 2 DirectX-based (the other is GLSL OpenGL based) they have not allowed this byte code to run on the GPU… yet. Here Flash and Pixel Bender actually are ahead there.

Although there are 3d planes which is very exciting, no good pseudo 3d engine exists yet matching the 3 in flash (papervision3d, away3d, sandy). When SL3 comes out I am sure we will see a few emerge or build them ourselves because this iteration of SL3 looks pretty fun.

Pixel based operations will be a huge advancement much like it was in earlier flash versions as it adds some demo scene type abilities and experiments with pixels that are fun.  This also lends to doing cool things like shaders, effects, AR, face recognition, motion detection etc.

Effects like Blur and Drop shadow are good and the ability to add custom ones, great. However currently they are pretty performance intensive.  They are also in Flash but there needs to be some refinement in SL3 effects before launch.

Desktop runnable apps in out of browser will be nice and this is a direct compete with Adobe AIR which was a surprise.

Local Communication supports desktop to browser communication.

Isolated storage (similar to shared objects) supports 1MB in browser, 25MB out of browser defaults.

This version of silverlight is really a 1.0 version as typical with most software.  Version 2 or version 3 is usually what the initial design goals pan out.  Much like the latest unity3d version (2.5) that has windows support and the iPhone SDK 3.0 that both came out this week, even actionscript 3 compared to earlier versions, these toolkits are finally iterated enough that they are really solid platforms for building cool stuff on and become platforms.  The next version of all these could be very, very dangerous.

Huge missing features:

Although there are some great features in SL3 beta, it is still not done and it is still missing some key components that Flash has which make it very attractive in the interactive space.

• Camera and Microphone support - Macromedia hired one of the smartest dudes around in Jeremy Allaire back in flash 6 days to help add support for Flash Communication Server (Flash Media Server now) Camer and Microphone support.  One of the best R&D periods at Macromedia. SL needs this soon.
• Printing support – what was long a problem in Flash is so in SL, there is no good printing support
• No GPU usage for Pixel Effects/Shaders – (neither flash nor silverlight support hardware accelerated shaders in PixelEffects/Pixelbender – Pixel Effects/Shaders need GPU support (see Kevin Goldsmith’s article on GPU mixed with CPU and how this may or may not be good. However processors are speeding up and multi-core helps software rendering, the quality of GPU is well beyond what software rendering can deliver for a few years to come at least while architecture advances, probably more like 5-10 years.
• No UDP plans yet - Adobe has RTMFP, SL sockets has no public plans for adding UDP that I have seen
• No Alpha Channel in Video - You can do this with a shader though but not supported by default.

Silverlight 3 Video

Flash has the upperhand in video and probably will still even though SL3 has H.264.  Flash added this at the same time and though they still have FLV which revolutionized web video they are now much broader in support in video than SL3.  Silverlight has H.264 and VC1 support (their own FLV like codec).  Still pretty cool a couple years ago there was no HD on the web now everyone has it in H.264 video support.

Currently nothing innovative, mainly catchup still, but here are some options

Silverlight 3 beta and the video below the features and highlights will look very similar to flash and flash community advancements over the last couple years.  There is no innovation just yet.

But where that could happen is in socket support with UDP. Flash has moved on this in RTMFP and the beginning of larger scale networking support with UDP with samples like stratus.  This is a huge differentiating feature for what I think will be game changer on the web (it already is on desktop mmos) in real-time or closer to real-time support for larger sets of users in online games like MMOs or virtual communities, even tools to make request based real-time sites like micro-blogging faster and able to handle more users (right now it is very linear if users get many followers, UDP will allow a better distributed framework for messaging).

Local Storage

Silverlight and Unity3D all need this, Flash could use better support for this.  Local saving of a files for cache beyond the internet cache and greater than the 1MB/25MB limits of SL3 IsolatedStorage.  This is an issue when you are making large scale games in that you need to save lots of assets to a client but to make it economical you want ot save more than the default internet cache amount. Flash Shared Object (Local) allow you to do this somewhat but it would be great to have a way to just download files for cache (upon user agreement) to store assets in bulk of allowable types (images, video, models, bundles) to the file system.

Hardware rendering for 3d support and UDP support will put Flash and SL3 on par with the killer Unity3D kit for making online web games and other activex/plugins like instantaction that allow you to do these things already.

The one thing SL has over Flash

Flash and Flex are great. But there is this massive division in the community and marketing of Flash. Silverlight is entirely unified and this has much to do with starting clean at a time that interactive development is heading more into a technology and developers control. Flash and Flex need to bring it together. AS3 has been out long enough that the people with skills have hopped on and taken it to a new level, mainly from programmers. If Adobe created a version of Flash that was a new IDE and consolidated Flex and Flash into just Flash, made the IDE as powerful as FDT or FlashDevelop3 there could be hope to bring the platform together. I understand they had to work it in slowly because it was a designers platform really (even though coders still pushed the limits in games and apps built on it) so they had to tip toe carefully on this to not alienate people. But now I think the division is a serious problem with the platform and must be addressed, noone expected Silverlight to be this quick on at least SL3 features. And even though the initial approach might have been bad as SL1 was a huge letdown, Microsoft does not give up and you can see in the XBOX360 and DirectX that they are very pursuant. DirectX really didn’t become huge until version 7 so these guys won’t relent.

I am not a huge fan of using the proprietary tools. Even in Flash I use as much open source as I can even though the player is locked, but Moonlight is something that trails Silverlight development and is a very unique thing in both open source and cross platform/multiplatform development. It is a clear relationship and aims to make Silverlight run on multiplatform mono including Linux. This could win out in the end who knows.

Futures

Great iterations of software happened this week in the latest unity3d version (2.5) that has windows support and the iPhone SDK 3.0 and now SL3 is quite a surprise in feature set.

I have been really busy this week just delving into all them and hope to start making more cool and useful projects in them.  The best part is right now is great to be an interactive or game developer as all major software companies and markets are focused on retaining good developers.  I don’t’ recall a time other than the beginning the the web virtual land rush that has so many options and markets that skilled developers and designers can choose from.  Good times.