Vuforia SDK + remote video streaming on iOS

I recently have undertaken a project on iOS that requires integration with the Vuforia SDK. It’s an augmented reality proprietary framework, built for iOS and Android and has been very popular due to its innovative recognition library. One of the coolest demos that are appealing to advertisers or people looking to incorporate commercial campaigns inside their applications concerns the ability to play a video on top of a target. Vuforia even provides a sample application for that. However,  remote video streaming on texture does not work on textures.

This is a long standing issue, with people on the forums asking for a solution, some providing either free solutions which are outdated and / or non-performant, or paid solutions that are very expensive.

The video-on-texture-rendering process in general

In order to stream a video onto an OpenGL texture, the following actions must happen:

  1. Initialise a renderable surface. This is a once-per-session operation.
  2. Create a texture with an ID.
  3. Assign the texture to this surface (applying shaders, etc, etc).
  4. On each render, apply transformations to the renderable surface, according to the coordinates of the recognised object.
  5. Get the video bytes and decode them to get actual video data.
  6. Convert the video data to OpenGL data (ready to be drawn)
  7. Apply those video data to the texture you have gained from step 2.

Steps 1 – 3 are already happening inside Vuforia’s sample. Step 4 is also the reason why the vuforia SDK exists; to give you the transformation and coordinates of a recognised object inside the world coordinate space. Therefore, step 4 is also included in the sample (and all samples from vuforia).

The difficult part, and the part that Vuforia SDK is not responsible for, is step 5 – 7. This is where we, the third party developers come into play.

The actual problem with Vuforia’s sample:

As I have already mentioned, Vuforia’s SDK is only responsible for recognising an object into world space, and providing you with its coordinate and transform. What you do with these information is up to you. Therefore, Vuforia’s VideoPlayback sample should be taken as a demonstration of what you can do with it, not see its limitations.

Inside the sample Vuforia makes heavy use of AVAssetReader and AVAssetReaderOutput in order to perform the following actions. As many people have already pointed out in the forums, AVAssetReader is responsible for reading from local file URLs, and does not support remote files.  So, Step 5 in the video-on-texture-rendering is the problematic one, as you need to decode the video data you get from a remote location to actual OpenGL data, and then render those data on-screen. Many people have said in the forums that remote on-texture rendering is not possible on iOS.

This couldn’t be further from the truth.

The solution

What we need to do is to get the actual OpenGL data ready to be rendered, and apply those data as a texture onto the texture created by Vuforia. The SDK and the sample have already created an OpenGL coordinate system, so all that’s left is to get the OpenGL data, and divert the data flow from the original sample code.

Instead of using AVAssetReader, we are going to use AVPlayerItemVideoOutput, which was introduced in iOS 6. This class has the method – copyPixelBufferForItemTime:itemTimeForDisplay: , which is exactly the one that we want to use, in order to get the raw OpenGL data to render on the texture.

The following code samples are intended to replace / update the corresponding functionality on Vuforia’s VideoPlayback sample. The code can certainly be improved.

First, let’s set up the the video player, and the video output item, in order to later extract the video buffer contents.

After each frame is called, the -updateVideoData is responsible of preparing the video data for display. The following code is a  modified sample code from Vuforia, and uses -copyPixelBufferForItemTime:itemTimeForDisplay: in order to extract the streamed video content, and bind it with the OpenGL texture that is being rendered at this point.

There are also some other interventions we must do, in order to change the setup of the video output.

Those are all the changes one can do in order to setup the video playback and render to texture the video streamed. However, Vuforia’s sample must also be updated in many areas in order to understand that now remote videos CAN be played.

 

That’s it! You may need to do some more minor changes, but this is the general concept in order to make the tutorial run. This methodology has been tested with Vuforia 4.0, and works perfectly (and is also used in an application released to the app store)

Want the full source?

Before you download the source, please understand that there are many optimisations to be made to the example. Vuforia’s example is constructed to support iOS 4, and as such, if you target iOS 6 and later, you can get rid at least half of the code, you can convert the project to ARC (which is certainly advised), and you can also optimise the video playback to use hardware acceleration. I have implemented all of these functionalities to my released applications, however, it would be confusing writing a tutorial here that would cope with many problems at once.

Grab the source here!

Edit 2017-06-14

I haven’t touched Vuforia’s SDK for a while. I will not be able to offer support with the newer SDK versions.

I have received some comments mentioning that this solution does not work with the newest versions of the Vuforia SDK.

The point is, however, is that it is impossible by definition for this solution to not work. Maybe the code will change, but the solution is valid, since it features streaming a video from a video source, decoding the data to OpenGL using CMSampleBufferGetImageBuffer(), and rendering the OpenGL data to a texture.

I’m sorry for the fact I am not able to offer code-level support for this solution but the methodology is still valid. It requires a little bit of effort and OpenGL knowledge, but it’s definitely doable.