ezAR Startup Kit ver. 0.2.11 Update

Check out the updated ezAR Startup Kit. Key improvements include:

  • new Camera api for accessing the horizontal and vertical angle of view
  • new Ionic2 Sidemenu example project that demonstrates how to show a video preview on a custom page
  • fix snapshot plugin black screen output
  • internal changes in preparation for computer vision algorithms in upcoming updates

Download here.
Have questions, get help here.

ezAR Startup Kit ver. 0.2.10 Available

ezAR Startup Kit ver. 0.2.10 is available for download. This release provides minor enhancements and bug fixes for the VideoOverlay, Snapshot and Flashlight plugins. For the best results please upgrade your projects to use the plugins of this latest release. If you experience and issue or have an enhancement request post a comment to the ezAR Forum.

An enhancement that may come in handy for some of you is a new option to control the size of the Cordova WebView in relationship to the ezAR VideoOverlay plugin's CameraView. The option is named fitWebViewToCameraView. Previously the WebView was resized to match that of the CameraView. This behavior can be noticeable on Android devices where the CameraView is automatically resized to maintain the aspect-ratio of the best fitting video size available on the device that is equal to or smaller than the display size less toolbars. The default behavior will continue to fit the WebView size to the CameraView. But you can override this as follows:

                  ezar.initializeVideoOverlay(
                      successCallback,
                      errorCallback,
                      { fitWebViewToCameraView: false })

Best wishes with your development.

Square images (sort of)

I've had several requests for square image support which I'm planning to implement natively. In the meantime I've hacked up a simple utility named squareCam that will simulate a square cameraView. squareCam consist of the squareCam.js file which creates a mask that effectively squares off the classic rectangular cameraView of the ezAR VideoOverlay plugin for Cordova. You can customize the mask styling by modifying the squareCam.css file. The default mask is styled with a semitransparent diagonal hatched image (see below). 

Download here

Limitations & Unknowns

I have not yet tried using squareCam with a UI frameworks such as ionic. I'll give it a try in the next few days and share my finding and any updates required for compatibility. 

squareCam does not work with the snapshot plugin to create square snapshot images. This capability will be implemented natively when I implement square cameraView support to the VideoOverlay plugin. In the meantime, I imagine with a little javascript hacking a snapshot image could be automatically cropped to create a square image. 

Other News

I've been working on some interesting computer vision experiments for performing realtime tagged and arbitrary object detection as well as simple lane and symbol detection for an automated driver. I will share more details as soon as I've advanced this work a bit further.

HTML5 AR City Location Demo

The new HTML5 Cordova ezar-cities demo project is a simple augmented reality app that presents info cards overlaying a camera view. The cards contain the distance to a major city and are displayed only when the mobile device is oriented in the direction of the city. As you spin 360 degrees, info cards slide in and out of the view when the device is pointing in the general direction of a city with a corresponding info card.

The project source code is available here.

The app uses the device's geolocation, compass and accelerometer sensors to determine your location and the direction you are facing. The camera view is provided by the ezAR VideoOverlay.

The ezar-cities project is an updated branch of an obsolete XDK mobile project by Rakshith Krisrak found here. I first learned of this project in an article Rakshith published about 3 years ago along with the project. Since that time, Intel removed the video overlay support from XDK making this project unrunnable in its current state. Now using the ezAR VideoOverlay plugin with a few small changes you can revive that project just as I have. 

New ezAR Support for iOS WKWebView

Today we are releasing an ezAR update ver. 0.2.9 with support for the high performance iOS browser component known as WKWebView.

Download ezAR ver. 0.2.9

Starting in 2014 with the iOS 8 release, Apple began shipping two embeddable browser components known as webviews. Hybrid-mobile SDK's such as Cordova and Ionic use a webview for rendering the web UI content of a hybrid-mobile app. The standard component known as UIWebView is the default webview component used by Cordova and Ionic on the iOS platform. The alternative webview component known as WKWebView is much more performant that UIWebView. But it has not been well supported by the Cordova SDKs until recently. With the help of the Ionic team your iOS app can now take advantage of the WKWebView performance and reliability by simply adding a new plugin to it. 

The WKWebView plugin only supports iOS 9 and greater. Learn more about using Ionic's WKWebView plugin here.

Only the ezAR VideoOverlay plugin required updating to work with the WKWebView plugin. See the updated README file of example projects included in the ezAR Startup Kit v0.2.9 to learn about using alternative webview components such as the WKWebView for iOS and the Crosswalk WebView for Android.

Using ezAR Plugins with Intel XDK

EDIT:  On July 10, 2017  Intel retired its XDK cloud-based build services . The new build process involves exporting your project and then either building your app locally with the Cordova SDK or uploading the project to a cloud service such as PhoneGap Build services. Learn more...

Recently a developer pinged me requesting assistance in using ezAR plugins in the Intel XDK mobile designer. I knew that with the recent introduction of Crosswalk WebView support in the ezAR 0.2.7 VideoOverlay plugin it should be possible to use it in an XDK project to create an Android app. Having not used XDK in several years, I downloaded and installed the latest version. The process was straight forward and took me about 10 mins, including creating an XDK cloud account and keystore. I then imported the hello-ezar project included in the ezAR Startup Kit and added the VideoOverlay plugin to the project. I then fired off an Android build of the project. The build process took about 5 mins. The binaries (apk file) downloaded and installed with no issues. So far so good. When I launched the app, it started right up with the VideoOverlay camera view and WebView overlay's working perfectly.

WooT!

 

 

ezAR adds Face Detection For Cordova

Face detection is a popular image processing capability that can be used as part of an augmented reality solutions. Today ezAR makes it easy to add face detection into your hybrid mobile apps with the new ezAR Startup Kit ver. 0.2.8. The Startup Kit includes our new facedetector plugin for Apache Cordova and Ionic. This plugin provides native OS-level face detection capabilities for the Android and iOS platforms. In addition to the facedetector plugin you will find the ezar-facedetector example project (see screenshot below) in the kit as well as on GitHub here

Download ezAR Startup Kit

Image from ezar-facedetector app

Image from ezar-facedetector app

Quick Start With Face Detection

In a future article I will walk through the details for implementing a cool face detection feature but for now here's a quick outline of the process with a matching code snippet. Additionally see the FaceDetector plugin docs to learn more.

Step-1: Add the ezAR FaceDetector and VideoOverlay plugins to your Cordova project
Step-2: Initialize the VideoOverlay plugin
Step-3: Select either front or back camera on the device and start the video preview
Step-4: Start face detection and process the face rectangles returned by the plugin

 ezar.initializeVideoOverlay(
  function() {
    ezar.getFrontCamera().start(             
      function() {
        ezar.watchFaces(onFaces,handleError);
      },
      handleError) //end camera start
    handleError);
    
//face detection callback, receives array of face rectangles 
function onFaces(facesArray) {
  console.log('faces cnt: ', facesArray.length);
  for (i=0; i < facesArray.length; i++) {
    var face = facesArray[i];
    console.log('face: ', face.left, face.top, face.right, face.bottom);
  }
}
    
//general error handler
function handleError(errMsg) {
  alert('Error: " + errMsg);
}

Update required for building on latest iOS 10 Platform

In addition to the new face detection capabilities the VideoOverlay and Snapshot plugins have been updated to create iOS 10 compatible apps. As small change to the plists was required to meet new iOS 10 requirements for access the device's camera and photo gallery.

Stay In Touch

In closing please let us know how you are using ezAR and what we can do to make it better.  @ezARTech (twitter), info@ezartech.com (email) or online forum.

Ready for Android Crosswalk WebView

The ezAR plugins for Cordova are now compatible with the Android Crosswalk WebView. Download the new ezAR Startup Kit (ver. 0.2.7) and check out the details below to get started using ezAR with Crosswalk.

I've long been a fan of the Crosswalk WebView as a modern alternative to the default Android WebView component. The key benefits of Crosswalk include:

  • Your app can take advantage of modern web standards not supported on older Android devices
  • A predictable consistent web environment across all versions of Android going back to Android 4.0
  • High performance especially on older devices
  • Updatable WebView component
  • Used by Intel's XDK mobile development tool

There is a tradeoff you must accept when you add the Crosswalk WebView to your project.  That is the Crosswalk WebView engine (a Cordova plugin) will increase the size of your project by about 18 Mb. To me, this is an acceptable trade-off for the benefit of being able to use web features not traditionally available on older versions of Android, e.g., service workers.

Getting Started with the Crosswalk WebView

Getting started with the Crosswalk WebView is as simple as adding a plugin to your Cordova, PhoneGap or Ionic project.

 
cordova plugin add cordova-plugin-crosswalk-webview

Visit the Crosswalk project for more details on using Crosswalk in your project.

Note

I want to clarify that the ezAR ver. 0.2.7 Crosswalk support is only for the Android platform. If you visit the Crosswalk project site you'll notice there is support for mobile platforms other than Android. For now the highest priority ezAR demand has been for Android Crosswalk WebView support.

What's Next

For sometime I have had face detection working in the lab. Next week I plan to finish up a few open issues on that work and release it as a new face-detection plugin for Android and iOS. I'm looking forward to seeing what you do with this cool new capability. Cheers!

Update ver. 0.2.6 Available

The ezAR Startup Kit provides plugins to your Cordova mobile application that enable it to overlay any HTML content on top of a camera view with live video feed. Additional snapshot and flashlight plugins provide the ability to control lighting and capture images of an app's user interface including a video frame image. 
 
The latest version 0.2.6 of ezAR Startup Kit includes new snapshot plugin enhancements, a new example project and videoOverlay plugin fixes for a couple of Android specific issues. Download.

Example Projects Enhancements

  • Bundled the ezar-compass project. You can read about the project here.
  • Removed manual plugin installation requirement. The config.xml of each example project now includes plugin dependencies that are automatically imported into the project whenever you add a platform to the project, e.g, cordova platform add ios.

Snapshot Plugin Enhancements

  • New option for the snapshot() function. Provide a custom name for your snapshot image when saving it to the device photo gallery (Android only). On an Android device the following example saves a snapshot image to the photo gallery with the title pokemon-go and the filename pokemon-go.jpg.  

 
ezar.snapshot(successCallback,
              errorCallback,
              {
                name: 'pokemon-go',
                saveToPhotoGallery: true
              })
  • New options for the saveToPhotoGallery() function. Customize the image quality, scale, name and image encoding (jpg or png).  The name option is only supported on Android.

 
ezar.saveToPhotoGallery(imageBase64,
            successCallback,
            errorCallback,
            {
               name: 'pokemon-go',
               quality: 50,     //reduce image quality 50% of original
               scale: 75,       //scale down to 75% of original
               encoding: ezar.ImageEncoding.PNG
            })

VideoOverlay Fixes

  • Android - Removed overly aggressive use of hardware acceleration that resulted in poor WebView performance in some cases.
  • Android - Removed dependence on Cordova 6 API to retain compatibility with Cordova 5.

Pokemon GO Experiment - Part 1

This is the first progress report for an experimental project I'm developing to emulate the augmented reality experience of Pokemon GO (PG) using ezAR plugins for Cordova along with other open source technologies.  Checkout this short video.

Takeaways

The early key takeways from the project are that with simple technologies such as ezAR combined with a 3D library such as three.js a developer can make good progress towards creating cool fun AR experiences. Also I have a great appreciation for the design and technical skills of the Ninantic Labs team that created the Pokemon GO app. 

About

This is my first real development with the three.js and Physijs JavaScript libraries. If you are unfamiliar with these libraries three.js is WebGL 3D graphics library. I used three.js to layout the scene and the objects within it. Physijs is a physics engine that works with three.js and provides the 3d motion simulation for animating the Pokeball when the user launches it towards a Pokemon character. A third library that I made frequent use of is Tween.js for non-physics based animation such as spinning and vibrating Pokemon characters. 

Cube object in three.js design tools

Cube object in three.js design tools

Example Physijs simulation

Example Physijs simulation

I used the Cordova SDK to create Android and iOS versions of the app. The background Pokemon GO theme is played using the Cordova Media plugin.  Similar to the PG app, the music while cute at first can get annoying and earworm-ish after a few minutes of playing. One of the first settings I'll introduce is an option to disable the background music.  

Code

The code is very early in development and can be found here. YMIMV. While I successfully tested the project running on iPhone 6S and iPad Air both running iOS 9.3, I encountered a rendering  issue on my older Nexus 5 running Android 6.0.1. So I can not generally say how well the app will perform on Android.

Being a noob at three.js and Physijs there is much room for improvement. This is a veiled request for leniency from any seasoned three.js developers that may evaluate my work. I plan to reimplement the demo once I've gained a better mastery of three.js. 

Lastly, I will be posting a pure web version of the code soon so you can evaluate the experience directly through your browser rather than building an app.

Request

Please consider following my progress and giving a shout out if you like ezAR on Twitter @ezArTech.

Resources

code on GitHub   
Cordova SDK
Cordova Device plugin
Cordova Media plugin
ezAR videoOverlay plugin
ezAR snapshot plugin
physijs
three.js
tween.js

 

 

 

Using ezAR with Ionic Framework

One of the most popular UI toolkits for creating professional grade hybrid-mobile user interfaces is the Ionic Framework (Ionic). Ionic Framework is free, open-source and well supported by the Ionic team and user community.  In this article I walk through an example project I created using the ezAR videoOverlay and snapshot plugins in an Ionic Framework app. If you want to jump straight to the code see the link at the end of this article.

The project is named ezAR-Ionic and is based on the Ionic blank framework.  If you are new to Ionic or Angular I recommend you check out the tutorials on the Ionic site as I will cover the basic modifications to enable the ezAR videoOverlay's CameraView to be visible within an Ionic app. 

Step-1: Create Ionic project based on blank template

From commandline:

> ionic start ezar-ionic blank

This will create the directory ezar-ionic populated with a simple single page Ionic app.

Step-2: Add the ezAR VideoOverlay plugin which provides the CameraView

Change directory to ezar-ionic then from commandline:

> ionic plugin add pathToEzAR/plugins/com.ezartech.ezar.videoOverlay

Step-3: Initialize and start the videoOverlay CameraView when the Ionic platform is ready

//app.js
angular.module('starter', ['ionic'])
.run(function($ionicPlatform) {
  $ionicPlatform.ready(function() {
    if(window.cordova && 
       window.cordova.plugins && 
       window.cordova.plugins.Keyboard) {
      // Hide the accessory bar by default 
      //(remove this to show the accessory bar above the keyboard
      // for form inputs)
      cordova.plugins.Keyboard.hideKeyboardAccessoryBar(true);

      // Don't remove this line unless you know what you are doing. 
      //It stops the viewport
      // from snapping when text inputs are focused. 
      //Ionic handles this internally for a much nicer keyboard experience.
      cordova.plugins.Keyboard.disableScroll(true);
    }
    if(window.StatusBar) {
      StatusBar.styleDefault();
    }

    //add ezar block snippet below
    if (window.ezar) {
      ezar.initializeVideoOverlay(
        function() {
          ezar.getBackCamera().start();
        },
              function(err) {
                  alert('unable to init ezar: ' + err);
              });      
    }
  });
})

Step-4: Modify CSS to enable ezAR videoOverlay CameraView to be visible

By setting the HTML body and the ion-pane elements with a transparent background the ezAR CameraView will be visible below the Cordova WebView component.

/*style.css*/
    body {
        background-color: transparent;
    }

   .pane {
      background-color: transparent;
   }

You can add additional transparency to ionic elements such as headers and footers. See the link to code at bottom of this article which includes a semi-transparent header and footer.

Step-5: Add a platform to the project and build it

From commandline add the platforms you plan to support. Then build the binary app for each platform.

> ionic platform add android
> ionic platform add ios
> ionic build

Step-6: Install and evaluate the effect

Note: ezAR plugins will not perform correctly on virtual devices that lack true camera sensors.

To install your app use your favorite mobile app deployment method. For example I use GapDebug (free) for simple drag-drop app installation and debugging.

Code

The code to my starter ionic project is available here on github. The screenshot above is from the ezar-ionic sample project that also includes the ezAR snapshot.

 

 

 

 

Pokemon GO - Simple AR Solution, Massive Success

Image courtesy Pokemon Video Gallery

Image courtesy Pokemon Video Gallery

In a very short time the mobile app, Pokemon GO has become a global mega-hit surpassing Twitter in daily average use. As a new player of the game my experience is limited but I'm impressed. If you're unfamiliar with the Pokemon GO app, it is a mobile game that employs geocaching and augmented reality to guide the user to virtual fictional creature that you capture and score points. 

As an AR developer what struck me is the simplicity and effectiveness of the game's use of augmented reality. I've reviewed a number of augmented reality app ideas from developers interested in using ezAR. In most cases the app idea was overly complex and had unrealistic expectations especially of the technical demands the app would place on your average mobile device. Years ago I adopted advice from Craig McCaw that "simple solutions can solve complex problems. Complex solutions rarely solve anything." 

I believe simplicity is a strategic quality that we should strive for in the design of our solutions, especially where augmented reality is applied. Before you incorporate augmented reality in your product, download the Pokemon GO app and learn how a simple solution can be engaging and fun. 

In my next blog I'll share how to create a simple Pokemon GO style app using HTML and ezAR plugins for Cordova and Ionic Framework.

Update ver. 0.2.5 Available

Updated

A new version of ezAR is available.  Download here.

VideoOverlay Plugin Changes

  1. Android - modified strategy for camera resolution selection and UI resizing to reduce letterbox mattes
    (black bars on opposing sides of the UI when camera aspect ratio does not match the display area). This
    change is most noticable when in fullscreen mode, a.k.a., immersive mode.
  2. Fixed error in initializeVideoOverlay(). On older OSes the JavaScript String.startsWith() method is not
    available.

Snapshot Plugin Changes

  1. New quality [0-100] and scale [0-100] options property for snapshot(successCB,errorCB,options).
  2. New saveToPhotoGallery(base64ImageData) function. Use this function along with the snapshot function
    to implement a preview-before-save UI for selfies or other images you capture in your app.

Update ver 0.2.4 Available

An new ezAR update is available, version 0.2.4.

 This is a minor release with several small enhancements and bug fixes. 

What's New

  • Added backgroundColor option for initializeVideoOverlay  
  • Added snapshot options to include/omit the cameraView and webView's respectively
  • Fixed camera freeze up that occured when saving snapshot image to photo-gallery and the permissions dialog appears.
  • Fixed distorted snapshot image that occurs on the first use of an iOS application

ezAR Snapshot Plugin for Cordova

One of the recent enhancements to the ezAR toolkit is the introduction of the Snapshot plugin for Cordova (Snapshot) to capture screenshots of a Cordova app user interface. The Snapshot works in conjunction with the ezAR VideoOverlay plugin or can be used standalone in any Cordova app, e.g., selfie app. The snapshot function can be configured with options to save the snapshot image to the device picture album (default) and to generate the image in either JPG (default) or a PNG format.

A key difference between ezAR Snapshot and an OSS snapshot plugin I reviewed sometime ago is that ezAR Snapshot will capture a combined image of the a video preview, if present, and the Cordova WebView content. This is a non-trivial process as there is no iOS or Android api that will directly grab a bitmap of the display. Instead the plugin has to grab a video frame and combine it with an image of the WebView content on the fly. 

Here's a snapshot image I created of a live CSS landscape app I'm working on, i.e., Snapshot used in standalone mode.

Getting started with the Snapshot plugin is easy.

1) Download and unzip the ezAR Startup Kit which includes the Snapshot plugin for Cordova

Download here

2) Install snapshot plugin using the Cordova CLI

cordova plugin add <path>/com.ezartech.ezar.snapshot

3) Call the JavaScript snapshot function with options

ezar.snapshot(
  function(aBase64Image) {
    //do something with image here
  }, 
  function(err) {
    //handle error
    alert('Error: ' + err);
  }, 
  { //options
    "saveToPictureAlbum": true,  //default
    "encoding": ezar.ImageEncoding.PNG //JPG default
  });

That's it. When this snapshot function is called it will create an image

An issue you may encounter is unwanted HTML artifacts such as a snapshot button appearing in your snapshot image. In my JavaScript function that controls the snapshot process, I first hide unwanted HTML artifacts. Then I wrap the ezar.snapshot() call in a setTimeout. This gives the DOM a chance to update the UI with the unwanted artifacts hidden from view before the snapshot image is created. Here is an example from my ezar-selfie app.

function doSnapshot() {
  document.getElementById('snapBtn').style.display = "none";  
  setTimeout( function() {
    ezar.snapshot(
      function() {
        document.getElementById('snapBtn').style.display = "block";
      },
      function(err) {
        alert('Error: ' + err);
        document.getElementById('snapBtn').style.display = "block";
      },
      {"saveToPhotoAlbum": true});
   },10);
}

The next time you're developing an app and think "hey it would be great if my users could capture an image of the display", think of the ezAR Snapshot plugin.

 

 

Using ezAR with Three.js - Cube Example

Today @jouderianjr tweeted asking if ezAR can be used with the 3D JavaScript library, Three.js? I have had the task of creating an ezAR demo that includes 3D objects overlaying a video preview on my plate for the last couple of weeks. Jouderian's tweet prompted me to follow through and create a simple rotating 3D cube overlaying a video preview created with the ezAR VideoOverlay plugin for Cordova.  A src link provide at the bottom of this article.

 

I tested this demo on an iPhone 6S, Nexus 5 & Nexus 7. The runs great on my iOS devices. On my Android device, both of which are running Android 6.0.1 I encountered WebGL errors. I'll be looking into these issues and provide more details. Also I plan to develop a more interesting demo using three.js. 

The src code for the demo is a modified version from Ajeet Yadev's article, Creating 3D Cube: A Practical Guide to Three.js.

Catch me on twitter at @ezARTech.

Cordova JS-native Bridge Performance

ezAR provides a set of Cordova plugins for accessing native device functionality such as the camera and light from a JavaScript api. Recently I received a question about how long does it take the Cordova runtime to make a JS-native function call? Is it slow? Is it fast? What is it? From experience I knew it was quick enough for most of my use-cases. But I did'nt know the exact numbers. To answer this question I created a small app and Cordova plugin to measure the time it takes to make a native NOP function call from a Cordova app. See source code here

The app built with Cordova 6 SDK performs a 1000 calls to a NOP plugin's doNop() JavaScript function which in turn immediately calls the native doNop() implementation with no side-effects.  The native doNop() function returns immediately with a success status. Here's the results:

iPhone 6S (iOS 9.2.2)

Android Nexus 5 (v6.0.1)

The sub-millisecond median time is a non-issue in most use-cases for the benefit of accessing native resources. Also notice how the first use of the api is significantly slower than the median time. From this we can see how JS optimization kicks in with dramatic improvement. This info and a few other tests that I plan to run (e.g., structured data transfer) will be helpful in determining the design and implementation of future ezAR functionality.

Lastly, I used techniques and guidance from Measuring JavaScript Performance to compute these results. Specifically the performance.now() function was new to me.

ezAR Compass updated with new Snapshot feature

The ezAR Compass app has been updated to include the new snapshot support introduced in ezAR 0.2. The Snapshot plugin works standalone or with the other ezAR plugins.  This new versatility to work standalone is one of the big new qualities of ezAR.  Use this app as a guide for how to add a snapshot capability to your Cordova app. 

You can find the src code here.