Augmented Reality Applications at VTT
Charles Woodward
VTT Technical Research Centre of Finland
Augmented Reality (AR)
Adding of virtual objects in view of real world C.f. Virtual Reality (completely virtual worlds) Properties: real-time, interactive, 3D registration
<-> Challenges: tracking, accuracy, speed, mobile
Introduction - AR
Related technologies
Augmented Virtuality Mediated Reality Diminished Reality, …
<->Mixed Reality (MR)
Reality / Video image
Virtual Reality / Computer graphics Augmented
Reality
AR at VTT
Augmented Reality / 3D Tracking research at VTT
Established in 2000, team based in Espoo & Oulu AR/MR core technology & high-end applications Supporting / application teams around VTT
Leading edge technology
First in the world with various AR applications Licensing for independent developers (ALVAR SDK, open source & commercial versions)
Seewww.vtt.fi/multimedia
Applications, incl.
Games, Entertainment Print media, Advertising Industrial operation, Assembly Interior design, Furniture, Rendering
Architecture, Engineering & Construction (AEC) Virtual Worlds Collaboration, Telepresence
Early Applications: Games & Entertainment
SymBall
Vappunaamari.com
Consumer Applications: Print & Advertising
Nestle bunny on cocoa package Dibidogs in Katso &
AR children’s book
DNV (Det Norske Veritas):
Technology Outlook 2020 Augmented business
cards (Painonet)
Veijarit & Vares movie trailers in Seiska magazine Sparkly in Elle magazine
https://youtu.be/AjDjsmr0G14
https://youtu.be/aR-g9RbYUYs
https://youtu.be/s1WfsnjJdu4
Interior Design
VividAR
AR interior design system by VividWorks, based on VTT technology Free download, various furniture retail companies around the world
Incl. Diminished Reality
https://youtu.be/lp5plNOT20U
Photorealistic Rendering
Challenges, e.g.
Materials, incl. reflections, transparency Lighting: direction, intensity, colors, soft shadows
Accounting for real world lighting conditions Matching computer graphics with webcam video quality
Comparisons
Left - standard 3D graphics Right - using VTT methods
https://youtu.be/fhFzStkoE50
Mixed Reality Telepresence
Connecting Real and Virtual Worlds
Mixed Reality teleconferencing between real people and Second Life avatars
Avatars appear at meeting room with real people, following real people’s gestures
Implemented in co-operation with IBM Corp. (USA)
https://youtu.be/DNB0_c-5TSk
Industrial Applications: Production Sites
Augmented maintenance, operation, training Plant asset management: Simantics Mobile Robot simulations in real environments
Partners: Wärtsilä, Metso, Valtra, Cargotec, Fortum, ..
Augmented Assembly
Pilot case
Valtra (tractor company) Hydraulic block (200 pcs)
3D puzzle
Data Pipeline
3D CAD STEP DFA CG AR
Project AugAsse
https://youtu.be/QSwZ5CINaPk
AR Maintenance
Project S-STEP
Smart technologies for lifecycle performance
Knowledge sharing solutions for field service personnel
Collecting tacit knowledge, multimodal interaction, mobile AR Industrial partners: Konecranes, KONE, Wärtsilä, others
VTT tasks, pilots
AR maintenance Remote assistance Wearable interfaces User experience, acceptance
AR Applications in Space
Project EdcAR
Commission for European Space Agency (ESA) VTT with partners TAS-I, TAS-F, ICCS Budget 0,5 M€, duration 2015-2017
AR solutions for space domain maintenance
Specification of AR solutions to address identified critical tasks and activities Selection of suitable technology components Demonstrators for identified use cases
Architecture and Construction
AROnSite system
Features: Photorealistic rendering, 4D etc.
Devices:PCs, mobile phones, tablets
Application cases, incl.
Skanska offices in Ruskeasuo, Helsinki Hotel complex, Billnäs Ironworks at Raseborg Kämp Tower, Jätkäsaari in Helsinki
First in the world to involve mobile AR at construction sites, and in real urban decision making cases
https://youtu.be/wvSPzG7AqLY
https://youtu.be/c1DUkVhqVsM https://youtu.be/rVt86NGXQv4
Some Recent Cases
Applications
Real estate, marketing
Citizen engagement, participatory design
Markerless 3D Tracking by VTT
Markerless tracking
Early experiments, Forchem case 2010
Model-based initialization + real time feature tracking
Point cloud based tracking
Properties: automatic, accurate, fast, robust Single solution to handle wide range of applications International licensing: Inglobe (Italy) / AR-Media
https://youtu.be/EUmNbNa3RYY https://youtu.be/WOVjISxlhpU
https://youtu.be/xpZDXih9zLg
Indoor Navigation
Tracking solutions
Point clouds obtained from photos Using Photoscan / VisualSFM tools Dynamic loading of point clouds
Updating of point clouds, edge computing Autonomous robot to collect & update point clouds
Example
Navigation to vending machines, find food items Operating instructions, showing which buttons to press
Facility Asset Management
Project DigiSpaces
Mobile AR visualization and interaction with Building Information Models (BIM) & service book
Applications: building's life cycle services, e.g.
maintenance & repair
Partners: Skanska, Granlund, Pöyry, Tekla, Solibri, Nokia
Functionality
Navigation to service targets
Interacting with targets by clicking in camera view
“X-ray vision”, seeing targets behind walls etc.
Tracking
Based on 3D point clouds
Supported by IMU, locationing using Nokia’s HAIP
User tests
Collecting user requirements
Evaluation, most positive feedback https://youtu.be/uYFtYbqvoq0
Mobile AR for Lean Construction
Project VisiLean
In co-operation with Aalto University Combining lean production practices with BIM, visualization by AR
Point clouds derived from panoramic images, coupled with photogrammetry data
Web based, device independent user interface, AR mode for all functionality
World Browsing
Task
Augmenting the camera view with point-of-interest (POI) information
E.g. tourist info, mobile advertising
Challenge
Current world browser solutions are based on GPS and compass, ínaccurate, bad user experience Accurate tracking of mobile device (both location and rotation) required
Solution
3D point cloud based tracking
Real-time implementation optimized for mobile phones
Providing extreme accuracy, robustness and speed Technology licensing to Nokia
Visualizing the Past
Augmenting historical photos
Pilot cases in Turku and Helsinki
Augmenting historical photos at real locations Showing how city used to look in the good old times
Projects
Futuristic History & Street Smart (EIT Digital)
ALVAR Subroutine Library
”A Library for Virtual and Augmented Reality”
Computer vision based 3D tracking for AR
World leading performance, accuracy, robustness, speed
ALVAR Desktop
Windows / Linux – marker based –Open Sourced
Users incl. Google, NASA, MIT, Vuzix, Columbia Univ., ROS
ALVAR Mobile
Marker, multimarker and 2D image tracking
iOS, Android, Windows Phone, Windows, Mac, Flash, Silverlight
ALVAR Tracker
Point cloud based 3D tracking, SLAM, sensor fusion … iOS, Android, Windows Phone, Windows, Mac
ALVAR Unity Plugin
Content authoring, point cloud editing tools
https://youtu.be/xc-4B4CljHI
https://youtu.be/hWwlxBefg_0
https://youtu.be/ttgHKtyKHKw
VTT Tracking in Space, by NASA, Google, …
NASA Spheres robots in zero-gravity
Co-operation with NASA and Google
ALVAR multimarkers used for evaluating ground truth of Project Tango tracking See Google videohttps://www.youtube.com/watch?v=Vc5YyLl1Ksg&t=130 See NASA video:https://www.youtube.com/watch?v=psZ6UAFUOT4&t=305
NASA Hedgehog robot in micro-gravity
Co-operation between NASA, Stanford and CalTech
ALVAR multimarkers used for orientation tracking of robot, under fast movement See news & videos:http://www.dailymail.co.uk/sciencetech/article-3222992/Nasa- reveals-bizarre-hedgehog-robot-roll-fall-alien-planets.html
