Projects in Virtual and Augmented Reality@DETI
Paulo Dias
RVA 2019-2020
Outline
• 3D Reconstruction
• RVA
• VR Frameworks
• RVA@deti
– pSIVE (Platform for Setting up Interactive Virtual Environments)
– Museu da Cidade de Aveiro
– VR4Rehab (Edição RVA 16-17 – videos) – Continuous AR
– Bosch
– Produtech
How to start a presentation – Top 3
• What is Virtual and Augmented Reality and what do you need to do it?
OR
• What are we doing at DETI in Virtual and Augmented Reality besides teaching the RVA course?
• 3: Start with a question!
How to start a presentation – Top 3
• 2: Shocking Facts
• AR/VR 3.33 Billion USD in 2015
• Expected 133.78 Billion USD by 2021
How to start a presentation – Top 3
• 3: Tell a story
• Ola, Eu sou o Paulo Dias!
3D reconstruction – Paulo’s PhD 1998 - 2002
• Available for small models
Shape Grabber
Minolta Vivid
Marble Statue of Aphrodite scanned with the VIVID 910 using the rotary stage option
3D reconstruction
• For larger models (building, room) one
possibility is to use “Laser Range scanners”.
Riegl
Zoller & Frölich
Cyra
SICK adaptado
3D reconstruction
Point cloud Reflectance
1000 x 175 (Riegl LMS Z210)
3D reconstruction
Triangulation
Registration (ICP) Digital Photos
Texture 3D Model
3D reconstruction
Ieeta Assemblée nationale Farmhouse Laveno
Adapted Sick (Aveiro) Riegl (Italy)
3D reconstruction
3D Reconstruction – google Tango (discontinued)
Ieeta amphitheatre
PhD - September 2003, 23
Why Virtual Reality?
• 3D information is common (3D reconstruction, games, CAD, Design, etc…)
• Interaction is still mainly 2D – Keyboard and mouse since 70s’
Apple II – from 1977
Reality Augmented Reality Augmented Virtuality Virtual Mixed Reality
Virtual reality Continuum
Projecto de 5º ano:
Mário Cruz Paulo Moreira 2004-2005
Low cost environment for Virtual and Augmented Reality
Virtual reality
• Equipment
– Tracker 3DOF (Yaw, Pitch, Roll)
– HMD SVGA stereoscopic
– Accelerometer– [Electrónica e Sistemas Médicos / DETI]
pose detection.
• Graphicl libraries
– OpenGL
– VTK (Visualization Toolkit)
Augmented Reality
• Equipment
– Virtual reality + Camera – Video see-through
• AR Toolkit
– Marker detecction
– Superposition of virtual elements
Magic book prototype - 2005
Results
Challenges in VR/AR
• User interaction with world – Input Devices
• User immersion in world – Output Devices
• =>No unique solution: several attempts
with different solutions
Input: Motion trackers
– Magnetic – Mechanical – Acoustic – Optical – Inertial
Polhemus Fast-Track
Leap
3D Mouse - Logitech
InertiaCube 3 - Intersense
Finger Tracking - ART Kinect
Razor Hydra
Output
– Monitors
– Surround screens – Workbenches
– Hemispherical Displays – HMD
– Arm Mounted Displays
Hardware@deti - Input
– Intersense 3DOF Inertial Tracker [2009 - 1940€]
– Intersense 3DOF Inertial Track. BT [2012 -1437€]
– Wintracker [2009 - 1285€]
– Phantom Omni [2012 – ~3500€]
– Kinect v1.0 (3) [2012 - 300€]
– Kinect v2.0 (1) [2014 - 300€]
– Swissranger [2012 – ~3000€]
– Leap Motion (3) [2016 – 60€]
– Google tango HTC and Lenovo [2015 – ~500€]
– Orbtec Astra [2019 – ~250€]
• Trackers:
– 3DOF wired and wireless – Wintracker 6DOF
– Google Tango – Leap Motion – Kinect 1/2 – WiiMote
– Razor Hydra – Phantom
Virtual Reality - Hardware
Hardware@deti - Output
– I-glasses SVGA HMD [2004 - 800€]
– VR 2000 HMD [2012 - 1359€]
– Oculus DK1 [2014 - 420€]
– Oculus DK2 [2015 - 500€]
– Gear VR + Game Pad [2016 - 250€]
– Samsung Galaxy S7 [2016 - 650€]
– HTC VIVE [2016 - 1350€]
– Meta Glasses [2018 – ~1800€]
– Xperia Projector [2018 – ~3000€]
– Myo gesture ArmBand [2018 – ~100€]
• HMD:
– Oculus DK1,2 – HTC VIVE
– Samsung Gear VR – HMD SVGA (2)
• I-glass
• VR 2000
Virtual Reality@deti – Hardware
Hardware@deti - Input
VR Frameworks Introduction - 2013
• Creating a Virtual Environment
– Cost (Time & Money) – Specific knowledge
• Several frameworks/tools
– Specific implementation – Particular architecture
• How to ease?
– Tools for ‘laymen’
– Flexible yet simple
Graphics engines
• OpenSceneGraph
– open source C++ cross-platform toolkit for simulators, virtual reality and games
– several NodeKits that fulfill a variety of purposes, such as particle
systems, text display, special effects, shadow , 3D interactive controls ,…
• OpenSG
– real-time graphics programs, e.g. for virtual reality applications similar to OpenSceneGraph
• VTK
– Open Source library based in Object Oriented Programming – multi-platform (Windows/Unix/Mac)
– C++ implementation with several wrappers for other languages (Tcl/Tk- Python-Java)
Real-time physics engines
• Bullet
– Free, open source – Collision detection
– Soft and rigid body dynamics
• Newton Game Dynamics
– Cross platform open source
– Simulates rigid bodies in games and other real-time application
• PhysX
– multi-platform game physics
– support several devices (smartphones, GPU, CPU) – Integrated in several game engines
– Acquired by Nvidia in 2008
– SDK provided to developers for free
3D Computer Graphics software
• Blender
– free/open-source 3D computer graphics software used for creating animated films, visual effects, art, 3D printed models, interactive 3D applications and video games.
– include 3D modelling and several effects: texturing, raster graphics editing, rigging and skinning, fluid and smoke simulation, particle simulation, soft body simulation,…
– integrated game engine.
• OpenSpace3D
– Free Open Source development platform for interactive real time 3D projects
– User friendly, possible to create interactive 3D scene without writing any code.
• 3D Studio Max
What else?
• Something missing?
Game engines
• Unity
– cross-platform game creation system that includes a game engine and an integrated development environment.
– Graphic working environment that facilitates 3D scene creation and layouts
– Physics engine that provides collision detection, forces, torque and physic materials
– Two licensing options Unity and Unity Pro
• Unreal Engine 4
– Multiplatform game development toolkit supports state of the art graphical fidelity
– full access to the source code of the engine
– In 2015 released for free usage with a 5% commission on commercial products.
– PhysX 3.3 physics engine for physical simulation
Virtual Environment Frameworks - not desktop
• CCG Cave setup /Guimarães
Virtual Environment Frameworks - not desktop
• inVRs
• VRJuggler
• Vizard
• 3DVIA
• others
inVRs
– Institute of Graphics and Parallel Programming Johannes Kepler University, Linz, Austria (2006)
– C++
– OpenSG and OpenSceneGraph as a scene graph engine
– Network distributed virtual world
– Pre-defined navigation and interaction techniques – Configurable via XML
– Mostly used on academic environments
inVRs Applications
VR Juggler
– Iowa Center for Emerging Manufacturing Technology, Iowa State University, Iowa, United States (1998)
– C++
– Supports many graphics Engines – OSG, OpenSG, OpenGL, VTK…
– Huge variety of modules
– Network distribution through NetJuggler – Extended over time – Ex.: VR JuggLua
– Used for many different purposes – Commercial / Academic
• Mina do Lousal (PT)
• Korea Ocean Research and Development Institute
– Configurable via VRJConf – Backend XML
VR Juggler
Vizard
Vizard Development Edition
Vizard Development Academic - Single License $ 3,990 Vizard Development Academic - SITE License 5 Seats $ 6,990
Vizard Enterprise Edition
Vizard Enterprise Academic - Single License $ 5,990 Vizard Enterprise Academic - SITE License 5 Seats $ 9,990
• WorldViz, Santa Barbara - CA, United States
• Pyton Scripts + GUI
• OpenSceneGraph - Possible to expand vizard funcionalities
• Support many comercial Devices
• Physics Engine
• Research & Science (VHIL - Stanford U), Architecture &
Construction (Archidimex - Netherlands)
Vizard
IDE
• Dassault Systèmes, Vélizy- Villacoublay, FRANCE
• Former Virtools
• Closed Engine
• Many GUI Tools to work on models, animations and scenes
• Price around 9000 USD + 3000 USD/yr
Source:
http://www.3ds.com/fileadmin/PRODUCTS/3 DVIA/3DVIAVirtools/demoshowcase/html/bro wse.html?br=1
3DVIA Studio
Peripherals Interfaces
• Virtual Reality Periphery Network (VRPN)
– network-transparent interface between applications and physical devices used in VR
– client-server approach
– layer of abstraction that classifies inputs into several categories (Analog, Button, Dial, Force Device,
Sound,Text, and Tracker), allowing to receive generic
input from different devices.
Other Frameworks
• FreeVR
• Avango NG
• DIVERSE
• ViRAL
• EON Studio
• …
Conclusion – VR frameWorks
• Difficult choice, many solutions
• Complex Architecture – Many Abstractions
• Often not trivial to Install / Configure
• Game engine emerging as alternative to expensive frameworks – Are almost
standard in research today
Download and install – Break - EdupARk & Unity
http://edupark.web.ua.pt/
https://play.google.com/store/search?q=edupARk
https://unity.com/
Projects in Virtual and Augmented Reality@DETI
Paulo Dias
RVA 2019-2020
Motivations for VAR projects
• Foster R&D in Virtual and Augmented Reality areas
• Investigate new interaction and navigation techniques
• Devise and demonstrate new scenarios for
the use of these technologies
Environment to configure RVA environment
Mestrados:
Danilo Souza João Parracho João Cardoso João Pinto
Interaction in Virtual reality
pSIVE – Danilo Souza 2013/2014
• Platform for Setting up Interactive
Virtual Environments (pSIVE) 2013/2014
– Virtual Visits – Training
– Marketing
• Easy Creation
• Abstraction Layer
– Users – Frameworks
System Requeriments
• Hardware Independent
• Simple Configuration
– 3D Model Loading
– Attatch/Display Multimedia Information – Technical aspects
– Reusable
• Well Defined Art Pipeline
– CAD softwares -> pSIVE – Avoid calculations
• Cover part of IEETA’s devices
Architecture
• VR Juggler
– Qualities met project requeriments – Active Community and Develop – Well documented
• OpenSceneGraph (OSG)
– Easy-to-work-with solution – Books available
– Very active project
– Support vast list of formats/software
• 3D Studio Max, SketchUp, Maya, Blender...
• Why not Game Engines?
Platform Overview
Multimedia Module
Interaction Module
Menu Module
Case of Study
PRODUTECH-PTI / Tegopi
VR for training and formation
Update pSIVE – João Pinto 2014/2015
• Support for newer HMD’s (oculus rift)
– OpenScenGraph with osgOculusViewer
• Kinect support
– VRPN and FAAST
João Pinto 2014/2015
Architecture
VR – Full body interaction- 2015
Interação em RV – Full body interaction
Interação em RV – Manipulation
Mobile Interaction – Luis Afonso 2015-2016
• Move to Unity
– Oculus, leap and Android support
Architecture
VR Interaction with mobile devices
VR Interaction with mobile devices
User evaluation in Virtual and Augmented Reality 2006-2007
Mestrado em Gestão de Informação:
Angela Pimentel Amado
Projecto de licenciatura:
Jan-Willem Baggerman
User evaluation
Virtual reality - User evaluation
Virtual reality - User evaluation - results
• Objective: Test navigation and visualization
• Methodology: Quantitative and qualitative
• 44 users (12 women / 32 men);
2 environments RV x DV
- HMD - Monitor - Mouse - Keyboard
- Tracker - Mouse
results
- Preference for RV environment;
- RV causes more dyziness and disorientation;
- navigationa and visualization easy in both;
2 Projectos 5º ano
Ricardo Casaleiro Ricardo Seco
Sound in VR
Sound in VR
• Auralization in closed spaces
• Use of HRTF
• Computation of first order reflections ( ray-tracing)
• Computation of RT 60 with differente materials
• Visualization and auralization in VR environments
• VTK, sound processing libraries (PortAudio,
libSoundFile, ...)
Stereo vs 3D Sound
Software to edit acoustic properties in 3D models to compute reverberation times
Sound in VR
3D reconstruction and auralization Anta Pintada de Antelas
• Auralization with first order reflections and
HRTFs
Auralization: Reflections
Som directo
Primeiras reflexões
reverberação tardia
Tempo dB
Tempo de reverberação Atraso do som directo
Atraso das primeiras reflexões
Atraso das reverberação tardia
Auralization final demo AcousticAVE project
Bolseiros:
Bernardo Marques João Alves Tiago Araujo Luis Afonso
Augmented and Virtual Reality projects
Mestrados:
Ricardo Silva
Raphael Carvalho
Anna Andreikanich
Rafael Esteves
VR4Rehab - Rovisco Pais
Continuous AR
Task 1: Catch the pumpkin Task 2: Count the spider
Continuous AR interfaces
AR for maintenance
AR for education
• EdupARk
Download and install - EdupARk & Unity
http://edupark.web.ua.pt/
https://play.google.com/store/search?q=edupARk
https://unity.com/
Unity Installation - Instalation
• Unity Installation
• Unity Account Asset store
Questions?
To do – Next week
• Install Unity
• Create Asset store account
• Remember to select for presentation:
The people!
• Beatriz Sousa Santos
• Guilherme Campos
• Joaquim Madeira
• Samuel Silva
• Os muitos alunos que trabalharam nos projectos:
– Ricardo Casaleiro, Ricardo Seco – Gustavo Corrente, Ana Cantanhede – Angela Pimentel
– Jan-Willem Baggerman, Pauline van der Horst, Frank van Huussen – Mário Cruz e Paulo Moreira
– Rui Palha, Filipe Duarte, Igor Cardoso, Tiago Sousa, João Parracho, João Cardoso
– André Oliveira – Danilo Souza
– José Mendes, João Rodrigues, Ricardo Silva, José Sequeira, Vasco Santos, Miguel Vicente, Joel Pinheiro, André Alves, Luis Morais
– João Pinto, Sérgio Eliseu, Luis Afonso, João Cardoso
– José Ferrão, Miguel Ferreira, Pedro Coelho, Rafael Almeida, Diogo Carvalheira – Raphael Carvalho, Anna Andreikavich, Ricardo Silva, Rafael Esteves, Inês Cruz – João Alves Bernardo Marques,
