Introduction to mixed realities

Introduction to mixed realities

Course 02

October 11, 2021 Adrian Iftene

[email protected]

Content

● Recapitulation Course 1

○ AR/VR/MR/XR

● Computer Graphics in AR/VR

○ 3D Modeling

○ Blender

● Working in a game engine

○ Unity Interface

● Conclusions

● Bibliography

R - Definitions

● Augmented Reality

● Virtual Reality

● Mixed Reality

● Extended Reality

3

Virtual Environment Rendering

In VR, rendering does not only correspond to the image synthesis process Must encompass as many senses as possible to induce and immersive experience

Image synthesis

Audio rendering (spatial)

Tactile rendering and force feedback (both localized and full body)

Smell simulation ⁴

3D Modeling

Used for the development of virtual models for computer simulation, XR Application for artificial intelligence (AI), big data analytics, etc.

5

Major Software in 3D Modeling

Tools: ZBrush, Blender, SketchUp, AutoCAD, SolidWorks, 3Ds Max, Maya, Rhino3D, CATIA, etc.

Used file formats of 3D models are FBX, OBJ, STL, etc.

7

Computer-Aided Industrial Design (CAID)

CAID tools provide 3D modeling features

CAID is used in various industries like 3D printing, animation, gaming, architecture, and industrial design for digital production

The CAID tools provide designers with improved freedom of creativity compared to typical CAD

Texture Painting Software

Texture painting is an essential step to enhance the visual effects and increase the realism of virtual environments

9

Key 3D Modeling Terms - 3D Model

The digital representation of a three-dimensional object that is created in dedicated 3D modeling software

A 3D model can be turned and looked at from every different angle and can be scaled, rotated, or freely modified

3D Modeling - NURBS Modeling

Non-Uniform Rational Basis Spline (NURBS) model is a mathematical modeling type commonly used to generate curves and surfaces

The main advantages of this modeling technique are the great flexibility and precision you have in generating your shapes (use interpolation)

In contrast to Polygon Modeling, the curve is drawn in a 3D space, and edited by moving a series of handles called CVs or control vertices along the x-, y-, or z-axis

11

3D Modeling - Polygon Modeling

Polygon models (also known as meshes) are a collection of vertices, edges and faces that define the model, which allows for easy and precise editing of parts of your object

By changing the coordinates of one or several vertices, you can change the shape of the model

Used in:

Animation and Film Games industry

Polygon Modeling - Vertices, Edges and Faces

Faces: are the most basic parts of a 3D polygon. When three or more edges are connected together, the face is what fills in the empty space between the edges and makes up what is visible on the polygon mesh

Edges: are defined by two vertices at their end points

Vertices: are the smallest components of a polygon model. It is simply a point in a three-dimensional space

13

Polygon Modeling - Polygons

Four-sided (quads) or three-sided (tris - used more commonly in game modeling)

The number of polygons in a mesh is called the poly-count, while polygon density is called resolution

Subdivision Surfaces/NURMS Modeling

Subdivision surfaces (NURMS - Non-Uniform Rational Mesh Smooth): it is a method used to smoothen out those pixelated meshes

NURMS actually subdivide each polygonal face into smaller faces that better approximate the smooth surface

15

Polygon Modeling - Textures and Shaders

Shaders: are a set of instructions applied to a 3D model that lets the computer know how it should be displayed. With tools from 3D software packages, we can control the way the surface of the model interacts with light, including opacity, reflectivity, specular

highlight (glossiness), etc.

Textures: are two-dimensional image files that are mapped onto the model's 3D surface (texture

mapping). Textures can range in complexity from

simple flat color textures up to completely photorealistic

surface detail ¹⁶

Polygon Modeling - UV Mapping

UV mapping is the process of projecting a 2D image texture onto a 3D object. Typically, such a texture is applied after a model, or body, is created UV mapping can be used while creating a new material. The material affects the object, while the UV mapping only affects the surface of the material

17

Polygon Modeling - Texture Maps

Specular Map - is a texture type that dictates which parts are glossy/specular based on the grayscale values that it has

Diffuse Map - it can add that bit of realism to your shader

Normal Map - It works more like a workaround

compared to what you would normally achieve when using dedicated normal maps, but the effects are relatively close and indistinguishable

3D Modeling - Rigging

Rigging is a technique used in skeletal animation for representing a 3D character model using a series of interconnected digital bones

This bone structure is used to manipulate the 3D model like a puppet for animation

The result is a hierarchical structure where each bone is in a parent/child relationship with the bones it connects to. This simplifies the animation process as a whole

19

3D Modeling - Animations

These bones can be transformed using digital animation software meaning their position, rotation, and scale can be changed

By recording these aspects of the bones along a timeline (using a process called keyframing)

animations can be recorded

A basic setup may take a few hours or less while a complex rig for a movie could take days

https://www.youtube.com/watch?v=_C2ClFO3FAY

Blender https://www.blender.org

Blender is the free and open source 3D creation suite

It supports the entirety of the 3D pipeline—modeling, rigging, animation,

simulation, rendering, compositing and motion tracking, video editing and 2D animation pipeline

21

23

25

Blender - Edit Mode

Annotate Measure

Extrude Region Inset Faces Bevel

Loop Cut Knife

Poly Build Spin

Smooth Edge Slide Shrink/Fatten Shear

Rip Region ²⁷

Blender - Sculpt Mode

Blender - Grabbing, Scaling, and Rotating

The three most basic ways of changing an object in a 3D scene are called transformations:

✓ Change location using location

✓ Change size using scale

✓ Change rotation using rotation

29

Blender - Differentiating Between Coordinate Systems

All coordinate systems in Blender are based on a grid consisting of three axes:

✓ The X‐axis typically represents side‐to‐side movement.

✓ The Y‐axis represents front‐to‐back

✓ The Z‐axis goes from top to bottom

Blender - Transform Orientations

✓ Global: the primary orientation to which everything else relates

✓ Local: each 3D object in Blender has a local coordinate system

✓ Normal: a set of axes that’s perpendicular to some arbitrary plane

✓ Gimbal: When you rotate an object about its X, Y, and Z axes

✓ View: relative to how you’re looking at the 3D View

31

Blender - Move, Rotate, Scale

Blender - Hotkeys

33

Blender - Hotkeys on the Numerical Keypad

Modeling and Computer Graphics in VR

Create 3D models in VR for real-time interaction - one approach is to

perform optimization to reduce the complexity by minimizing the mesh size of the models => visual realism of the models may be affected

Computer graphics techniques used to create 3D models in VR:

Mesh Shading

Flat-shading (left)

Smooth-shading (right)

35

Techniques used to create 3D models in VR

Auto smooth shading filter to quickly and easily changes the way the shading

Mesh editing tools such as bevel, subdivision, loop cut, etc. may need to be applied at the edge to create better

visual effects. Below, 3D models applied the bevel modifiers with 20 segments, 2 segments and 6 loop cuts (from left to right)

Blender - Render Engine Eevee

The corresponding models are rendered using the real-time render engine Eevee in Blender

Visual effects of the models represented by red, green and blue color (from left to right)

37

39

Free 3D Models

Poly https://poly.google.com

Turbosquid https://www.turbosquid.com

Unity Asset Store https://assetstore.unity.com/

CGTrader https://www.cgtrader.com/free-3d-models

Unity Interface

A - Toolbar

B - The Hierarchy Window C - The Game View

D - The Scene View

E - The Inspector Window F - The Project Window

41

User Interface - Toolbar

On the left it contains the basic tools for manipulating the Scene view and the GameObjects within it

In the centre are the play, pause and step controls

In the right you have access to Unity Collaborate, Unity Cloud Services and your Unity Account

Finally there is a layer visibility menu and the Editor layout menu

User Interface - Hierarchy window

It is a hierarchical text representation of every GameObject in the Scene.

Each item in the Scene has an entry in the hierarchy, so the two windows are inherently linked. The hierarchy reveals the structure of how GameObjects attach to each other

43

User Interface - Game view

Simulates what your final rendered game will look like through your Scene Cameras. When you click the Play button, the simulation begins.

User Interface - Scene view

Allows you to visually navigate and edit your Scene. The Scene view can show a 3D or 2D perspective, depending on the type of Project you are working on

45

User Interface - Inspector Window

Allows you to view and edit all the properties of the currently selected GameObject. Because different types of GameObjects have different sets of properties, the layout and contents of the Inspector window change each time you select a different GameObject

User Interface - Project window

Displays your library of Assets that are available to use in your Project.

When you import Assets into your Project, they appear here

47

Assets in Unity

An asset is a representation of any item you can use in your project An asset might come from a file created outside of Unity, such as a 3D Model, an audio file, or an image

You can create some asset types in Unity, such as a ProBuilder Mesh, an Animator Controller, an Audio Mixer, or a Render Texture

How Unity Imports Assets

Unity automatically imports assets and manages additional data about them:

1. Unity assigns the asset a unique ID 2. Unity creates a .meta file

3. Unity processes the asset

Unity reads and processes any files that you add to the Assets folder, converting the contents of the file to internal game-ready data

Sometimes Unity creates multiple assets when it imports a single complex asset file

49

Common Types of Assets

Image files - Unity supports BMP, TIF, TGA, JPG, and PSD FBX and Model files - import data from any 3D modeling software that supports FBX

Meshes and animations - Unity imports the Meshes and animations from each file

Audio files

Other assets: 2D, Cameras, Characters,

CrossPlatformInput, Effects, Environment, ParticleSystems,

Prototyping, Utility, Vehicles 50

Conclusions

VR/AR/MR/XR need 3D models for their assets

You can create your 3D model in Blender, Maya, etc. and you can import then in Unity

There are many available 3D models on the Internet

51

Questions

What are key elements in 3D Modeling?

Unity: give examples of assets.

Bibliography

● Tang, Y. M., Ho, H.L. (2020) 3D Modeling and Computer Graphics in Virtual Reality. Open access peer-reviewed chapter - ONLINE FIRST DOI: 10.5772/intechopen.91443

● Watkins, A. (2011) Creating Games with Unity and Maya. Focal Press. Elsevier.

http://index-of.co.uk/Game-Development/Elsevier%20Creating%20Games%20with%20Unity%20and%20Maya.pdf

● Gumster, v. J. (2015) Blender for Dummies, 3rd Edition

http://2.droppdf.com/files/5IzWw/blender-for-dummies-3rd-edition.pdf

● Blackman, S. (2013) Beginning 3D Game Development with Unity 4. Springer

● https://profs.info.uaic.ro/~avitcu/FII%202015-2016/Animatie%203D_Documentatie/

● Keller, E. (2010) Mastering Autodesk Maya 2011. Wiley Publishing, Inc.

● Watkins, A. (2012) Getting Started in 3D with Maya. Elsevier, Inc.

● Cabrera, C. (2008) An Essential Introduction to Maya Character Rigging. Elsevier.

● Derakhshani, D. (2010) Introducing Maya 2011. Wiley Publishing, Inc.

● Palamar, T. (2014) Mastering Autodesk Maya 2015. Wiley Publishing, Inc.

● Murdock, K. (2004) Maya 6 Revealed. Thomson Course Technology.

● Lenier, L. (2007) Maya. Professional tips and techniques. Autodesk Maya Press.

● Mechtley, A., Trowbridge, R. (2011) Maya Python for Games and Film: A Complete Reference for Maya Python and the Maya Python API. CRC Press; 1st Edition.

● Sherman, W.R., Craig, A. B. (2003) Understanding Virtual Reality. Elsevier, Science. ⁵³

Links

● Blender 2.90 Reference Manual: https://docs.blender.org/manual/en/latest/

● Blender 3D Modeling Tutorials For Beginners: The Ultimate Collection https://conceptartempire.com/blender-modeling-tutorials/

● Blender Tutorial: Creating a Chess Piece for 3D Printing

https://i.materialise.com/blog/en/tutorial-creating-a-simple-chess-piece-in-blender/

● Key 3D Modeling Terms Beginners Need to Know https://i.materialise.com/blog/en/3d-modeling-terms/

● 3D Model Components—Vertices, Edges, Polygons & More https://www.lifewire.com/3d-model-components-1952

● Part 3, Level 2: Texture Painting - Blender Beginner Tutorial https://www.youtube.com/watch?v=nht2RoYBUfA

● Shading Editor - Blender 2.80 Fundamentals https://www.youtube.com/watch?v=0zrd37k2tJM&app=desktop

● Blender: UV Mapping – Simply Explained https://all3dp.com/2/blender-uv-mapping-simply-explained/

● How to generate texture maps from a single image

http://www.reynantemartinez.com/how-to-generate-texture-maps-from-a-single-image.html

● What is 3D Rigging For Animation & Character Design? https://conceptartempire.com/what-is-rigging/

● Character Animation: How to Animate a Backflip in Blender

https://cgi.tutsplus.com/tutorials/character-animation-how-to-animate-a-backflip-in-blender--cms-26511

● Free textures: https://www.texturehaven.com/