Analysis of 3D User Interfaces: A Comprehensive Report

This report, authored by Ahmed Ahmed, provides a comprehensive overview of 3D User Interfaces (3D UIs). It begins by introducing the concept, highlighting the shift from 2D to 3D interactions with advancements in VR and AR technologies. The report delves into background research, defining 3D interaction and classifying different contexts such as augmented reality, virtual reality, and large-screen displays. It emphasizes the multidisciplinary nature of 3D UI development, touching upon cognitive psychology and information visualization. The report then discusses the challenges in usability, proposing a design framework based on specificity, complex interactions, and the integration of emerging technologies. The framework suggests tailoring designs to specific applications, domains, tasks, devices, and users, advocating for more complex and natural interactions. A 3D mock-up of a virtual conference environment is presented as an example, illustrating the practical application of these design principles. The report concludes by emphasizing the importance of user experience (UX) and the need for ongoing research and development to overcome the usability issues that currently plague 3D UIs.

3d User Interfaces
Report
Ahmed

Abstract
The three dimensional user interface or 3D UIs allows a person to interact with different
environments, virtual objects or information using a direct 3D input in virtual or physical
space. With the advent of modern generation devices such as Nintendo Wii or Holographic
virtual reality interfaces such as Microsoft HoloLens, the 3D UIs are now longer a work of
fiction, rather they have started to become mainstream. With this kind of intervention, the
biggest problem that 3D UI researchers and interface designers face is how to tackle the
usability aspect of 3D interface as the user interfaces have predominantly been made in 2D.
As a result, usability aspect is still surprisingly low when it comes to 3D UI based
applications.
1

Table of Contents
Introduction................................................................................................................................3
Background Research.................................................................................................................3
Proposed Design.........................................................................................................................5
Benefits and Risks......................................................................................................................7
Conclusion..................................................................................................................................8
References..................................................................................................................................9
2

Introduction
With the advent of virtual reality based environments combined together with technologies
such as augmented reality, three dimensional applications and large-screen format display
screens, a new trend has emerged in the Human Computer Interaction. Although, there are
decades’ worth of experience in applying design principles for traditional UIs, they are still
not sufficient in 3D spatial context that involves multiple degrees of freedom. The application
developers, designers and researchers quickly realized that the 3D domain had some
fundamental differences between them and that newer methods would be needed to develop
applications for the 3D User Interface. This particular area is known as 3D User Interfaces,
3DUI or 3D interaction.
A typical 3D based interface involves special spatial based input devices such as 3D pointing
devices, gestural input, head-mounted displays or HMDs, stereoscopic projects, haptic
feedback devices and even spatial audio systems. However, with these new technologies,
people find it difficult to understand and perform action in 3D spaces[2]. Although, we live in
a 3D world, it’s an inherently very different than a 3D world as for performing actions in the
real world there are lot of hints, cue and feedbacks available for which the same cannot be
represented in a computer simulation. As a result, simply adapting a traditional WIMP or
Windows Icon Menu Pointers based interface into a 3D will not be an effective solution.
Instead, a new design process has to be established that allows successful transformation of
the traditional interface into a 3D based interface.
Background Research
3D interaction is defined as “Human Computer Interaction wherein the tasks performed by
the user is directly in a D spatial context” . However, an interactive system that would display
a 3D graphic does not necessarily means a 3D interaction. For instance, a user viewing a 3D
model of a building on their computer does not mean a 3d interaction has taken place. Now,
these definition may not necessarily classify all of the 3D interaction that takes place,
however, we could possibly classify them based on the context wherein the 3D interaction
took place [20].
These include:
3

 Augmented Reality: A physical object or a virtual object can be overlaid on a real
environment allowing the objects to be moved, modified and placed in the physical
world.
 Virtual Reality: Head Mounted Displays are the best examples of virtual reality which
depicts a user could move through in a virtual environment by pointing 3D directions
in the real world [19].
 Large-screen format displays: A motion based controller that allows to control objects
and information on large screens giving users the ability to pan, zoom, shuffle and
move around the screen and its contents.
 Pervasive and ubiquitous computing: These type of interfaces includes interacting
between devices using gestures and actions performed in a 3D spatial context.
The 3D User Interface is an interesting area of user experience development since it’s multi-
disciplinary and is impacted by many other interesting research areas such as cognitive
psychology, perception, information visualization, educational training and entertainment [7].
Also, unlike the physical environment which is limited in organizing and moving
information, digital environments have additional capabilities that allow them to have
digitized annotations, easier viewing and moving of data along with simple and advanced
text search, different levels of access for users as well as exporting [18] and importing of
multiple digital forms. Another advantage over physical environment is that theoretically the
amount of information that can be stored is limitless. Shared VR based systems also support a
greater degree of collaboration among populations who are physically spread out or
geographically dispersed. An example of such system would be VR VIBE that allows
collaborators to view the web in a same shared space while also making it possible to view
what others are watching [3]. Although, the user browsing in such a 3D based virtual
environment may face fatigue and disorientation.
When developers are creating a virtual environment, it is crucial that they pay attention to the
experience of users using the system. This phenomenon is known as User Experience or UX
and has several different interpretations. For instance, [4] suggest that a user experience
typically includes hedonic and usability aspect of the system thereby making it more effective
and experiential. According to them, user experience is the resulting interaction between the
user and the product wherein the experience in using the product includes physical, sensual,
emotional and cognitive aspects. Also, user experience is traditional human computer
interaction with a design that is aesthetically pleasing.
4

If all of the major 3D interaction techniques has been discovered, then one might expect it to
find a corresponding boom in real-world usable applications that involves 3D interaction.
Unfortunately, in reality, this is not the case. Most of the demonstrations that involve 3D
application prototypes are that it still involves a high-degree of usability problems. This lack
of boom can be interpreted in different ways. First of all, it means that many fundamental 3D
interaction techniques are yet to be invented Secondly, the developers and researchers may
have ignored the research results surrounding 3D User Interfaces. Or in other words it means
that the knowledge could be available to produce real-world functional 3D UIs, but they
aren’t been used for its full potential. Third reason is that it could be possible that developer
of a 3D User Interface may not see any commercial benefit beyond entertainment and games.
Although, this third problem does have some truth to itself, it still doesn’t explain the reason
why even prototypes have bad usability when it comes to 3D User Interfaces. The fourth
possible interpretation could be that, all possible 3D UI interactions has been discovered, but
this knowledge alone is not sufficient to ensure that the 3D UIs have high usability standards.
Keeping this in mind, it means that there needs to be a body of knowledge or a framework
developed specifically that turns out to be a design principles when it comes to the designing
of Three Dimensional User Interfaces.
Proposed Design
This section will help address the issue of the usability aspect of 3D UIs so as to formulate a
framework that would help in designing of future 3D User Interfaces. The main issue this
framework intends to solve is to provide a high levels of usability and interactivity of 3D UIs.
The proposed framework is as follows:
Specificity
Typically in user interface, generality is considered to be a desirable trait. This means that if
the design or interaction is more general, it can be applied to many different applications and
situations. However, in terms of 3D UI does not apply very well because it 3D applications
vary significantly and the same design pattern or technique cannot be applied over and over.
This is enumerated as follows:
 Application and Domain generality: The interaction technique wasn’t designed with
any application or domain in mind and is meant to work with all kinds of applications
[8].
5

 Task Generality: The interaction technique was meant to work with all kinds of tasks
and it isn’t designed for a specific type of task [9].
 Device Generality: Many designs are made for 3D user interaction without having
considered the characteristics of the devices they would be used on. For instance a 3D
User Interface design may have been designed to better work on a 6-Degree of
Freedom devices but they are being used on HMDs with a wand-like input device.
 User Generality: The type of design did not effectively consider the target group of
users and rather considered a ‘typical’ user group.
In order to resolve this designers, developers and researchers should introduce specificity in
their designs. The following types of specificity should be included:
 Application specificity: An obvious way to increase the usability of the application is
to start the design process from scratch using usability engineering process,
specifically for this application [6]. However, the downside with this type of design
process is that there is no guarantee that any interaction techniques produced by this
type of method could be potentially reused in any other application.
 Domain specificity: In order to accomplish this the designers and developers should
gain knowledge of the application domain and develop usability principles for that
particular application domain. The advantage to this type of design is that the design
is usable in other applications [13].
 Task specificity: The interaction techniques can also be made more usable if they are
being designed for specific task requirements rather a general task category. For
instance, in a virtual 3D travel exploration application, the interaction techniques
should be unique and specific to each of tasks such as exploration, navigation,
searching and so on [17].
 Device specificity: The devices being used for input and output for a 3D application
may all require unique interaction methods for them. For instance, an application
design for a single-button 6-DOF may exhibit critical usability issues when the same
application is accessed by a multi-button 6-DOF device. Simultaneously, a 3D UI
designed for a large-screen display format may again exhibit serious design problems
when instead an HMD is being used.
 User Specificity: The 3D UIs should be user specific. Several factors such as age,
demographics, experience level and gender should be used while designing for the
users [9].
6

Complex interactions
Designers, developers and researchers try to make their 3D interaction extremely simple
by designing the application to perform one thing extremely well. While, this method of
design works well in traditional HCI designs, it does not do so in 3D UI context. Instead,
the designers should try and include as many complex and real-world interactions as
possible so that there is much more fluidity at the end of the users resulting in more
intuitive and natural interaction.
3D UIs in emerging technologies
As technologies progress, so should the user interactions for them. For instance, in lot of
working environments, multi-monitor setup is becoming increasingly common and user
interactions designed for a single monitor setup should be adapted to the multi-monitor setup.
Researchers and developers should also come up with novel methods of interaction that
simulate more of the interactions in the real-world and even beyond. Interactions such as
walking, gazing, gestures etc. should be incorporated for a more natural and realistic
interaction [11].
3D Diagram representing the guidelines mentioned
above
Based on the design principles outlined earlier the below mentioned 3D Mock-up has been
prepared. The application represented below is a representation of a futuristic 3D based
virtual conference. The participants may be located in the same facility or they might be
geographically dispersed, however, they could still join together in a single virtual conference
room facility. This would be made possible with the help of either AR or VR with a Head
Mounted Display. The participants would appear in the same room as they are physically
present. This virtual conference provides additional advantages over the traditional method
by allowing participants to project applications, documents, presentation files, reports, charts
and other data on to the large projector screen behind them. Users can navigate in this virtual
conference room is done with the help of a Wand like device that supports 6-axis motion
sensing capabilities so that the system can track their movements in full 3D spatial context.
The selection and clicking is done with the help of a single button present on the wand
device. The below diagram portrays the usage of the wand based device on the six-axis : +X,
-X, -Y, +Y, +Z and –Z.
7

The below diagram uses a unique method of navigation based on the principles outlined
earlier and conforms to each of the principles namely task specificity, application specificity,
complex interactions among others.
Benefits and Risks
Compared to a more traditional 2d based interfaces, some of the benefits and risks for a 3D
based conferencing facility are as follows:
8

 3D based simulated conference will help enhance the participants’ comprehension
capabilities including enhanced information observation, perception, enhanced
awareness and even possible increase in memory and learning [15].
 3D UI goes a step beyond in simple information visualization. This is because 3D UI
has a virtually infinite real-estate when it comes to information visualization and is
able to show complex information easily when compared to a traditional approach. It
goes even a step beyond by also showing the relationships between different
information.
 Information such as charts, graphics, diagrams etc. appear more natural and easy to
comprehend in the virtualized conference room. This is because graphics and
representations help a user in grasping the information easily.
 Since a 3D based UI may feel more intuitive and responsive, it would also mean that
when a user fully learns the interface and becomes an advanced user, then he or she
may just be able to perform several actions at a faster pace compared to traditional
conference systems [16].
 3D based user interaction shows a great deal of potential in interacting with the
information. The information being presented with these new methods appear more
natural and rich compared to the traditional HCI based UIs. The information
presented with 3D UIs are also more interactive and could possibly support direct
manipulation.
 However, with these great advantages and novel methods of interacting with the
information, comes great shortcomings too. The biggest problem with 3D based
interaction is the complex map scenario that gets rendered in front of the user viewing
the conferencing facility which can cause disorientation, exhaustion and feeling of
getting lost within the system [12].
 Another major issue with 3D interaction is that users would need to re-learn the entire
mechanism of how they typically used to interact with the previous VOIP based
conference as they have been trained and practiced for over decades on traditional
HCI systems.
 There is also the problem of limited hardware capabilities that restricts the full
potential of 3D systems as running a full-fledged 3D interaction system with multiple
interaction points and 3D simulation requires intense hardware resources. This means
that every system that a user would use to access the conference would. However, this
problem is only temporary as hardware prowess is ever-increasing.
9

 Finally, 3D based conference does not help resolve problems for people with
disabilities, and if any, it only enhances the problem and at present there are no on-
going research as to how people with disabilities will be making use of 3D
conferencing system [14].
Conclusion
In summary, the report above tries to present that there are various unresolved issues towards
the pathway of a world dominated by 3D User Interface. It has already begun happening in
the form of Microsoft HoloLens, Nintendo Wii, Apple AR and countless other AR and VR
based applications and it will only grow further from here. However, it is important to
understand that traditional design principles for user interface design does not apply here and
designing for 3D based system require a newer understanding of the system and many a times
would require building it from the scratch. This will enhance the chances for right skill
transfer thereby increasing the usability of various interaction techniques to be used in 3D.
This would in turn lead to a better development of 3D applications and a broader adoption of
3D UIs.
However, there would still be many questions unanswered such as: Will a standard 3D
interface would ever be practical? How much interaction would differ between different
devices? How does one determine what 3D interaction works best for any given application?
How much benefit does a 3D User Interface has over a more traditional system? Perhaps,
some of these questions may be answered when 3D UI becomes more mainstream.
References
[1]M. Campbell, "Conference preview: UIST 2001: the 14th annual ACM symposium on
user interface software and technology", interactions, vol. 8, no. 6, pp. 49-50, 2001.
[2]K. Herndon, A. van Dam and M. Gleicher, "The challenges of 3D interaction", ACM
SIGCHI Bulletin, vol. 26, no. 4, pp. 36-43, 1994.
[3]S. Benford, C. Greenhalgh, G. Giannachi, B. Walker, J. Marshall and T. Rodden,
"Uncomfortable user experience", Communications of the ACM, vol. 56, no. 9, p. 66,
2013.
[4]J. Forlizzi and A. Battarbee, "Confessions of a human-centered designer", interactions,
vol. 20, no. 2, p. 10, 2013.
10

[5]N. Tractinsky and M. Hassenzahl, "Arguing for Aesthetics in Human-Computer
Interaction", i-com, vol. 4, no. 32005, 2005.
[6]J. Gabbard, D. Hix and J. Swan, "User-centered design and evaluation of virtual
environments", IEEE Computer Graphics and Applications, vol. 19, no. 6, pp. 51-59,
1999.
[7]F. Chairman-Gey, M. Chairman-Hearst and R. Chairman-Tong, "Proceedings of the 22nd
annual international ACM SIGIR conference on Research and development in
information retrieval", Dl.acm.org, 2018. [Online]. Available:
https://dl.acm.org/citation.cfm?id=312624. [Accessed: 08- May- 2018].
[8]D. Yonca, J. Chen, C. Wingrave, J. Lucas, A. Ray and N. Polys, "New Directions in 3D
User Interfaces", The International Journal of Virtual Reality, vol. 5, no. 2, pp. 3-14,
2006.
[9]I. Poupyrev, S. Weghorst, M. Billinghurst and T. Ichikawa, A Framework and Testbed for
Studying Manipulation Techniques for Immersive VR. 2015.
[10]R. Tintner, C. Wingrave1, B. Walker and D. Bowman, Exploring Individual Differences
in Raybased Selection: Strategies and Traits. 2014.
[11]F. Guimbretière, M. Stone and T. Winograd, "Fluid interaction with high-resolution wall-
size displays", 2018. .
[12]H. Wenjun, The Three-Dimensional User Interface. 2018.
[13]J. Chen and D. Bowman, Effectiveness of Cloning Techniques for Architectural Virtual
Environments. 2016.
[14]F. Rebelo and M. Soares, Advances in usability evaluation. Boca Raton, FL: CRC
Press/Taylor & Francis Group, 2013.
[15]N. Liberati, "Augmented reality and ubiquitous computing: the hidden potentialities of
augmented reality", AI & SOCIETY, vol. 31, no. 1, pp. 17-28, 2014.
[16]X. Wang, "Augmented Reality: A new way of augmented learning", eLearn, vol. 2012,
no. 10, p. 1, 2012.
[17]X. Wang, "Augmented Reality: A new way of augmented learning", eLearn, vol. 2012,
no. 10, p. 1, 2012.
11

[18]H. Ling, "Augmented Reality in Reality", IEEE MultiMedia, vol. 24, no. 3, pp. 10-15,
2017.
[19]F. Liarokapis, "An augmented reality interface for visualizing and interacting with virtual
content", Virtual Reality, vol. 11, no. 1, pp. 23-43, 2006.
[20]M. Billinghurst and D. Schmalstieg, "Editorial: special issue on augmented
reality", Virtual Reality, vol. 15, no. 2-3, pp. 97-98, 2011.
12