Examples of Relationships
In Fig. 6.3, a fragment of an inheritance hierarchy build for the chemistry domain is presented. Virtual, Real, and Scene AR-Classes are denoted with rectangles, rounded rectangles, and hexagons, respectively. Abstract classes are denoted with their names in italic, as opposed to concrete classes, whose names are in normal text.
Three kinds of relationships between AR-Classes are presented in Fig. 6.3. Specialization is denoted by an open arrowhead that points to a super class. Composition is indicated by a solid diamond on the end of the relationship line that points to a composite AR-Class. Similarly, containment is indicated by an empty diamond pointing to a containing AR-Class. For the composition and containment relationships, a solid line denotes definition of a new relationship for an AR-Class, whereas a dashed-dotted line denotes concretization of a relationship inherited from a super class.
Fig. 6.3 Examples of the relationships between AR-Classes
The Real Object class has one subclass Manipulator, which in turn has two subclasses Round Paddle and Square Marker. There are three subclasses of the Virtual Object class—two simple classes Container and Liquid, and one composite class Container with liquid. The latter class is composed of the two others. The Container has two subclasses: Beaker and Tube. The composite class Container with liquid has two subclasses Beaker with liquid and Tube with liquid. As the names of these subclasses point out, they are specializations of their super class. The Beaker with liquid class remains in the composition relationships with Beaker and Liquid. The relationship with Liquid is inherited from the super class, whereas the relationship with Beaker is specialization of the relationship with Container. The Tube with liquid class is defined in the same way.
Geometry of any Virtual AR-Class can be included in geometry of any AR-Class belonging to any category, i.e., Virtual, Real, or Scene. For example, the Beaker class has geometry, whereas Liquid does not. The geometry of Beaker with liquid embeds geometry of Beaker. Also, Beaker with liquid specifies geometry of the liquid in the context of the Beaker geometry. Analogously, the Tube with liquid embeds geometry of Tube and specifies geometry of the liquid according to the Tube shape. Thus, the liquid can be represented in 3D in different ways depending on the shape of a container.
The Scene class has one subclass Pouring from tube to beaker with manipulator. The scene class contains the Tube with liquid and Beaker with liquid Virtual AR-Classes. The scene class also contains one abstract Real AR-Class Manipulator which represents the class of Real AR-Objects that can be used for manipulating the Virtual AR-Objects contained in the scene. The scene class has one specialization that concretizes the containment relationship from the Manipulator to the Round Paddle class. Each contained AR-Class in the Pouring from tube to beaker with paddle has geometry that is not related to the geometry of the Scene AR-Class. The behavior of the Scene AR-Class is able to manipulate the geometries of contained AR-Classes. In the presented example, the behavior of the Scene AR-Class may specify transformations of the geometries of Tube with liquid and Beaker with liquid objects in the context of the Scene AR-Class geometry. On the contrary, as Round Paddle is a Real AR-Class, its geometry cannot be transformed by the Scene AR-Class behavior. The geometry of Real AR-Classes is transformed automatically in their AR-Objects by the use of real objects present in an ARE.
Persistent and Volatile AR-Objects
The AR-Objects can be persistent or volatile. Persistent AR-Objects are created in the content preparation phase and stored in the ARCR repository. In the AR content use phase, Persistent AR-Objects can be selected for presentation in an augmented reality environment. To this end, volatile copies of the Persistent AR-Objects, called Volatile AR-Objects, are created by the AREPS system. A set of Volatile AR-Objects constitute an augmented reality scene which is presented by AREPS in a real environment. Users can interact with the Volatile AR-Objects using the real objects corresponding to the Volatile Real AR-Objects contained in the presented augmented reality scene.
A Volatile AR-Object combines the attribute values from the corresponding Persistent AR-Object with the attributes, behavior, geometry, and media objects defined in the AR-Class. If the Persistent AR-Object is composed of other Persistent AR-Objects, then an appropriate Volatile AR-Object is created for each component Persistent AR-Object. Both Persistent and Volatile AR-Objects are instances of AR-Classes, but there are the following differences:
• Persistent AR-Objects are instances of AR-Classes in the object-oriented paradigm meaning, i.e., they only contain attribute values. Contrarily, Volatile AR-Objects are volatile copies of the Persistent AR-Objects combined with the contents of their AR-Classes. A number of Volatile AR-Objects of one Persistent AR-Object can exist at a time.
• Any changes made to Persistent AR-Objects are stored in the ARCR repository. Any changes made to Volatile AR-Objects do not influence the corresponding Persistent AR-Objects stored in the repository.
• Persistent AR-Objects are created in the content preparation phase, whereas Volatile AR-Objects are created dynamically in the content use phase.
• Persistent AR-Objects can be modified during the content preparation phase by a content designer. Volatile AR-Objects can be modified during the content use phase as a result of their behavior or user interaction.
In Fig. 6.4, the main components used in the AR content use phase are presented. The ARCR repository is used to store AR-Classes and Persistent AR-Objects. The AREPS component is responsible for presentation of augmented reality scenes comprised of Volatile AR-Objects being copies of Persistent AR-Objects selected from the repository. Persistent AR-Objects have a solid border, whereas Volatile AR-Objects have a dashed border. The relationships between AR-Classes, Persistent AR-Objects, and Volatile AR-Objects are presented.
Fig. 6.4 Persistent vs. Volatile AR-Objects
In the AR content use phase, an AR-Scene is selected from the AR Content Repository and loaded into the AREPS system. After the AR-Scene is loaded, the AREPS system builds an augmented reality environment by creating Volatile AR-Objects for the main AR-Scene and the AR-Objects comprising the content of the augmented reality scene. An augmented reality scene is built by the AREPS system to enable presentation of AR-Objects in the context of a real environment. As a result, an augmented reality environment is constructed and presented to a user.
The AR-Scene loaded into AREPS has at least one activity defined. The activity describes the way the AR-Scene and its component and contained AR-Objects develop over time depending on user interaction and behavior of the AR-Objects. In particular, the AR-Scene behavior can activate and deactivate activities of the AR-Objects. After the whole content of the augmented reality scene is loaded, the AREPS system starts execution of the activity of the main AR-Scene. The structure of an augmented reality scene is represented in AREPS by an augmented reality scene graph. An example of a simple augmented reality scene graph within the AREPS component is presented in Fig. 6.4.
Augmented Reality Scene Graph
The augmented reality scene graph is a directed tree whose nodes are Volatile AR-Objects, whereas the edges represent a parent-child relationship, called attachment relationship. A tree node can have an arbitrary number of children but only a single parent. Nodes having the same parent node are called sibling nodes. There is exactly one root node in the tree that does not have a parent and represents the main AR-Scene object. Nodes in a tree can be divided into interior and leaf nodes. While an augmented reality scene is constructed by AREPS, the attachment relationships between Volatile AR-Objects reflect the containment relationships defined between the corresponding Persistent AR-Objects. During presentation of the augmented reality scene, the attachment relationships can be modified, i.e., the parents of AR-Objects can be changed.
Each AR-Object has its own local coordinate system. The coordinate system of a main AR-Scene is called a main coordinate system. A local coordinate system of an AR-Object is a frame of reference for child AR-Objects, i.e., position and orientation of the children are specified in the context of a parent coordinate system. The child AR-Objects can be transformed (translated, rotated, scaled) relative to their parent coordinate system. Real AR-Objects can be transformed by users by manipulating corresponding real objects in an augmented reality environment. The position and orientation of Real AR-Objects cannot be controlled by the AREPS system. On the contrary, Virtual and Scene AR-Objects can be transformed by AREPS as a result of execution of their behavior.
Two kinds of an attachment relationship can be distinguished: strong and weak. When two AR-Objects are connected by a strong attachment relationship, then transformations applied to a parent node automatically propagate to the attached child nodes. When two AR-Objects are connected by a weak attachment relationship then transformations of a parent node are not applied to the attached child nodes. The strong attachment relationship appears for Virtual and SceneAR-Objects attached to an AR-Object. The weak attachment relationship appears for Real AR-Objects attached to an AR-Object, because their position and orientation cannot be changed by the AREPS system, which is not capable of moving or rotating real objects.
During the presentation of an augmented reality scene, users can manipulate a Real AR-Object and change its position and orientation relative to the parent AR-Scene coordinate system. When the manipulated Real AR-Object is attached with any child Virtual or Scene AR-Objects, they are automatically transformed according to the changes applied to the parent Real AR-Object. Similarly, the changes of position and orientation of a Scene or Virtual AR-Object made by the AREPS system automatically propagate to the transformation of the attached child Virtual or Scene AR-Objects.
In Fig. 6.5, example modifications of an augmented reality scene graph are presented. There are two states of the graph presented: before (Fig. 6.5a) and after modifications (Fig. 6.5b). The graph nodes are depicted by different geometric shapes, depending on the type of AR-Objects they represent. Hexagons, squares, and ellipses represent Virtual, Real, and Scene AR-Objects, respectively. The attachment relationships between the AR-Objects are depicted by arrows. Solid and dashed lines represent strong and weak attachment relationships, respectively.
In Fig. 6.5a, an initial state of an augmented reality scene graph built basing on an AR-Scene is presented. The root AR-Scene has 6 children: three Real AR-Objects, two Virtual AR-Objects, and one embedded AR-Scene. The latter AR-Scene has 3 children: one Real AR-Object and two Virtual AR-Objects. In the initial state of the graph, all leaf nodes are Virtual or Real AR-Objects. The only interior node is the AR-Scene. The leaves represent the AR-Objects contained in the main or the embedded AR-Scene.
Fig. 6.5 Example modifications of an augmented reality scene graph
In 6.5b, a modified state of the scene graph is presented. There are three modifications of the attachment relationships comparing to the initial state:
1. the VO1 node (a Virtual AR-Object) has been detached from the parent SO1 node (the main AR-Scene) and attached to the RO1 sibling node (a Real AR-Object),
2. the SO2 node has been detached from the parent SO1 node and attached to the RO3 sibling node (a Real AR-Object),
3. the VO3 node (a Virtual AR-Object) has been detached from the parent SO2 node (the embedded AR-Scene) and attached to the RO4 sibling node (a Real AR-Object).
