The AREM approach is based on the Augmented Reality Scene Model (ARSM), in which the concepts of AR-Class and AR-Object are defined. In the content preparation phase, AR-Classes and Persistent AR-Objects are created and stored in the AR Content Repository denoted by R. The R repository which is defined as a pair {CR, OR), where:
CR is the set of AR-Classes defined in the repository, OR is the set of Persistent AR-Objects defined in the repository.
Overview of ARSM Model
Attributes
An AR-Class defines the set of attributes for AR-Objects being its instances and defines the types of the values that can be assigned to each attribute. AR-Objects instantiated from an AR-Class may specify a value for each attribute defined in their AR-Class. Each definition of an attribute in an AR-Class contains a default value for this attribute. The default value specified for an attribute in an AR-Class is assigned to this attribute in an AR-Object of the AR-Class unless another value is specified.
Attributes can be defined as constant or variable. Constant attributes contain constant values specific to each AR-Object of an AR-Class, e.g., the diameter of a beaker, or the boiling temperature of a liquid. The values of constant attributes cannot be changed at runtime in the AR content use phase. On the contrary, variable attributes contain variable values specific to each AR-Object of an AR-Class, e.g., the quantity of a liquid in a baker, or the current temperature of a liquid. The values set in Persistent AR-Objects in the AR content preparation phase are just initial values that may be modified in the Volatile AR-Objects at runtime in the AR content use phase.
Attribute Types The ARSM Model
Each attribute is assigned with an appropriate type. The type of an attribute represents the domain of values that are valid for the attribute. For example, an attribute which is of type Integer will require that its value be an integer. Other values are considered invalid.
Attribute types can be either atomic types or composite types. An attribute of an atomic type can store a single atomic value. Composite types are constructed from a number of simple types. A value of a composite type is composed of several sub-values. Atomic types include:
• simple types, i.e., Integer, Float, String, Boolean,
• media object types, i.e., Image, Movie, Sound, and
• class types denoted by identifiers of AR-Classes.
The Integer and Float types enable storing integer and floating-point values, respectively. The String type enables to store string data. The Boolean type enables to store logical values: true or false. The media object types allow storing references to media objects of specific media types, i.e., images, movie clips, and sounds, respectively. The values of a class type are names of the AR-Objects instantiated from the AR-Class specified as the type.
Two kinds of attributes are distinguished: property attributes and relationship attributes. Property attributes describe the structure of an AR-Class for storing properties of the AR-Objects instantiated from that class. Property attributes are of simple types, media types, or composite types based on simple and media types. All AR-Classes share the set of attributes specifying properties common to all AR-Objects. These properties are: position, orientation, scale, dimensions, opacity, and visibility. The common attributes are specified within the root Object AR-Class, and they are inherited by all AR-Classes.
Relationship attributes specify the structure for describing relationships of an AR-Class with other AR-Classes. Each relationship attribute is of an atomic class type. An AR-Class containing attributes of class types has component or containment relationships to the AR-Classes specified as the attribute types. The relationships defined in the AR-Class are reflected in the AR-Objects created based on that AR-Class.
Direct and Derived Attributes
The value of an attribute can be set directly or can be calculated from the values of other attributes. The attributes of the first category are called direct attributes, whereas the values of the other category are called derived attributes. The value of a derived attribute is calculated by a derivation function. The value returned by a derivation function must conform to the type of the attribute. Derivation functions can be used instead of direct values only for variable attributes. Each argument of a derivation function is a constant value or a value of another attribute. The attribute which is used as a function argument can be declared in the same AR-Class as the attribute with the calculated value, or in one of the related AR-Classes.
For variable attributes, derivation functions can be used as the default values in AR-Classes. An example of a derived attribute is the quantity of a solution composed of two liquids. By default, the value of the solution quantity attribute is calculated by adding the quantities of both component liquids. In the content use phase, each change of the quantity of a component liquid is followed by immediate recalculation of the solution quantity.
AR-Classes explicitly address both technical and semantic properties of AR-Objects. Both kinds of properties are defined as attributes in AR-Classes. Values of properties of an AR-Object can be derived from other properties of this object. The derivation mechanism permits transformation of the values of high-level semantic properties into the values of low-level technical properties. For instance, the liquid amount property of a container with a liquid object can be automatically mapped to the values of low-level visual properties describing the geometrical representation of the liquid. In particular, these mappings can be different for different container shapes, but the liquid amount property will be common to all of them. Also, the values of properties of an AR-Object can be derived from the properties of other AR-Objects related with this object by either the composition or containment relationship. For instance, the liquid color property in a container with a liquid object can be derived from the color property of the related liquid object. Thus, the liquid color property of the container will be automatically set to the color of the filling liquid.
Geometry
Each AR-Class contains geometry which, in particular, can be empty. Geometry is a 3D digital model describing presentation of AR-Class instances in an augmented reality environment. If an AR-Class in its definition does not specify geometry, it inherits the geometry from its super class. Otherwise, the geometry of the AR-Class overrides the geometry of the super class. In this case, the subclass geometry may explicitly embed the geometry of the super class. The AR-Classes with non-empty geometry are called visual AR-Classes, while the AR-Classes containing empty geometry are called non-visual AR-Classes. The visual and non-visual AR-Classes can be instantiated as visual AR-Objects and non-visual AR-Objects, respectively. Both kinds of AR-Objects can be presented in an ARE, but only visual AR-Objects are visible to users.
Geometry of an AR-Class can be parameterized with the attributes of this AR-Class. Also, geometry of an AR-Class can be indirectly parameterized with the attributes of its related AR-Classes through the derived attributes. Moreover, the geometry may embed the geometry of both the super class and the component AR-Classes. Thus, the geometry can be customized in the AR-Objects by setting the attribute values. Persistent AR-Objects of an AR-Class share the same geometry, but they may have different attribute values. By setting attribute values, it is possible to customize geometry associated with AR-Objects. The attribute values can be further modified in the Volatile AR-Objects during the content use phase. The extent to which the modifications of the geometry are possible depends on the geometry model. The geometry may be encoded in any ASCII-based language enabling description of 3D contents, such as VRML or X3D, extended with parameterization features. A notable example of such language is X-VRML [24], described in Chap. 4.
Parameterization of geometry together with derivation of attributes provides a flexible mechanism for building highly dynamic 3D graphics content. The dynamism of the content visualization can be fully controlled by the behavior of the objects comprising the content. In particular, the visualization of the objects can be dynamically changed as a result of user actions performed in the environment. For instance, when a user pours different liquids between different containers, the visualization of the liquids filling the containers will be dynamically adjusted according to the properties of the liquids. The attributes in an AR-Class can represent both properties and relationships. Consider the AR-Class Beaker that has properties specifying the diameter and height of geometry. Different Persistent AR-Objects created based on the Beaker AR-Class may represent beakers of different diameters and heights. However, it can be forbidden to change these parameters in the Volatile AR-Objects because they may play the role of digital surrogates of real beakers whose dimensions are fixed. Consider another AR-Class called Beaker with liquid with an attribute specifying the liquid quantity. In this case, the value of the attribute can be modified in both the Persistent and Volatile AR-Objects of the AR-Class.
An AR-Class may remain in composition relationships with other AR-Classes. Then, geometry of the composite AR-Class may include geometries of the component AR-Classes, and it may depend on the attributes of those AR-Classes. Consider the AR-Class Beaker with liquid composed of two component AR-Classes: Beaker and Liquid. The Beaker AR-Class contains geometry that can be displayed in an ARE. The Liquid AR-Class has empty geometry so it cannot be directly visualized, because the liquid’s shape is confined to the container it fills. Therefore, geometry of Beaker with liquid describes visualization of the associated liquid in the context of geometry included from the Beaker AR-Class. Geometry of the liquid can be parameterized by the following attributes:
• the quantity of the liquid in the Beaker with liquid,
• the diameter of the Beaker, and
• the color and opacity of the Liquid.
As shown in the example, geometry of the Beaker with liquid AR-Class is directly parameterized by attributes of this AR-Class and indirectly by attributes of its component AR-Classes.
Media Objects
Each AR-Class contains a set of media objects which, in particular, can be empty. The media objects are images, movie clips, and sounds. Media objects contained in an AR-Class can be referenced in the specification of geometry and behavior of the AR-Class. In the geometry specification, images and movies can be used as textures. Sounds can be used as audio clips embedded within geometry model if supported by the language used for the model specification. Media objects can also be used as a supplement to the presentation in an augmented reality environment. They may be presented using appropriate commands specified in the operations defined in AR-Classes. Images and movies can be displayed as 2D overlays on top of the view of an augmented reality environment, while sounds can be played in the background. Images and movies can provide additional information on the contents presented in the environment. Sounds can represent audio effects, background music, or voice instructions.
Some attributes of AR-Classes can be of media object types. These attributes are called media object attributes. There are three types of media object attributes that can appear in an AR-Class, i.e., Image, Movie, and Sound. Media object attributes can be used for parameterization of both geometry and behavior. Thus, textures and audio clips used within geometry of an AR-Class can be parameterized with appropriate media object attributes. In the same way, the parameterization of behavior of the AR-Class can be performed. The values of media object attributes defined in an AR-Class are references to media objects contained in this AR-Class. The values of media object attributes can be different for each Persistent AR-Object created from an AR-Class. These values can be further modified in the Volatile AR-Objects throughout execution of behavior operations.
Behavior
AR-Classes define both the structure of AR-Objects and their behavior. In the AREM approach, behavior is defined by two kinds of operations: methods and activities. Methods and activities defined in an AR-Class are inherited by all its subclasses. In particular, some inherited operations can be overridden in the subclasses. All operations are composed of an interface and implementation. The operation interface consists of parameters, which can be referenced in the operation implementation. The values of the parameters can be set to different values in instances of an AR-Class; thus, the behavior can be customized in AR-Objects and can dynamically change at runtime.
The operations can manipulate Volatile AR-Objects created based on Persistent AR-Objects. After the creation of a Volatile AR-Object from a Persistent AR-Object, the created Volatile AR-Object contains attribute values that are copies of the values from the Persistent AR-Object. During the presentation of the Volatile AR-Object in an ARE, the attribute values can be modified by executing the operations defined in the corresponding AR-Class. If there are several Volatile AR-Objects following from a Persistent AR-Object present in an ARE at a time, each of them can be modified independently.
A method is a sequence of commands that can access and process data in an immediate way. When a method is called by another operation, all its commands are executed instantaneously. During method execution other methods can be called. Method execution does not depend on the events occurring in an ARE.
An activity is a new concept extending the traditional object-oriented approach. Each activity denotes some distinctive interactive behavior of the AR-Objects of an AR-Class. Each object can contain a number of different activities, which can be activated at runtime. Activities describe behavior of AR-Objects in time; in particular describe reactions to some events occurring in an augmented reality environment.
Execution of an activity depends on user interaction and behavior of other Volatile AR-Objects present in an ARE. Each activity of a Volatile AR-Object can be activated or deactivated. When an activity is activated for a Volatile AR-Object, an activity instance is created.
An activity instance is executed by the AREPS system until it is explicitly deactivated or it is completed. The system executes the commands contained in the activity implementation. Activity instance execution can be dependent on the events occurring in an ARE. A number of instances of one activity can be running simultaneously. For one AR-Object, a number of instances of different activities can be executed at the same time. During activity instance execution other methods and activities can be called. Each activity of an AR-Class is implemented as a deterministic finite state machine (FSM) [11]. A finite state machine is a model of behavior composed of a finite number of states, transitions between those states, and actions. A deterministic FSM is a machine such that for every state and for any event there is at most one transition to another state. While an activity instance is being executed, at each time it is in one state, called the current state. A state contains commands that can be executed when the state is current or when transitions to other states are performed.
Interactive behavior in activities is specified by interaction rules and transition rules. Interaction and transition rules are based on the concept of Event-Condition-Action (ECA) rules known from active databases. The ECA rules were originally devised to enhance conventional database systems with the capability of performing some actions in response to certain events. The rules are comprised of three parts: an event, a condition, and an action. An action can be performed as a response to an event, provided that a condition is true [7]. In the AREM approach, for interaction rules performing an action consists in execution of a sequence of commands. In the case of transitions rules, the action is comprised of execution of a sequence of commands followed by a transition to a target state.
Modeling AR-Classes AR-Class
An AR-Class is a 7-tuple (cName, sName,c,S,B,G, M), where:
cName is the name of the AR-Class, sName is the name of the super AR-Class, c denotes the category of the AR-Class,
S is structure of the AR-Class,
B is behavior of the AR-Class,
G is geometry of the AR-Class,
M is a set of media objects.
The cName element specifies an alphanumeric string that uniquely identifies the AR-Class in the R repository. The sName element specifies the name of a super class of the AR-Class. An AR-Class can inherit from only one super class. Inheritance concerns all the elements of a super class, except cName and sName. A subclass is a specialization of its super class, i.e., it may be extended by extra elements and/or it may concretize the data types of attributes and the related AR-Classes. Geometry and each element of behavior inherited from a super class may be overloaded in its subclasses. The c element denotes the name of the category of the AR-Class. It can assume one of the following values: Real, Virtual, or Scene.The values assigned to the attribute must conform to that type. Attributes are divided into properties and relationships. Behavior B is defined by a set of operations for manipulating instances of the AR-Class. The operations are divided into methods and activities. Geometry G is a 3D model associated with an AR-Class. The model can be visualized in an augmented reality environment. The model is parameterized by the use of the properties defined in the AR-Class. Media objects associated with an AR-Class are included in the M set. The set contains images, movie clips, and sounds that can be used in geometry and behavior of the AR-Class.
The aName element specifies an alphanumeric string that uniquely identifies the attribute within the AR-Class. Let a be an attribute, then N (a) denote the name of the attribute a. The type of the attribute is specified in the aType element. The attribute type can be either an atomic or a composite type.
The aConst can assume one of the following values:
• true—denotes a constant attribute, i.e., the value of the attribute cannot be changed in a Volatile AR-Object in the content use phase,
• false—denotes a variable attribute, i.e., the value of the attribute can be modified in a Volatile AR-Object in the content use phase.
If the attribute a is of a particular type, then the set of valid values for a is denoted Dom(a). The aDefValue element specifies a default value that may be used in any AR-Object instantiated from the AR-Class if any other value is not specified. The value must conform to the attribute type, i.e., aDefValue e Dom (a). If the attribute is constant (aConst equal to true), the aDefValue element must contain a direct value. If the attribute is variable (aConst equal to false), the element may contain both a direct value and a derivation function.
The set of attributes S of an AR-Class is a union of three disjoint subsets:
where:
Prop denotes the set of properties of an AR-Class,
Comp denotes the set of composition relationships to other AR-Classes,
Cont denotes the set of containment relationships to other AR-Classes.
