Databases Reference
In-Depth Information
One thing that we cannot do is add extra text into the
FORMAT
attribute. For instance,
in the second case, it would be nice to use a
FORMAT
attribute such as this one:
FORMAT = $1_id::$2
Unfortunately, this will not function as we hope and will instead create the field
id
.
Chaining transforms
As covered before in the
Attributes with class
section, transforms are executed in a
particular order. In most cases, this order does not matter, but there are occasions
when you might want to chain transforms together, with one transform relying on a
field created by a previous transform.
A good example is the source flattening that we used previously, in the
Overriding
source
section. If this transform happened before our transform in the
Creating a
session field from source
section, our session field would always have the value
x
.
Let's reuse two transforms from previous sections and then create one more
transform. We will chain them to pull the first part of
session
into yet another
field. Say we have these transforms:
[myapp_session]
SOURCE_KEY = MetaData:Source
REGEX = session_(.*?)\.log
FORMAT = session::$1
WRITE_META = True
[myapp_flatten_source]
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Source
REGEX = (.*session_).*.log
FORMAT = source::$1x.log
[session_type]
SOURCE_KEY = session
REGEX = (.*?)-
FORMAT = session_type::$1
WRITE_META = True
To ensure that these transforms run in order, the simplest thing would be to place
them in a single
TRANSFORMS
attribute in
props.conf
, like so:
[source:*session_*.log]
TRANSFORMS-s = myapp_session,myapp_flatten_source,session_type
Search WWH ::
Custom Search