Recommending Environmental Big Data Using Semantically Guided Machine Learning - Large Scale and Big Data: Processing and Management - page 470

FIGURE 15.3 The architecture for Big Data to big knowledge representation.

15.3.1 w eb a DaPtors l ayer
Dedicated data adaptors were developed to form this layer. The environmental Big
Data sources used for this study were accessible through ftp web data portals, so
automatically downloading the data, checking the data redundancy, and saving all
the data sources dynamically was the main purpose of this layer. Individual web
data adaptors were implemented as raw data sources which were available in various
file formats (i.e., .txt, .csv, .nc, .shp, and .hdf), so it was essential to have dedicated
data processors for the individual file formats. The other aspect of accessing the
different web-based data was the data security, so web adaptors were designed in
a way that it could automatically use the data agreement protocols already in place
and where applicable. Publicly available data were accessed as open data sources.
For any given latitude and longitude combination point and time period, the near-
est BOM weather station was selected based on minimum geographical distance
from that point and corresponding SILO data files were downloaded and processed.
CosmOz data was also downloaded from the nearest available CosmOz cosmic ray
data station. SILO and CosmOz data were available as time series data sources. The
AWAP database was connected through a secured ftp server and gridded files were
downloaded locally. The ASRIS database was downloaded from publicly available
ASRIS website. For the given location, a pixel position was derived on the daily
continental AWAP gridded map, the historical time series were extracted for all the
available variables by combining individual daily pixel-based data points for the
desired time period. Similarly, a pixel position was also calculated from the ASRIS
gridded map to extract soil resource information for the same latitude and longitude.
The NASA MODIS satellite images were downloaded and processed to extract the
environmental time series data, using image processing framework.