Analysis Ready Data: Open Data Cube

The Open Data Cube (ODC) is an open-source solution for accessing, managing, and analysing large quantities of Geographic Information System (GIS) data — namely Earth observation data. It is a global initiative to increase the value and use of satellite data by providing users with access to free and open data management technologies and analysis platforms.

At its core, ODC is a set of Python libraries and a PostgreSQL database that allows you to work with geospatial raster data.

This article will show you how to query the DEA Open Data Cube.

Example 1 – Analysis Ready Data product

The following Python code is an example Open Data Cube query to return the following properties:

  • green, red and near-infrared measurements from the GA Landsat 5 TM NBAR Collection 3 and GA Landsat 5 TM NBART Collection 3 products

  • Fmask layer from the GA Landsat 5 TM OA Collection 3 product

  • Region of Interest (ROI): Longitude 146.54 → 147.94; latitude -19.42 → -21.03

  • Acquisition time of interest: 1990-07-20

All products are nestled under the parent product GA Landsat 5 TM Analysis Ready Data Collection 3 (Open Data Cube product ID: ga_ls5t_ard_3).

data = dc.load(
    product='ga_ls5t_ard_3',
    measurements=['nbart_swir_2', 'nbart_blue', 'nbart_green', 'nbart_red', 'nbart_nir', 'nbar_green', 'nbar_red', 'nbar_nir', 'oa_fmask', 'nbart-contiguity'],
    lon=(146.54, 147.94),
    lat=(-19.42, -21.03),
    time=('1990-07-20', '1990-07-21'),
    output_crs='EPSG:3577',
    resolution=(-30, 30)
)

The Data Cube query produces the following images:

AB 3

Image A is the false colour composite [‘nbart_nir’, ‘nbart_red’, ‘nbart_green’] displayed as [Red, Green, Blue] from the **GA Landsat 5 TM NBART Collection 3 **product.

Image B is the false colour composite [‘nbart_nir’, ‘nbart_red’, ‘nbart_green’] displayed as [Red, Green, Blue] from the **GA Landsat 5 TM NBAR Collection 3 **product.

CD 2

Image C is another GA Landsat 5 TM NBART Collection 3 false colour composite, but uses a different combination of spectral bands [‘nbart_swir_2’, ‘nbart_nir’, ‘nbart_blue’] displayed as [Red, Green, Blue].

Image D is the relative slope dataset.

E 2

Image E is the Fmask classification result from the GA Landsat 5 TM NBART Collection 3 product. The colours for the Fmask classification are displayed as:

  • Black = clear

  • Magenta = cloud

  • Yellow = cloud shadow

  • Cyan = snow

  • Dark blue = water

FG 1

Images F and G represent the GA Landsat 5TM NBART Collection 3 false colour composite, with pre- and post-application of the contiguity mask, respectively.

Note that the different spectral bands have different data extents at the edges of the scene (denoted by yellow and red strips). When using the contiguity mask, the data edges have been cut back to where data coexists for all spectral bands.

Example 2 – NBAR product

The following Python code is an example Open Data Cube query to return the following properties:

  • green, red and near-infrared measurements from the GA Landsat 5 TM NBAR Collection 3 product

  • Fmask layer from the GA Landsat 5 TM OA Collection 3 product

  • Region of Interest (ROI): Longitude 146.54 → 147.94; latitude -19.42 → -21.03

  • Acquisition time of interest: 1990-07-20

Both products are nestled under the parent product GA Landsat 5 TM Analysis Ready Data Collection 3 (Open Data Cube product ID: ga_ls5t_ard_3).

data = dc.load(
    product='ga_ls5t_ard_3',
    measurements=['nbar_green', 'nbar_red', 'nbar_nir', 'oa_fmask'],
    lon=(146.54, 147.94),
    lat=(-19.42, -21.03),
    time=('1990-07-20', '1990-07-21'),
    output_crs='EPSG:3577',
    resolution=(-30, 30)
)

The Data Cube query produces the following image, which is the false colour composite [‘nbar_nir’, ‘nbar_red’, ‘nbar_green’] displayed as [Red, Green, Blue].

nbar new

Example 3 – NBART product

The following Python code is an example Open Data Cube query to return the following properties:

  • green, red and near-infrared measurements from the GA Landsat 5 TM NBART Collection 3 product

  • Fmask layer from the GA Landsat 5 TM OA Collection 3 product

  • Region of Interest (ROI): Longitude 146.54 → 147.94; latitude -19.42 → -21.03

  • Acquisition time of interest: 1990-07-20

Both products are nestled under the parent product GA Landsat 5 TM Analysis Ready Data Collection 3 (Open Data Cube product ID: ga_ls5t_ard_3).

data = dc.load(
    product='ga_ls5t_ard_3',
    measurements=['nbart_green', 'nbart_red', 'nbart_nir', 'oa_fmask'],
    lon=(146.54, 147.94),
    lat=(-19.42, -21.03),
    time=('1990-07-20', '1990-07-21'),
    output_crs='EPSG:3577',
    resolution=(-30, 30)
)

The Data Cube query produces the following images:

nbart new

Image A is the false colour composite [‘nbart_nir’, ‘nbart_red’, ‘nbart_green’] displayed as [Red, Green, Blue].

Image B is the Fmask classification result. The colours for the Fmask classification are displayed as:

  • Black = clear

  • Magenta = cloud

  • Yellow = cloud shadow

  • Cyan = snow

  • Dark blue = water

Product file naming convention

DEA product files follow a specific naming convention on Open Data Cube:

{product}-{minor version}-{patch version}{spatial location}{acquisition date}{label}{description id}.{extension}

The fields are defined as follows:

{product} Open Data Cube product ID (e.g. ga_ls5t_nbar_3)
{minor version} A single integer
{patch version} A single integer
{spatial location} The combination of 'path' and 'row' locations. Both 'path' and 'low' are zero-padded strings of length 3, which make a 6-integer number when combined (e.g. 092 and 084 make 092084)
{acquisition date} Has the form yyyy-mm-dd
{label}

Can be one of the following:

nrt

  • If the acquisition file is processed within 48 hours of the acquisition date

interim

  • If the acquisition date is pre-2002-07-01 AND fallback water vapour data is used (i.e. water vapour is not sourced from the same day as the acquisition date)
    OR
  • If the acquisition date is post-2002-07-01 AND fallback water vapour data is used
    OR
  • If the acquisition date is post-2002-07-01, fallback water vapour data is not used (i.e. water vapour is sourced from the same day as the acquisition date) AND fallback BRDF data is used

final

  • If the acquisition date is pre-2002-07-01 AND fallback water vapour data is not used
    OR
  • If the acquisition date is post-2002-07-01, fallback water vapour is not used AND fallback BRDF data is not used

(See flow process diagram below)
{description ID} e.g. band01, band02, band03, thumbnail, fmask, satellite-view
{extension} e.g. tif, jpg

Flow process for determining the label (nrt, interim or final)

logic max

Example files

Below is a list of representative samples which demonstrate the application of the naming conventions. The folders follow the product ID, path, row, year, month, day format.

Example 1 - GA Landsat 5 TM Analysis Ready Data Collection 3

└── ga_ls5t_ard_3              <== Product ID
     └── 092                   <== Path
         └── 084               <== Row
             └── 2009          <== Year
                 └── 12        <== Month
                     └── 17    <== Day
                         ├── ga_ls5t_ard_3-0-0_092084_2009-12-17_final.odc-metadata.yaml
                         ├── ga_ls5t_ard_3-0-0_092084_2009-12-17_final.proc-info.yaml
                         ├── ga_ls5t_ard_3-0-0_092084_2009-12-17_final.sha1
                         ├── ga_ls5t_nbar_3-0-0_092084_2009-12-17_final_band01.tif
                         ├── ga_ls5t_nbar_3-0-0_092084_2009-12-17_final_band02.tif
                         ├── ga_ls5t_nbar_3-0-0_092084_2009-12-17_final_band03.tif
                         ├── ga_ls5t_nbar_3-0-0_092084_2009-12-17_final_band04.tif
                         ├── ga_ls5t_nbar_3-0-0_092084_2009-12-17_final_band05.tif
                         ├── ga_ls5t_nbar_3-0-0_092084_2009-12-17_final_band07.tif
                         ├── ga_ls5t_nbar_3-0-0_092084_2009-12-17_final_thumbnail.jpg
                         ├── ga_ls5t_nbart_3-0-0_092084_2009-12-17_final_band01.tif
                         ├── ga_ls5t_nbart_3-0-0_092084_2009-12-17_final_band02.tif
                         ├── ga_ls5t_nbart_3-0-0_092084_2009-12-17_final_band03.tif
                         ├── ga_ls5t_nbart_3-0-0_092084_2009-12-17_final_band04.tif
                         ├── ga_ls5t_nbart_3-0-0_092084_2009-12-17_final_band05.tif
                         ├── ga_ls5t_nbart_3-0-0_092084_2009-12-17_final_band07.tif
                         ├── ga_ls5t_nbart_3-0-0_092084_2009-12-17_final_thumbnail.jpg
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_azimuthal-exiting.tif
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_azimuthal-incident.tif
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_combined-terrain-shadow.tif
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_exiting-angle.tif
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_fmask.tif
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_incident-angle.tif
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_nbar-contiguity.tif
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_nbart-contiguity.tif
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_relative-azimuth.tif
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_relative-slope.tif
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_satellite-azimuth.tif
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_satellite-view.tif
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_solar-azimuth.tif
                         ├── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_solar-zenith.tif
                         └── ga_ls5t_oa_3-0-0_092084_2009-12-17_final_time-delta.tif

Example 2 - GA Landsat 7 ETM+ Analysis Ready Data Collection 3

└── ga_ls7e_ard_3              <== Product ID
     └── 092                   <== Path
         └── 084               <== Row
             └── 2016          <== Year
                 └── 06        <== Month
                     └── 04    <== Day
                         ├── ga_ls7e_ard_3-0-0_092084_2016-06-04_final.odc-metadata.yaml
                         ├── ga_ls7e_ard_3-0-0_092084_2016-06-04_final.proc-info.yaml
                         ├── ga_ls7e_ard_3-0-0_092084_2016-06-04_final.sha1
                         ├── ga_ls7e_nbar_3-0-0_092084_2016-06-04_final_band01.tif
                         ├── ga_ls7e_nbar_3-0-0_092084_2016-06-04_final_band02.tif
                         ├── ga_ls7e_nbar_3-0-0_092084_2016-06-04_final_band03.tif
                         ├── ga_ls7e_nbar_3-0-0_092084_2016-06-04_final_band04.tif
                         ├── ga_ls7e_nbar_3-0-0_092084_2016-06-04_final_band05.tif
                         ├── ga_ls7e_nbar_3-0-0_092084_2016-06-04_final_band07.tif
                         ├── ga_ls7e_nbar_3-0-0_092084_2016-06-04_final_band08.tif
                         ├── ga_ls7e_nbar_3-0-0_092084_2016-06-04_final_thumbnail.jpg
                         ├── ga_ls7e_nbart_3-0-0_092084_2016-06-04_final_band01.tif
                         ├── ga_ls7e_nbart_3-0-0_092084_2016-06-04_final_band02.tif
                         ├── ga_ls7e_nbart_3-0-0_092084_2016-06-04_final_band03.tif
                         ├── ga_ls7e_nbart_3-0-0_092084_2016-06-04_final_band04.tif
                         ├── ga_ls7e_nbart_3-0-0_092084_2016-06-04_final_band05.tif
                         ├── ga_ls7e_nbart_3-0-0_092084_2016-06-04_final_band07.tif
                         ├── ga_ls7e_nbart_3-0-0_092084_2016-06-04_final_band08.tif
                         ├── ga_ls7e_nbart_3-0-0_092084_2016-06-04_final_thumbnail.jpg
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_azimuthal-exiting.tif
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_azimuthal-incident.tif
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_combined-terrain-shadow.tif
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_exiting-angle.tif
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_fmask.tif
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_incident-angle.tif
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_nbar-contiguity.tif
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_nbart-contiguity.tif
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_relative-azimuth.tif
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_relative-slope.tif
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_satellite-azimuth.tif
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_satellite-view.tif
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_solar-azimuth.tif
                         ├── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_solar-zenith.tif
                         └── ga_ls7e_oa_3-0-0_092084_2016-06-04_final_time-delta.tif

Example 3 - GA Landsat 8 OLI/TIRS Analysis Ready Data Collection 3

└── ga_ls8c_ard_3              <== Product ID
     └── 092                   <== Path
         └── 084               <== Row
             └── 2016          <== Year
                 └── 06        <== Month
                     └── 28    <== Day
                         ├── ga_ls8c_ard_3-0-0_092084_2016-06-28_final.odc-metadata.yaml
                         ├── ga_ls8c_ard_3-0-0_092084_2016-06-28_final.proc-info.yaml
                         ├── ga_ls8c_ard_3-0-0_092084_2016-06-28_final.sha1
                         ├── ga_ls8c_nbar_3-0-0_092084_2016-06-28_final_band01.tif
                         ├── ga_ls8c_nbar_3-0-0_092084_2016-06-28_final_band02.tif
                         ├── ga_ls8c_nbar_3-0-0_092084_2016-06-28_final_band03.tif
                         ├── ga_ls8c_nbar_3-0-0_092084_2016-06-28_final_band04.tif
                         ├── ga_ls8c_nbar_3-0-0_092084_2016-06-28_final_band05.tif
                         ├── ga_ls8c_nbar_3-0-0_092084_2016-06-28_final_band06.tif
                         ├── ga_ls8c_nbar_3-0-0_092084_2016-06-28_final_band07.tif
                         ├── ga_ls8c_nbar_3-0-0_092084_2016-06-28_final_band08.tif
                         ├── ga_ls8c_nbar_3-0-0_092084_2016-06-28_final_thumbnail.jpg
                         ├── ga_ls8c_nbart_3-0-0_092084_2016-06-28_final_band01.tif
                         ├── ga_ls8c_nbart_3-0-0_092084_2016-06-28_final_band02.tif
                         ├── ga_ls8c_nbart_3-0-0_092084_2016-06-28_final_band03.tif
                         ├── ga_ls8c_nbart_3-0-0_092084_2016-06-28_final_band04.tif
                         ├── ga_ls8c_nbart_3-0-0_092084_2016-06-28_final_band05.tif
                         ├── ga_ls8c_nbart_3-0-0_092084_2016-06-28_final_band06.tif
                         ├── ga_ls8c_nbart_3-0-0_092084_2016-06-28_final_band07.tif
                         ├── ga_ls8c_nbart_3-0-0_092084_2016-06-28_final_band08.tif
                         ├── ga_ls8c_nbart_3-0-0_092084_2016-06-28_final_thumbnail.jpg
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_azimuthal-exiting.tif
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_azimuthal-incident.tif
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_combined-terrain-shadow.tif
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_exiting-angle.tif
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_fmask.tif
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_incident-angle.tif
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_nbar-contiguity.tif
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_nbart-contiguity.tif
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_relative-azimuth.tif
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_relative-slope.tif
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_satellite-azimuth.tif
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_satellite-view.tif
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_solar-azimuth.tif
                         ├── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_solar-zenith.tif
                         └── ga_ls8c_oa_3-0-0_092084_2016-06-28_final_time-delta.tif