DEA Surface Reflectance OA (Sentinel-2A MSI)

DEA Surface Reflectance OA (Sentinel-2A MSI)

ga_s2am_ard_3

Version:

3.2.1

Type:

Baseline, Raster

Resolution:

10-60 m

Coverage:

12 Jul 2015 to Present

Data updates:

Daily frequency, Ongoing

Overview

About

DEA Surface Reflectance OA (Sentinel-2A MSI) is part of a suite of Digital Earth Australia’s (DEA) Surface Reflectance datasets that represent the vast archive of images captured by the US Geological Survey (USGS) Landsat and European Space Agency (ESA) Sentinel-2 satellite programs, which have been validated, calibrated, and adjusted for Australian conditions — ready for easy analysis.

Access the data

For help accessing the data, see the Access tab.

See it on a map

DEA Maps

DEA Explorer

DEA Explorer

Access the data on AWS

Data sources

Access the data on NCI

Data sources

View code examples

Code examples

Get via web service

Web services

Key specifications

For more specifications, see the Specifications tab.

Technical name	Geoscience Australia Sentinel-2A Observation Attributes Collection 3
Bands	27 bands of data (nbart_coastal_aerosol, nbart_blue, and more)
DOI	10.26186/146568
Currency	See currency and the latest update date
Collection	Geoscience Australia Sentinel-2 Collection 3
Tags	geoscience_australia_sentinel_2_collection_3, analysis_ready_data, satellite_images, earth_observation, observation_attributes
Licence	Creative Commons Attribution 4.0 International Licence

Cite this product

Data citation

Geoscience Australia, 2022. Geoscience Australia Sentinel-2A Observation Attributes Collection 3 - DEA Surface Reflectance OA (Sentinel-2A MSI). Geoscience Australia, Canberra. https://dx.doi.org/10.26186/146568

Description

Background

Sub-product

This is a sub-product of DEA Surface Reflectance (Sentinel-2A). See the parent product for more information.

The contextual information related to a dataset is just as valuable as the data itself. This information, also known as data provenance or data lineage, includes details such as the data’s origins, derivations, methodology and processes. It allows the data to be replicated and increases the reliability of derivative applications.

Data that is well-labelled and rich in spectral, spatial and temporal attribution can allow users to investigate patterns through space and time. Users are able to gain a deeper understanding of the data environment, which could potentially pave the way for future forecasting and early warning systems.

The surface reflectance data produced by NBART requires accurate and reliable data provenance. Attribution labels, such as the location of cloud and cloud shadow pixels, can be used to mask out these particular features from the surface reflectance analysis, or used as training data for machine learning algorithms. Additionally, the capacity to automatically exclude or include pre-identified pixels could assist with emerging multi-temporal and machine learning analysis techniques.

What this product offers

This product contains a range of pixel-level observation attributes (OA) derived from satellite observation, providing rich data provenance:

• null pixels

• clear pixels (fmask)

• cloud pixels (fmask)

• cloud shadow pixels (fmask)

• snow pixels (fmask)

• water pixels (fmask)

• clear pixels (s2cloudless)

• cloud pixels (s2cloudless)

• cloud probability (s2cloudless)

• spectrally contiguous pixels

• terrain shaded pixels.

It also features the following pixel-level information pertaining to satellite, solar and sensing geometries:

• solar zenith

• solar azimuth

• satellite view

• incident angle

• exiting angle

• azimuthal incident

• azimuthal exiting

• relative azimuth

• relative slope

• timedelta.

Technical information

How observation attributes can be used

This product provides pixel- and acquisition-level information that can be used in a variety of services and applications. This information includes:

• data provenance, which:

  • denotes which inputs/parameters were used in running the algorithm

  • demonstrates how a particular result was achieved

  • can be used as evidence for the reasoning behind particular decisions

  • enables traceability

• training data for input into machine learning algorithms, or additional likelihood metrics for image feature content, where pre-classified content includes:

  • cloud

  • cloud shadow

  • snow

  • water

• additional pixel filtering (e.g. exclude pixels with high incident angles)

• pre-analysis filtering based on image content (e.g. return acquisitions that have less than 10% cloud coverage)

• input into temporal statistical summaries to produce probability estimates on classification likelihood

This product allows you to screen your data for undesired anomalies that can occur during any phase: from the satellite’s acquisition, to the processing of surface reflectance, which relies on various auxiliary sources each having their own anomalies and limitations.

Pixel-level information on satellite and solar geometries is useful if you wish to exclude pixels that might be deemed questionable based on their angular measure. This is especially useful if you are using the NBART product, where pixels located on sloping surfaces can exhibit a lower than expected surface reflectance due to a higher incidence or solar zenith angle.

Example - Cloud and cloud shadow

These images depict an area partially occluded by cloud with visible shadow. Applications, such as land cover, can mis-classify regions if cloud or shadow is misinterpreted as ground observation.

Figure 1. (A) Surface Reflectance (Sentinel-2A) image; (B) Fmask (purple: cloud, yellow: cloud shadow); (C) s2cloudless mask (white: cloud, black: clear); (D) s2cloudless probability.

Terminology for satellite, solar and sensing geometries

• Zenith The point in the sky or celestial sphere directly above a point of interest (in this case, the point being imaged on Earth).

• Solar zenith (degrees) The angle between the zenith and the centre of the sun’s disc.

• Solar azimuth (degrees) The angle of the sun’s position from true north; i.e. the angle between true north and a vertical circle passing through the sun and the point being imaged on Earth.

• Satellite view or satellite zenith (degrees) The angle between the zenith and the satellite.

• Satellite azimuth (degrees) The angle of the satellite’s position from true north; i.e. the angle between true north and a vertical circle passing through the satellite and the point being imaged on Earth.

• Incident angle (degrees) The angle between a ray incident on a surface and the line perpendicular to the surface at the point of incidence.

• Exiting angle (degrees) The angle between a ray reflected from a surface and the line perpendicular to the surface at the point of emergence.

• Azimuthal incident (degrees) The angle between true north and the incident direction in the slope geometry.

• Azimuthal exiting (degrees) The angle between true north and the exiting direction in the slope geometry.

• Relative azimuth (degrees) The relative azimuth angle between the sun and view directions.

• Relative slope (degrees) The relative azimuth angle between the incident and exiting directions in the slope geometry.

• Timedelta (seconds) The time from satellite apogee (the point of orbit at which the satellite is furthest from the Earth).

Figure 2. Zenith angles. Image modified from Support to Aviation Control Service (2011).

Figure 3. Zenith and azimuth angles. θs = solar zenith; θν = satellite view; Φs = solar azimuth (green); Φν = satellite azimuth (blue); Φ = relative azimuth (red). Image modified from Hudson et al. (2006).

Figure 4. Incident (i) and exiting (e) angles for a level and inclined surface. Image modified from Dymond and Shepherd (1999).

The Fmask algorithm

Fmask allows you to have pre-classified image content for use within applications. This can include:

• additional confidence metrics in image content classifiers

• pre-labelled data for machine learning classifiers

• pixel screening for cloud and cloud shadow

• on-the-fly mapping applications for water and snow.

The result of the Fmask algorithm contains mutually exclusive classified pixels, and the numerical schema for the pixels are as follows:

• 0 = null

• 1 = clear

• 2 = cloud

• 3 = cloud shadow

• 4 = snow

• 5 = water.

The s2cloudless algorithm

Sentinel Hub’s cloud detection algorithm is a specialised machine-learning-based algorithm for the Sentinel-2 MSI sensors. This algorithm includes both a per-pixel cloud probability layer (i.e. probability of each satellite pixel being covered by cloud), and an integer cloud mask derived from these cloud probabilities. The numerical schema for the integer cloud mask is:

• 0 = null

• 1 = clear

• 2 = cloud.

Contiguity and terrain

The spectrally contiguous pixels which have a valid observation in each spectral band. This is particularly useful for applications undertaking band math, as it allows non-contiguous data to be ignored during the band math evaluation or masked during post-evaluation. The product can be utilised as a strict mask, and the numerical schema for the pixels are as follows:

• 0 = non-contiguous

• 1 = contiguous.

The terrain-shaded pixels product can be utilised as a strict mask and exclude pixels that were unobservable by the sun or sensor. The numerical schema for the pixels are as follows:

• 0 = shaded

• 1 = not shaded.

Figure 5. Different types of terrain-shaded pixels. C = point of interest; D = point located along the direction of the sun; 90-θS = solar zenith; Z0 = elevation at location C; Zd = elevation at location D. Image sourced from Li et al. (2012).

Example - Fmask

Some analyses might want to exclude targets that are obscured by cloud or cloud shadow. This is particularly useful for applications looking to harvest statistical information for particular regions of interest, such as field crops, where large swaths of data aren’t required to be loaded into computer memory. Instead, only the regions of interest are loaded, analysed and summarised, reducing computational costs.

The following images represent the surface reflectance image and derived Fmask classification result for visual context. The colours for the Fmask classification are displayed as:

• Black = clear

• Magenta = cloud

• Yellow = cloud shadow

• Cyan = snow

• Dark blue = water.

Figure 6. (A) False colour composite; (B) the resulting Fmask classification.

For this product, the Fmask dataset has had the object dilation for the cloud and cloud shadow layers removed. This enables you to customise object dilation to meet your needs for specific applications. For example, one application might work better having a 7-pixel dilation, whereas another might require 5.

You can also choose your own kernel shape and size in which to apply a particular dilation. Dilation can be useful for filling holes within objects and extending the edges of detected objects. It is important to note that small objects (e.g. 1 or 2 pixels in size) will be dilated and become large objects. If this is an undesired outcome, it is best to filter out any small objects prior to applying dilation filters.

For more information on dilation, see:

• Mathematical morphology (Wikipedia)

• Dilation (Wikipedia)

• Multi-dimensional binary dilation (SciPy.org).

Other uses of Fmask:

• For training data for use with machine learning classifiers Fmask can help refine the result and produce a more accurate classification result. The data can also be combined with other classifiers, creating a confidence metric that users can then filter by. For example, you can filter cloud pixels rated >70% as a combined metric from the combination of cloud classifiers.

• For input into a statistical summary It can provide another information product that can be used to indicate the probability of being a particular classified feature. For example, a statistical summary of cloud and/or cloud shadow can highlight pixels that are consistently being detected as a cloud or cloud shadow. As clouds and cloud shadows are non-persistent features, pixels with a high cloud or cloud shadow frequency can be labelled or attributed as highly probable of not being cloud or cloud shadow.

Image format specifications

fmask

Format	GeoTIFF
Resolution	20m
Datatype	UInt8
Classification ENUM	0 = null 1 = clear 2 = cloud 3 = cloud shadow 4 = snow 5 = water
Valid data range	[0,5]
Tiled with X and Y block sizes	512x512
Compression	Deflate, Level 9, Predictor 2
Pyramids	Levels: [8,16,32] Compression: deflate Resampling: mode Overview X&Y block sizes: 512x512
Contrast stretch	None
Output CRS	As specified by source dataset; source is UTM with WGS84 as the datum

s2cloudless-mask

Format	GeoTIFF
Resolution	60m
Datatype	UInt8
Classification ENUM	0 = null 1 = clear 2 = cloud
Valid data range	[0,2]
Tiled with X and Y block sizes	512x512
Compression	Deflate, Level 9, Predictor 2
Pyramids	Levels: [8,16,32] Compression: deflate Resampling: mode Overview X&Y block sizes: 512x512
Contrast stretch	None
Output CRS	As specified by source dataset; source is UTM with WGS84 as the datum

s2cloudless-prob

Format	GeoTIFF
Resolution	60m
Datatype	Float64
No data value	NaN (IEEE 754)
Valid data range	[0,1]
Tiled with X and Y block sizes	512x512
Compression	Deflate, Level 9, Predictor 2
Pyramids	Levels: [8,16,32] Compression: deflate Resampling: mode Overview X&Y block sizes: 512x512
Contrast stretch	None
Output CRS	As specified by source dataset; source is UTM with WGS84 as the datum

nbart-contiguity

Format	GeoTIFF
Resolution	10m
Datatype	UInt8
Classification ENUM	0 = non-contiguous (spectral information not present in each band) 1 = contiguous (spectral information present in each band)
Valid data range	[0,1]
Tiled with X and Y block sizes	512x512
Compression	Deflate, Level 9, Predictor 2
Pyramids	Levels: [8,16,32] Compression: deflate Resampling: GDAL default (nearest) Overview X&Y block sizes: 512x512
Contrast stretch	None
Output CRS	As specified by source dataset; source is UTM with WGS84 as the datum

combined-terrain-shadow

Format	GeoTIFF
Resolution	20m
Datatype	UInt8
Classification ENUM	0 = terrain shadow 1 = not terrain shadow
Valid data range	[0,1]
Tiled with X and Y block sizes	512x512
Compression	Deflate, Level 9, Predictor 2
Pyramids	None
Contrast stretch	None
Output CRS	As specified by source dataset; source is UTM with WGS84 as the datum

incident-angle, exiting-angle, azimuthal-incident, azimuthal-exiting, relative-azimuth, relative-slope, timedelta

Format	GeoTIFF
Resolution	20m
No data value	NaN (IEEE 754)
Tiled with X and Y block sizes	512x512
Compression	Deflate, Level 9, Predictor 2
Pyramids	None
Contrast stretch	None
Output CRS	As specified by source dataset; source is UTM with WGS84 as the datum

Processing steps

1. Longitude and Latitude Calculation

2. Satellite and Solar Geometry Calculation

3. Elevation Retrieval and Smoothing

4. Slope and Aspect Calculation

5. Incidence and Azimuthal Incident Angles Calculation

6. Exiting and Azimuthal Exiting Angles Calculation

7. Relative Slope Calculation

8. Terrain Occlusion Mask

9. Function of Mask (Fmask)

10. Contiguous Spectral Data Mask Calculation

11. Sentinel Hub’s cloud detector for Sentinel-2 imagery

References

Sanchez, A.H., Picoli, M.C.A., Camara, G., Andrade, P.R., Chaves, M.E.D., Lechler, S., Soares, A.R., Marujo, R.F., Simões, R.E.O., Ferreira, K.R. and Queiroz, G.R. (2020). Comparison of Cloud cover detection algorithms on sentinel-2 images of the amazon tropical forest. Remote Sensing, 12(8), 1284.

Quality

Accuracy

For information on the accuracy of the algorithms for test locations, see Zhu and Woodcock (2012) and Zhu, Wang and Woodcock (2015).

Limitations

Fmask

Fmask has limitations due to the complex nature of detecting natural phenomena, such as cloud. For example, bright targets, such as beaches, buildings and salt lakes often get mistaken for clouds.

Fmask is designed to be used as an immediate/rapid source of information screening. The idea is that over a temporal period enough observations will be made to form a temporal likelihood. For example, if a feature is consistently being masked as cloud, it is highly probable that it is not cloud. As such, derivative processes can be created to form an information layer containing feature probabilities.

Edges and fringes of clouds tend to be more opaque and can be missed by the cloud detection algorithm. In this instance, applying a morphological dilation will grow the original cloud object and capture edges and fringes of clouds. However, it is important to note that other cloud objects could also be dilated. Be mindful of single-pixel objects that could grow to become large objects. Consider filtering out these small objects prior to analysis.

s2cloudless

Compared to Fmask, one limitation of the s2cloudless algorithm is the lack of cloud shadow detection. Cloud detection without a thermal band in the Sentinel-2 MSI is difficult, so most of the caveats around the interpretation of the Fmask classification also applies here. However, the machine-learning approach offers some advantages over the traditional physics-based approach here, and the cloud probability layer may be utilized to tune the cloud mask to specific applications.

Angular measurement and shadow classification

The Digital Elevation Model (DEM) is used for identifying terrain shadow, as well as producing incident and exiting angles. It is derived from the Shuttle Radar Topography Mission (SRTM) and produced with approximately 30 m resolution. As such, any angular measurements and shadow classifications are limited to the precision of the DEM itself. The DEM is known to be noisy across various locations, so to reduce any potential extrema, a Gaussian smooth is applied prior to analysis.

Quality assurance

The first Cloud Mask Intercomparison eXercise (CMIX) validated the Fmask and the s2cloudless algorithms together with 8 other algorithms on 4 different test datasets. Both performed well (>85% average accuracy) among the single-scene cloud detection algorithms.

The calculation of the satellite and solar positional geometry datasets are largely influenced by the publicly available ephemeris data and whether the satellite has an on-board GPS, as well as the geographical information that resides with the imagery data and the metadata published by the data providers. The code to generate the geometry grids is routinely tested and evaluated for accuracy at >6 decimal places of precision.

Specifications

Bands

Bands are distinct layers of data within a product that can be loaded using the Open Data Cube (on the DEA Sandbox or NCI) or DEA’s STAC API. Here are the bands of the product: ga_s2am_ard_3.

	Aliases	Resolution	No-data	Units	Type	Description
nbart_coastal_aerosol	nbart_band01 coastal_aerosol	60	-999	-	int16	-
nbart_blue	nbart_band02 blue	10	-999	-	int16	-
nbart_green	nbart_band03 green	10	-999	-	int16	-
nbart_red	nbart_band04 red	10	-999	-	int16	-
nbart_red_edge_1	nbart_band05 red_edge_1	20	-999	-	int16	-
nbart_red_edge_2	nbart_band06 red_edge_2	20	-999	-	int16	-
nbart_red_edge_3	nbart_band07 red_edge_3	20	-999	-	int16	-
nbart_nir_1	nbart_band08 nir_1 nbart_common_nir	10	-999	-	int16	-
nbart_nir_2	nbart_band8a nir_2	20	-999	-	int16	-
nbart_swir_2	nbart_band11 swir_2 nbart_common_swir_1 swir2	20	-999	-	int16	-
nbart_swir_3	nbart_band12 swir_3 nbart_common_swir_2	20	-999	-	int16	-
oa_fmask	fmask	20	0	-	uint8	-
oa_nbart_contiguity	nbart_contiguity	10	255	-	uint8	-
oa_azimuthal_exiting	azimuthal_exiting	20	NaN	-	float32	-
oa_azimuthal_incident	azimuthal_incident	20	NaN	-	float32	-
oa_combined_terrain_shadow	combined_terrain_shadow	20	255	-	uint8	-
oa_exiting_angle	exiting_angle	20	NaN	-	float32	-
oa_incident_angle	incident_angle	20	NaN	-	float32	-
oa_relative_azimuth	relative_azimuth	20	NaN	-	float32	-
oa_relative_slope	relative_slope	20	NaN	-	float32	-
oa_satellite_azimuth	satellite_azimuth	20	NaN	-	float32	-
oa_satellite_view	satellite_view	20	NaN	-	float32	-
oa_solar_azimuth	solar_azimuth	20	NaN	-	float32	-
oa_solar_zenith	solar_zenith	20	NaN	-	float32	-
oa_time_delta	time_delta	20	NaN	-	float32	-
oa_s2cloudless_mask	s2cloudless_mask	60	0	-	uint8	-
oa_s2cloudless_prob	s2cloudless_prob	60	NaN	-	float64	-

For all ‘nbart_’ bands, Surface Reflectance is scaled between 0 and 10,000.

Product information

This metadata provides general information about the product.

Product ID	ga_s2am_ard_3	Used to load data from the Open Data Cube.
Short name	DEA Surface Reflectance OA (Sentinel-2A MSI)	The name that is commonly used to refer to the product.
Technical name	Geoscience Australia Sentinel-2A Observation Attributes Collection 3	The full technical name that refers to the product and its specific provider, sensors, and collection.
Version	3.2.1	The version number of the product. See the History tab.
Lineage type	Baseline	Baseline products are produced directly from satellite data.
Spatial type	Raster	Raster data consists of a grid of pixels.
Spatial resolution	10-60 m	The size of the pixels in the raster.
Temporal coverage	12 Jul 2015 to Present	The time span for which data is available.
Update frequency	Daily	The expected frequency of data updates. Also called ‘Temporal resolution’.
Update activity	Ongoing	The activity status of data updates.
Currency	See the Currency Report	Currency is a measure based on data publishing and update frequency.
Latest update date	Currency Report	See Table A of the report.
DOI	10.26186/146568	The Digital Object Identifier.
Catalogue ID	146568	The Data and Publications catalogue (eCat) ID.
Licence	Creative Commons Attribution 4.0 International Licence	See the Credits tab.

Product categorisation

This metadata describes how the product relates to other DEA products.

Collection	Geoscience Australia Sentinel-2 Collection 3
Tags	geoscience_australia_sentinel_2_collection_3, analysis_ready_data, satellite_images, earth_observation, observation_attributes

Access

Access the data

DEA Maps	See it on a map	Learn how to use DEA Maps.
DEA Explorer	DEA Explorer	Learn how to use the DEA Explorer.
Data sources	Access the data on AWS Access the data on NCI	Learn how to access the data via AWS.
Code examples	View code examples	Learn how to use the DEA Sandbox.
Web services	Get via web service	Learn how to use DEA’s web services.

How to access Sentinel-2 data using the Open Data Cube

This product is contained in the Open Data Cube instance managed by Digital Earth Australia (DEA). This simplified process allows you to query data from its sub-products as part of a single query submitted to the database.

Introduction to DEA Surface Reflectance (Sentinel-2, Collection 3)

How to access DEA Maps

To view and access the data interactively via a web map interface:

1. Visit DEA Maps

2. Click “Explore map data”

3. Select “Baseline satellite data” -> “DEA Surface Reflectance (Sentinel-2)”

4. Click “Add to the map”

History

Version history

Versions are numbered using the Semantic Versioning scheme (Major.Minor.Patch). Note that this list may include name changes and predecessor products.

v3.2.1	-	Current version
v1.0.0	of	DEA Surface Reflectance NBART (Sentinel-2 MSI)

Credits

Acknowledgments

This research was undertaken with the assistance of resources from the National Computational Infrastructure (NCI), which is supported by the Australian Government.

Contains modified Copernicus Sentinel data 2015-present.

The authors would like to thank the following organisations:

• National Aeronautics and Space Administration (NASA)

• Environment Canada

• The Commonwealth Scientific and Industrial Research Organisation (CSIRO)

• National Oceanic and Atmospheric Administration (NOAA) / Earth System Research Laboratories (ESRL) / Physical Sciences Laboratory (PSD)

• The National Geospatial-Intelligence Agency (NGA)

• The United States Geological Survey (USGS) / Earth Resources Observation and Science (EROS) Center

• Spectral Sciences Inc.

License and copyright

© Commonwealth of Australia (Geoscience Australia).

Released under Creative Commons Attribution 4.0 International Licence.