Working with Landsat Data in the Amazon Web Services (AWS) Cloud

This tutorial demonstrates how users access Landsat Data stored within the AWS Cloud environment using basic commands. Landsat data stored in the AWS Cloud is located within U.S. West (Oregon) us-west-2 region in a requester pays Simple Storage Services (S3) bucket. Users interested in utilizing direct access to Landsat data stored in S3 are encouraged to visit the Requester Pays Documentation Page (https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) for information concerning potential egress costs to accessing Landsat data with this method.

AWS Bucket Name: usgs-landsat (Requester Pays)
AWS Region: US West (Oregon) us-west-2

An AWS account is required before undertaking this tutorial. Please see the AWS Account website to establish an account (https://aws.amazon.com/account/).

Use Case Example

This tutorial uses AWS Command Line Interface (CLI) to demonstrate how to list and download Landsat objects within an S3 Bucket. Users can find additional information regarding AWS CLI via the AWS CLI documentation website (https://aws.amazon.com/cli/). This tutorial also uses commonly available Python Tools to draw a preview of a Landsat scene by reading the Landsat Data directly from S3.

Topics Covered

1. Accessing Landsat Data without an AWS Account
   1.1 This section demonstrates the response that users should expect to receive when trying to access the usgs-landsat S3 Bucket with AWS CLI without an AWS Account.
   1.2 This section demonstrates the response that users should expect to receive when trying to access the usgs-landsat S3 Bucket using Python tools without an AWS Account.
2. Accessing Landsat Data with an AWS Account
   2.1 This section demonstrates the response users should expect to receive when trying to access the usgs-landsat S3 Bucket with AWS CLI with an AWS Account.
   2.2 This section demonstrates the response users should expect to receive when trying to access the usgs-landsat S3 Bucket using Python tools with an AWS Account.

1. Accessing Landsat Data without an AWS Account

Important: The usgs-landsat S3 bucket is a Requester Pays Bucket where the requester (users) may incur costs for accessing the data. If a requester pays acknowledgement is not properly passed with the request the AWS CLI command or python call will return a denial of service error. This section provides an example of the errors returned when the user has not properly included the requester pays acknowledgement within the AWS CLI or python request.

1.1 AWS Command Line Response when users to not have an AWS Account

This command will list the contents within the AWS Bucket

! aws s3 ls s3://usgs-landsat/collection02/

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

This command will download an object in S3 to the user's local computer

! aws s3api get-object --bucket usgs-landsat --key collection02/level-2/standard/oli-tirs/2020/026/027/LC08_L2SP_026027_20200827_20200906_02_T1/LC08_L2SP_026027_20200827_20200906_02_T1_MTL.txt LC08_L2SP_026027_20200827_20200906_02_T1_MTL.txt

An error occurred (AccessDenied) when calling the GetObject operation: Access Denied

1.2 Python Response when users do not have an AWS Account

Import Packages

from matplotlib.pyplot import imshow
import rasterio as rio
%matplotlib inline

Load Cloud Optimized GeoTIFF (COG) Band File from the S3 Bucket

cog = 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2020/026/027/LC08_L2SP_026027_20200827_20200906_02_T1/LC08_L2SP_026027_20200827_20200906_02_T1_SR_B2.TIF'

Read COG Band File from the S3 Bucket and Display It

with rio.open(cog) as src:
    profile = src.profile
    arr = src.read(1)
imshow(arr)

---------------------------------------------------------------------------
CPLE_AWSAccessDeniedError                 Traceback (most recent call last)
rasterio/_base.pyx in rasterio._base.DatasetBase.__init__()

rasterio/_shim.pyx in rasterio._shim.open_dataset()

rasterio/_err.pyx in rasterio._err.exc_wrap_pointer()

CPLE_AWSAccessDeniedError: Access Denied

During handling of the above exception, another exception occurred:

RasterioIOError                           Traceback (most recent call last)
<ipython-input-5-e4d29a41fd09> in <module>
----> 1 with rio.open(cog) as src:
      2     profile = src.profile
      3     arr = src.read(1)
      4 imshow(arr)

/srv/conda/envs/pangeo/lib/python3.7/site-packages/rasterio/env.py in wrapper(*args, **kwds)
    431 
    432         with env_ctor(session=session):
--> 433             return f(*args, **kwds)
    434 
    435     return wrapper

/srv/conda/envs/pangeo/lib/python3.7/site-packages/rasterio/__init__.py in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
    219         # None.
    220         if mode == 'r':
--> 221             s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
    222         elif mode == "r+":
    223             s = get_writer_for_path(path, driver=driver)(

rasterio/_base.pyx in rasterio._base.DatasetBase.__init__()

RasterioIOError: Access Denied

Important: This will produce an error message because we have not properly authenticated to with AWS to agree to the terms of the requester pays bucket. Before running the next section clear the kernel before continuing.

2. Accessing Landsat Data with an AWS Account

Important: The usgs-landsat S3 bucket is a Requester Pays Bucket where the requester (user) may incur costs for accessing the data. This section demonstrates sample responses returned to the user’s computer when users have acknowledged requester pays (agreed to pay any incurred charges) for both AWS CLI commands and python requests.

2.1 AWS Command Line Response when the user has an AWS Account

This command will list the contents within the AWS Bucket

! aws s3 ls s3://usgs-landsat/collection02/level-2/standard/oli-tirs/2020/026/027/LC08_L2SP_026027_20200827_20200906_02_T1/ --request-payer requester

This commnad will download an object in S3 to the user's local computer

! aws s3api get-object --bucket usgs-landsat --key collection02/level-2/standard/oli-tirs/2020/026/027/LC08_L2SP_026027_20200827_20200906_02_T1/LC08_L2SP_026027_20200827_20200906_02_T1_MTL.txt  --request-payer requester LC08_L2SP_026027_20200827_20200906_02_T1_MTL.txt

2.2 Python Response when users have an AWS Account

Import Packages

import boto3
from matplotlib.pyplot import imshow
import rasterio as rio
from rasterio.session import AWSSession

Setup AWS Requester Pays Environment

aws_session = AWSSession(boto3.Session(), requester_pays=True)

Load Cloud Optimized GeoTIFF (COG) Band File from the S3 Bucket

cog = 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2020/026/027/LC08_L2SP_026027_20200827_20200906_02_T1/LC08_L2SP_026027_20200827_20200906_02_T1_SR_B2.TIF'

Read COG Band File from the S3 Bucket and Display It

with rio.Env(aws_session):
    with rio.open(cog) as src:
        profile = src.profile
        arr = src.read(1)

imshow(arr)

Congratulations! You have now not only learned how to access the Landsat Data stored within the AWS S3 Cloud Storage and how to properly work with a requester pays bucket. You can incorporate this tutorial into your workflow. We anticipate releasing future tutorials on how users can benefit from using Landsat Data stored in AWS in their workflows

Contact Information

Material written by Nathan Roberts¹

Contact: custserv@usgs.gov
Voice: +1-605-594-6151
Date last modified: 11-16-2020

¹KBR Inc., contractor to the U.S. Geological Survey, Earth Resources Observation and Science (EROS) Center, Sioux Falls, South Dakota, 57198-001, USA. Work performed under USGS contract TSSC.