This tutorial demonstrates how users access Landsat Data stored within the AWS Cloud environment using basic commands. Landsat data stored in the AWS Cloud is located within U.S. West (Oregon) us-west-2 region in a requester pays Simple Storage Services (S3) bucket. Users interested in utilizing direct access to Landsat data stored in S3 are encouraged to visit the Requester Pays Documentation Page (https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) for information concerning potential egress costs to accessing Landsat data with this method.
AWS Bucket Name: usgs-landsat (Requester Pays)
AWS Region: US West (Oregon) us-west-2
An AWS account is required before undertaking this tutorial. Please see the AWS Account website to establish an account (https://aws.amazon.com/account/).
This tutorial uses AWS Command Line Interface (CLI) to demonstrate how to list and download Landsat objects within an S3 Bucket. Users can find additional information regarding AWS CLI via the AWS CLI documentation website (https://aws.amazon.com/cli/). This tutorial also uses commonly available Python Tools to draw a preview of a Landsat scene by reading the Landsat Data directly from S3.
Important: The usgs-landsat S3 bucket is a Requester Pays Bucket where the requester (users) may incur costs for accessing the data. If a requester pays acknowledgement is not properly passed with the request the AWS CLI command or python call will return a denial of service error. This section provides an example of the errors returned when the user has not properly included the requester pays acknowledgement within the AWS CLI or python request.
! aws s3 ls s3://usgs-landsat/collection02/
An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
! aws s3api get-object --bucket usgs-landsat --key collection02/level-2/standard/oli-tirs/2020/026/027/LC08_L2SP_026027_20200827_20200906_02_T1/LC08_L2SP_026027_20200827_20200906_02_T1_MTL.txt LC08_L2SP_026027_20200827_20200906_02_T1_MTL.txt
An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
from matplotlib.pyplot import imshow
import rasterio as rio
%matplotlib inline
cog = 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2020/026/027/LC08_L2SP_026027_20200827_20200906_02_T1/LC08_L2SP_026027_20200827_20200906_02_T1_SR_B2.TIF'
with rio.open(cog) as src:
profile = src.profile
arr = src.read(1)
imshow(arr)
--------------------------------------------------------------------------- CPLE_AWSAccessDeniedError Traceback (most recent call last) rasterio/_base.pyx in rasterio._base.DatasetBase.__init__() rasterio/_shim.pyx in rasterio._shim.open_dataset() rasterio/_err.pyx in rasterio._err.exc_wrap_pointer() CPLE_AWSAccessDeniedError: Access Denied During handling of the above exception, another exception occurred: RasterioIOError Traceback (most recent call last) <ipython-input-5-e4d29a41fd09> in <module> ----> 1 with rio.open(cog) as src: 2 profile = src.profile 3 arr = src.read(1) 4 imshow(arr) /srv/conda/envs/pangeo/lib/python3.7/site-packages/rasterio/env.py in wrapper(*args, **kwds) 431 432 with env_ctor(session=session): --> 433 return f(*args, **kwds) 434 435 return wrapper /srv/conda/envs/pangeo/lib/python3.7/site-packages/rasterio/__init__.py in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs) 219 # None. 220 if mode == 'r': --> 221 s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs) 222 elif mode == "r+": 223 s = get_writer_for_path(path, driver=driver)( rasterio/_base.pyx in rasterio._base.DatasetBase.__init__() RasterioIOError: Access Denied
Important: The usgs-landsat S3 bucket is a Requester Pays Bucket where the requester (user) may incur costs for accessing the data. This section demonstrates sample responses returned to the user’s computer when users have acknowledged requester pays (agreed to pay any incurred charges) for both AWS CLI commands and python requests.
! aws s3 ls s3://usgs-landsat/collection02/level-2/standard/oli-tirs/2020/026/027/LC08_L2SP_026027_20200827_20200906_02_T1/ --request-payer requester
! aws s3api get-object --bucket usgs-landsat --key collection02/level-2/standard/oli-tirs/2020/026/027/LC08_L2SP_026027_20200827_20200906_02_T1/LC08_L2SP_026027_20200827_20200906_02_T1_MTL.txt --request-payer requester LC08_L2SP_026027_20200827_20200906_02_T1_MTL.txt
import boto3
from matplotlib.pyplot import imshow
import rasterio as rio
from rasterio.session import AWSSession
aws_session = AWSSession(boto3.Session(), requester_pays=True)
cog = 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2020/026/027/LC08_L2SP_026027_20200827_20200906_02_T1/LC08_L2SP_026027_20200827_20200906_02_T1_SR_B2.TIF'
with rio.Env(aws_session):
with rio.open(cog) as src:
profile = src.profile
arr = src.read(1)
imshow(arr)