EarthCARE MAAP
It is recommended to save all of your files and folders within my-private-bucket, since everything that is outside of this folder is permanently deleted after 30 days from the last access into Coding.
Getting to know ESA MAAP's new data access in 15 min
Before Starting
# %pip install pystac_client h5py requests aiohttp h5netcdf
from pystac_client import Client
import fsspec
import xarray as xr
import h5py
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors
from IPython.display import Image, display
import pathlib
import pandas as pdHow to access data
To access EarthCARE data without downloading entire files manually, you can stream the data directly into your Python script using the STAC API and python libraries like fsspec and xarray.
1. Get an Access Token
While most EarthCARE data is open and free it still is protected and requires authentication. Certain collections require more advanced authorization, such as collections only for the Commissioning Team or Cal/Val users.
So you need an access token to check what type of user you are and to then read or download the data:
Generate your token here: MAAP Token
- The token is like a password that tells the server: "This user is allowed to access this data."
- It is currently valid for 90 days.
- Your authorization rights are mapped from eo-login, so you should be able to access the same data as on OADS.
- You can regenerate it anytime and paste the token string into your scripts, or save your token in a separate file.
It is highly recommended to create a .txt file (e.g. credentials.txt) containing all the authentication information that you need.
In Coding, you can create a new .txt file by clicking on "Text File" from the Launcher:
The credentials.txt file should be located in the same directory as the script.
Now you can populate your file with your keys and tokens in this format:
CLIENT_ID=offline-token
CLIENT_SECRET=p1eL7uonXs6MDxtGbgKdPVRAmnGxHpVE
OFFLINE_TOKEN=your_esamaap_longlasting_token_hereYou only have to change OFFLINE_TOKEN, pasting your long lasting token generated before. The other two entries should be kept as they are in the above example.
Be careful to not use quotes around the values and ensure there are no trailing spaces or extra characters.
2. Catalog Query
EarthCARE data, along with data from other EO science missions (Earth Explorers, TPMs, Heritage missions), is stored in an object storage and made accessible via a STAC (SpatioTemporal Asset Catalog) catalog. STAC catalogs provide a standardized, flexible way to describe geospatial data, making it easier to search, discover, and retrieve datasets. The following steps show how to use the STAC API to query the catalog and retrieve the url (https) with which you can then remotely read the data.
Find out more about EarthCARE Data (product definitions): https://earthcarehandbook.earth.esa.int/
The Collections available on ESA MAAP are the following:
COLLECTIONS (open and free):
- Level 1 Data: EarthCAREL1Validated_MAAP
- Level 2 Data: EarthCAREL2Validated_MAAP
COLLECTIONS (Cal/Val):
- Level 1 Data: EarthCAREL1InstChecked_MAAP
- Level 2 Data: EarthCAREL2InstChecked_MAAP
COLLECTIONS (Commissioning):
- Level 0 and 1 Data: EarthCAREL0L1Products_MAAP
- Level 2 Data: EarthCAREL2Products_MAAP
# Point to catalogue endpoint (STAC API)
catalog_url = 'https://catalog.maap.eo.esa.int/catalogue/'
catalog = Client.open(catalog_url)# Use API to query the catalogue
search = catalog.search(
# STEP 1 (manatory): Define a collection -- see naming convention above
collections = ["EarthCAREL2Validated_MAAP"], # Level 2 open and free
# STEP 2 (optional): Apply different filters
# Filter by product type or other parameters: frame, orbit number, etc.
filter="productType = 'ATL_FM__2A' and frame = 'E'",
# BOUNDING BOX is defined by the bottom left corner (longmin latmin) and the top right corner coordinates (longmax latmax)
bbox = [0, -20, 10, -10],
# DATETIME can be a range, or just a start or endtime
datetime=['2025-06-20T00:00:00Z', '2025-07-01T00:00:00Z'],
# This is optional, and can be removed,
max_items=5, # Adjust as needed
method = 'GET' # do not remove
)Our search returns items, each item represents a spatiotemporal asset (one granule, e.g. an EarthCARE file). Each item has, an id and an assets dictionary -- where each case is a named asset. For EarthCARE data we will have a look at:
- enclosure_1: This is where the .h5 (data) is stored.
- product: This is the url to the entire zipped product (.h5 an .HDR)
- thumbnail: This contains the link to a quicklook of the product.
results = {
item.id: {
"enclosure_h5": item.assets.get("enclosure_h5").href if "enclosure_h5" in item.assets else None,
"product": item.assets.get("product").href if "product" in item.assets else None,
"thumbnail": item.assets.get("thumbnail").href if "thumbnail" in item.assets else None,
}
for item in search.items()
}
print(f"Your search returned {len(results)} items.")
if results:
print("\nAvailable item IDs:")
keys = list(results.keys()) # Convert keys to list for indexing
for idx, item_id in enumerate(keys, start=1):
print(f"{idx}. {item_id}")
print(f"\nIn the next cell please select one of the items")
selected_id = 1 # Change this number as desired
if 1 <= selected_id <= len(keys):
chosen_key = keys[selected_id - 1] # map number to key
print(f"Selected item ID: {chosen_key}")
4. Quicklook
# Get the thumbnail URL
thumbnail_url = results[chosen_key].get("thumbnail")
# Display the image if it exists
if thumbnail_url:
display(Image(url=thumbnail_url))
else:
print("No thumbnail found for the first item.")
5. Reading Remote Data On-the-Fly with fsspec and xarray
(No Download Required)!
First, you need to extract your authorization information from the credentials.txt file:
import os
from pathlib import Path
import requests
import aiohttp
# --- Path to credentials.txt ---
CREDENTIALS_FILE = Path('path/to/credentials.txt').resolve().parent / "credentials.txt" # Insert the .txt path
def load_credentials(file_path=CREDENTIALS_FILE):
"""Read key-value pairs from a credentials file into a dictionary."""
creds = {}
if not file_path.exists():
raise FileNotFoundError(f"Credentials file not found: {file_path}")
with open(file_path, "r") as f:
for line in f:
line = line.strip()
if not line or line.startswith("#"):
continue
if "=" not in line:
continue
key, value = line.split("=", 1)
creds[key.strip()] = value.strip()
return creds
# --- ESA MAAP API ---
def get_token():
"""Use OFFLINE_TOKEN to fetch a short-lived access token."""
creds = load_credentials()
OFFLINE_TOKEN = creds.get("OFFLINE_TOKEN")
CLIENT_ID = creds.get("CLIENT_ID")
CLIENT_SECRET = creds.get("CLIENT_SECRET")
print(CLIENT_SECRET)
if not all([OFFLINE_TOKEN, CLIENT_ID, CLIENT_SECRET]):
raise ValueError("Missing OFFLINE_TOKEN, CLIENT_ID, or CLIENT_SECRET in credentials file")
url = "https://iam.maap.eo.esa.int/realms/esa-maap/protocol/openid-connect/token"
data = {
"client_id": CLIENT_ID,
"client_secret": CLIENT_SECRET,
"grant_type": "refresh_token",
"refresh_token": OFFLINE_TOKEN,
"scope": "offline_access openid"
}
response = requests.post(url, data=data)
response.raise_for_status()
response_json = response.json()
access_token = response_json.get('access_token')
if not access_token:
raise RuntimeError("Failed to retrieve access token from IAM response")
return access_tokentoken = get_token()
data_href = results[chosen_key].get("enclosure_h5")
fs = fsspec.filesystem("https", headers={"Authorization": f"Bearer {token}"})
f = fs.open(data_href, "rb")
ds = xr.open_dataset(f, engine="h5netcdf", group="ScienceData")
ds
# Let's actually plot our data!
ds["featuremask"].plot(x="along_track", y="ATLID_height") 
Our plot doesn't look quite right compared to the quicklook -- let's clean it up!
# Define the discrete feature mask classes (matching your classification values)
classes = [-3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
colors = [
"#000000", # -3: Surface
"#999999", # -2: No retrievals
"#0000ff", # -1: Attenuated
"#c0efff", # 0: Clear Sky
"#40cfff", # 1: Likely clear sky
"#40cfff", # 2: Likely clear sky
"#40cfff", # 3: Likely clear sky
"#40cfff", # 4: Likely clear sky
"#cccc99", # 5: Low altitude aerosols
"#ffff66", # 6: Aerosol/thin cloud
"#ffff66", # 7: Aerosol/thin cloud
"#ff6600", # 8: Thick aerosol/cloud
"#ff6600", # 9: Thick aerosol/cloud
"#990000", # 10: Thick cloud
]
# Since you have 14 classes, create colormap with these colors
cmap = mcolors.ListedColormap(colors)
# Labels for colorbar ticks, in order of the classes:
labels = [
"Surface",
"No retrievals",
"Attenuated",
"Clear Sky",
"Likely clear sky",
"Likely clear sky",
"Likely clear sky",
"Likely clear sky",
"Low altitude aerosols",
"Aerosol/thin cloud",
"Aerosol/thin cloud",
"Thick aerosol/cloud",
"Thick aerosol/cloud",
"Thick cloud",
]
# Plotting
#ds = ds.assign_coords(height_km=ds["height"] / 1000.0)
ds = ds.assign_coords(
latitude=ds["latitude"],
longitude=ds["longitude"],
height_km=ds["height"] / 1000.0
)
quadmesh = ds["featuremask"].plot(
x="latitude",
y="height_km",
cmap=cmap,
figsize=(14, 6),
vmin=-3,
vmax=10,
)
quadmesh.axes.set_ylim(0, 30)
quadmesh.axes.set_ylabel("Height [km]")
quadmesh.axes.set_xlabel("Latitude")
quadmesh.axes.set_title("ATLID Feature Mask")
# Adjust colorbar ticks and labels
cbar = quadmesh.colorbar
cbar.set_ticks(classes)
cbar.set_ticklabels(labels)
cbar.set_label("Feature Mask Classification")
No comments to display
No comments to display