Overview of the Product Algorithm Laboratory

The Product Algorithm Laboratory (PAL) in the framework of the ESA MAAP project is built on a platform named Insula. PAL serves as the core environment enabling the ESA MAAP Project’s goals of collaborative Earth Observation (EO) data analysis, algorithm development, and processing.

Reflecting a modular architecture the high-level functionalities of the PAL are grouped according to the following components:

1. Visualisation & Analytics (Perception): an advanced application within the PAL that empowers users to combine and analyze multiple data sets, including time series vectors, enabling comprehensive big data analytics. It facilitates deep insights by integrating diverse data sources, allowing users to detect patterns, trends, and correlations, driving informed decision-making in complex analytical tasks. This mirrors how perception interprets and analyses sensory data to form an understanding

2. Processing (Intellect): Refers to the section where users (like Service Developers) can manage services, including installing new processors, running processes, and publishing processors to the platform. This captures the cognitive processes of decision-making and execution, much like how intellect guides the brain's higher functions, including service operations and execution.

3. Account Management (Awareness): Refers to the section where users manage storage, set quotas, request data publication, and handle account-related tasks. This reflects the idea of managing and keeping track of data resources and capacities, akin to how awareness in the brain manages the understanding of surroundings and the state of internal systems.

4. Coding (Experiment): The Jupyter Hub environment within PAL, providing users with an interactive platform to conduct real-time experimentation. This application is ideal for developing, testing, and running code in a collaborative and flexible environment, empowering users to experiment with algorithms and workflows seamlessly.

Within PAL the following user profiles are considered:

• Analyst: This role focuses on the detailed processing, visualization, and analysis of EO data, leveraging PAL’s advanced capabilities for scientific and operational outcomes.

• Developer: This profile supports users responsible for creating, testing, and integrating new EO services, including custom algorithms and automated processing chains.

• Curator: Responsible for managing data cataloging, and ensuring that the datasets meet project requirements, as well as maintaining accessibility and compliance within the ESA MAAP environment.

The different brands in the document figures reflect the possibility for ESA MAAP to onboard different initiatives, each with their own branding and unique identity.

This user guide provides a comprehensive overview of how PAL has been adapted for the ESA MAAP project, equipping users with the knowledge and tools to fully exploit its capabilities in the context of Earth Observation research and service development.

Generic capabilities for PAL

With its powerful big data analytics capabilities, PAL allows users to perform complex analyses on this integrated and harmonized dataset. This includes trend analysis, anomaly detection, predictive modelling, and much more, providing users with deep insights into Earth-related phenomena.

The PAL is designed to support and optimize the use and analysis of Earth Observation data. You can tailor the PAL to your specific needs, from selecting the data sources that matter most to creating customized reports and dashboards.

The following generic capabilities can be tailored to support various Product Algorithm Laboratory (PAL) initiatives within the ESA MAAP project.

• Combine Data and Services for Information Generation: seamlessly combination of data and services to transform raw information into actionable insights, empowering informed decision-making. Its standard interface enables to integrate new dataset driven by user needs.

• Integrate New Services: adaptable platform allows users to effortlessly integrate new services, ensuring access to cutting-edge technologies without constraints.

• Run processing campaigns with autoscaling: autoscaling feature optimizes large-scale processing for efficiency and cost-effectiveness.

• Monitor Processing Execution: robust monitoring system offers real-time oversight, enabling users to track progress, identify issues, and make quick adjustments for enhanced efficiency and accuracy.

To summarize, PAL provides a set of functionalities such as the possibility to integrate new services and datasets driven by user needs, a scalability of resources to optimize processing and enable the execution of processing campaigns, a monitoring system that provides real-time supervision to allow rapid adjustments for greater efficiency and accuracy. Flexible and powerful capabilities provide a solid foundation for the various initiatives within the ESA MAAP project.

Key concepts

The following key concepts are used:

• A Bookmark is a logical grouping of satellite products and/or Output products and/or reference data held together for a particular purpose. A Bookmark is identified by a name and a description. Users can view and manage their Bookmarks either within the Data Panel or the Manage & Share interface.

• A Group is a collection of users with a common interest. Groups allow users to both communicate and share material within a selected number of peers. They are managed via the Manage & Share interface.

• A Service is a data processing or visualization application, or workflows made available to the users.

• A Job is a specific data processing activity that has been instigated by the user. Clicking on the Jobs section of the Data Panel allows users to explore Job status. If it has multiple input products, then a job may also be comprised of several sub jobs.

• A Collection is a group of data grouped together due to their common nature. For example, the set of all outputs of a given service would typically be grouped as a Collection.