Evaluating the Success of the Digital Earth Project

Abstract

The Digital Earth project aims at a strong interrelation between Data and Earth Science and a step-change in implementing data science methods within Earth science research. During the project, the progress of interdisciplinary collaboration and adoption of data science methods has been measured and assessed with the goal to trace the success of the project. This chapter provides the set-up of this evaluation and the results from two online questionnaires that were held after the start and before the end of the project.

8.1 Objective

The Digital Earth project addresses the challenge of digital transformation and adoption of data science methods in Earth sciences. Therefore, its focus is on linking natural science and data science and to develop approaches for (i) data analysis and exploration; (ii) data collection and monitoring; and (iii) interdisciplinary collaboration, which is of special importance in the digital transformation (see Chap. 2).

During the Digital Earth project, the success of adopting data science methods in the field of Earth sciences has been shown, as well as the scientific progress that can be achieved by doing this. Chapters 3‚ 4‚ 5 and 6 of this book give several examples of this. In addition, the collaboration between the different Earth Science fields and the different research centres involved in the project, and specifically, the collaboration between the Earth Sciences and data science disciplines was another important focus. Thus, we wanted to evaluate the process of interdisciplinary research and the application of data science methods and how this has evolved during the project.

Project monitoring and evaluation are an important process, to identify challenges before and during the project, and to reflect and improve the research project outcomes, learn and adjust activities during the project lifetime, and also to set clear goals for follow-up activities or projects afterwards. Such an evaluation was also deemed useful for the Digital Earth project, as it was the goal to deliver a step-change in the use of data science methods within the different fields of Earth Sciences within the Helmholtz Association. This activity can be seen in the context of the evaluation of other efforts in digitalisation, such as the evaluation of the development of Virtual Research Environments in the United Kingdom (see Junge et al., 2007).

The evaluation aimed at measuring the difference between what the research centres could do at the start of the project, versus what has been achieved after implementation of the Digital Earth project. We also wanted to learn in the course of the project about possible challenges, and the progress and successes of the project, and we wanted to identify possible needs for improvement in process, content and tools during the project.

The approach for this evaluation consisted of the method of online questionnaires. This chapter gives an overview about the online questionnaires; it provides the setting and criteria for evaluation, as well as some of the evaluation results.

8.2 Approach for Evaluation in the Digital Earth Project

The first online questionnaire was done shortly after the start of the project, to establish the important capacities and needs of the research teams to reach success at the start of the project, and to define suitable criteria for measuring such needs. The overarching questions for the evaluation were:

• What are the requirements for data science?

• What is the scientific progress?

• What is the usability of the scientific workflows?

• What is the success of Digital Earth?

These evaluation questions have been used to gather information about the current and desired capacities at the research centres that are involved in the Digital Earth project, as well as about current and anticipated collaboration between the centres, available scientific workflows, digital data and tools and applications. In addition, an assessment was made of the application of the FAIR principles and other measures to enable transfer of results, data and information to other users. The building blocks of the evaluation have included:

• 1. The development of criteria and indicators to be applied;

• 2. Development of survey questions;

• 3. Implementation of a questionnaire at the start of the project;

• 4. The analysis of the questionnaire results, including an analysis of the current scientific and data standards applied at the centres;

• 5. Refinement of the criteria and evaluation questions;

• 6. The repeated monitoring of the progress through a second questionnaire;

• 7. Reporting in a final assessment report.

Afterwards the questionnaire was analysed, and questions were refined and extended.

8.3 Evaluation Criteria

To develop the questionnaire, we adopted criteria for measuring the project status at the beginning of the Digital Earth project. These overall criteria are listed in Table 8.1 and consist of several sub-criteria. For these sub-criteria, evaluation questions have been formulated for the questionnaire (see Appendix) that have been answered by the members of the Digital Earth consortium from all involved research centres. The main categories are:

• Capacities for doing data science;

• Project success and scientific progress in Digital Earth;

• Usability of the results.

Table 8.1 Criteria for assessing success in Digital Earth

The capacities for doing data science were assessed in order to learn shortly after the start of the project what additional capacities and collaborations may be needed. In addition, we wanted to evaluate how the capacities have improved after the implementation of the project. Therefore, the questions were repeated shortly before the end of the project. The project success and scientific progress were assessed, in order to evaluate expectations and status before and after the project implementation.

Finally, the usability of the results was an important topic. We wanted to assess to what extent the methods, scientific workflows as well as the generated data are usable for the scientific community, but also for other users within society. Here, we used the FAIR principles for scientific data (Wilkinson et al., 2016; Stall et al., 2019) to evaluate how data science implementation is done with the multitude of environmental data that is being used and produced.

FAIR stands for Findable, Accessible, Interoperable and Reproducible. We focus on both observational and model data, as well as the scientific software and tools, as they are the essential basis for the research in the Digital Earth project. The FAIR principles allow for a check on the accessibility, usability and quality of the research results and have become an important basis for scientific practice in all research areas worldwide. The implementation of the FAIR principles for the field of Earth and environmental research has gained importance. Making sure that data are “FAIR” is a major prerequisite for applying data science methods. This is in line with other efforts to advance the FAIRness of digital assets and provide open and seamless access to a set of interoperable FAIR data services, such as through the ENVRI-FAIR project (https://envri.eu/home-envri-fair/). This latter project has developed the “FAIRness” assessment methodology, to evaluate the findability, accessibility, interoperability and reproducibility of provided digital assets, including the datasets that are being provided, but also the scientific methods, workflows and software that have been developed in Digital Earth.

In addition, we have assessed to what extent external users have access to project results and are supported in using the data products, methods, workflows and tools. This includes also users outside academia, thereby underlining the need that Earth and Environmental Science research should also benefit society.

The questionnaire is structured according to the three criteria listed in Table 8.1. In the following sections, we present the three main criteria categories and the related sub-criteria: 1) Capacities for doing data science; 2) Project success and scientific progress; and 3) Usability of results. In the Appendix, the text of the questionnaire is provided with all questions related to the sub-criteria presented below.

8.3.1 Capacities for Doing Data Science

Table 8.2 presents the criteria for measuring the capacities to do data science, and additional requirements for improving those capacities. Each sub-criterion is assessed using a specific indicator. For instance, the sub-criterion “team size” is measured using the indicator “number of persons”, which is a quantitative indicator. For other criteria, qualitative descriptions are used, such as for the description of available Data Science capacities within each group, which is measured along a qualitative scale, ranging from “We are doing fine within our Centre”, to “We may need more support from Digital Earth partners or others”.

Table 8.2 Criteria for capacities for doing data science

8.3.2 Scientific and Project Goals

Table 8.3 presents the criteria for assessing the scientific and project goals. These criteria are related to the scientific goals that were set in the project and the extent to which the participants in the questionnaires found they were relevant for their work, or have been achieved towards the end of the project. In addition, we analysed the project process goals, as intermediate steps in the research, including quality of data and models, better guidance for field measurements, saving of resources and time, and improved usability of data, information and workflows.

Table 8.3 Criteria for assessing the scientific and project goals

8.3.3 Usability of Results

Table 8.4 presents the criteria for assessing the usability of the project results. They focus specifically on making project results, data and tools available and accessible within and outside the research field. The criteria and indicators are largely based on the FAIR criteria, and related indicators were developed or adopted from the FAIR framework. In particular, we have used several of the metrics developed by Wilkinson et al. (2018).

Table 8.4 Criteria for assessing usability of results

8.4 Results from the Questionnaires

The questions listed in Appendix were used to evaluate the status and results of the Digital Earth project through a questionnaire, held after 10 months after the start of the project, and shortly before the project end, after 31 months. The questionnaire was set up in the commercial digital online software package Survey Hero (https://www.surveyhero.com). The questionnaire was filled in without personal details, except for the institution and type of research expertise, and staff role. A total number of 54 respondents of the research staff submitted replies to the questionnaire on both occasions. 47 of these respondents completed most questions in the first questionnaire, and 48 respondents during the second time. A total of 118 invitations were sent in the first questionnaire, which implies a response rate of about 50%, which is reasonable for this type of surveys.

About two-thirds of the research staff that responded identifies themselves as Earth Scientist (Table 8.5). In the second questionnaire, the number of staff identifying as data scientists has increased by half.

Table 8.5 Scientific disciplines of the questionnaire participants

The most important results and conclusions from two questionnaires and evaluation within the Digital Earth project include the following:

Several collaborations have been established during the Digital Earth project, during the proposal writing process, and also through collaborations during the project. This is documented through the responses from the researchers in Digital Earth on their collaborations and exchanges with other research centres. In the final questionnaire, each respondent indicated on average 1.7 collaborations with other centres. Towards the end of the project, there was still even more potential and wishes for collaboration reported in the responses. Collaboration with a few specific centres working on data science was highlighted as these are desired for their competences in the field of data science methods.

The project has also progressed on interdisciplinary approaches, showing collaborations between different fields of Earth and Environmental Science, but also tying in data science expertise, in particular related to visual data exploration, Artificial Intelligence, and scientific workflows. In terms of required capacities for doing Data Science, about half of the participants indicate that they have found and benefited from collaboration within the Digital Earth consortium. This has clearly improved during the project, as initially only 32% of the respondents had found collaboration within the consortium after 11 months and underlines the importance of Digital Earth as a platform for inter-institutional cooperation. Considerably fewer respondents indicate they could have done the work alone (23 and 5% of responses in the first and second questionnaire, respectively). This may indicate that through collaboration between centres many new developments have been made possible. The wish for additional support in data science is also frequently mentioned, in about 26% of the responses in the second questionnaire.

We also identified how the Digital Earth project has contributed to progress on integrating data science methods in Earth System Science research, and for which goals (Fig. 8.1). Especially visual data exploration methods have contributed to data-gap closing, and improved scientific understanding within specific disciplines. AI approaches have contributed to data-gap closing, improved scientific understanding and to a lesser extent to data to proxy improvements. Other approaches such as other statistical methods and data quality assessment and controls have contributed to data to proxy approaches, and data quality and uncertainty specification, respectively.

Fig. 8.1

Percentage responses to the question how different data science methods have contributed to achieving the overall scientific goals in the Digital Earth project

As a very important requirement for data science, it was reported that appropriate observation instruments, data collection as well as data infrastructure are indispensable for doing data science. The Digital Earth project therefore would not have been possible without a strong basis of infrastructure and data. It builds on and profits from several complementary efforts focused on field observations and data infrastructure at the individual research centres, as well as targeted collaborative projects in this field.

With regard to science practices, several aspects of FAIR Science were reported. Most respondents make their data (77%) and Software code (67%) available, most often through open-access repositories. However, licences and policies for the publication and use are not applied yet in several cases, and appropriate policy for licensing at the research organisations were reported as an important hurdle. Scientific and observational data, software tools and information are made available beyond academia, reported by about 50% of respondents in the final survey, which is a surprisingly large extent. This availability is complemented by the publication of guidelines for use, tailoring for specific applications and quality assessment.

The researchers report several important indicators that demonstrate the progress and scientific success of Digital Earth. First of all, researchers aimed to increase the usability of data, information and scientific workflows. In addition, they strived for better integration and collaboration between Earth/Environment—and data science disciplines. Joint scientific publications, conference presentations, new research proposals, and (open) software and data publications are regarded as most important signs of success for the Digital Earth project.

8.5 Conclusions

The evaluation reported here has been very useful in documenting the success of the Digital Earth project. The evaluation made use of criteria and indicators to assess the research capacities, goals and usability of results from the endeavour to adopt data science methods for Earth System Science. The framework and questions that are presented have made it possible to demonstrate and analyse the progress made during the project, as we have documented the capacities, goals and usability at the start and close to the end of the project, and the progress made during the collaboration between scientists from different disciplines. During the project, the criteria and indicators have been updated and extended, based on the feedback and evolving insights on what is required in terms of evaluation. In general, the criteria and sub-criteria and indicators presented here can also be applied to other projects in Earth System Sciences, but possibly also to other fields of research. We hope that other researchers and projects also feel encouraged to apply such an evaluation, in order to improve and progress in their research, and analyse and improve their success.

Appendix: Survey Questions

Category*	Included in first questionnaire	Included in second questionnaire	Question
C	1	1	Please indicate at which Helmholtz Center you are working
C	2	2	My role in Digital Earth is
C	3	3	My discipline is. Earth Scientist; Data scientist
C		4	What is your specific discipline?
C	4		How many persons do you have within your research team (team = department, or researchers) working on the Digital Earth project (DE)?
C	5		How many people in your team are working in the following fields of data science: AI / ML / DL; Advanced Data Visualisation; Data management
C	6		How are the capacities in your team for doing data science?
C		5	How are the capacities in your team for doing data science after the Digital Earth project? Have they improved?
C	7	6	With which data science experts from the following centres have you collaborated, within Digital Earth and externally?
C	8		With which Data Scientists and which data science institutes would you like to collaborate, within DE and external?
P	9		Where do you see the limitations of and challenges for data science?
P		7	In your opinion, on which of these limitations have we improved as a result of the Digital Earth project? And which ones do you still encounter?
P	10		What are the key observational and model (digital) datasets you will be using or producing?
P	11		Which observational infrastructure will you be using?
P	12		Which data and information infrastructure will you be using? This can be simulation infrastructure including HPC infrastructure, data storage infrastructure, etc
P	13		Which computer models (software/published code) or model output/data will you be using?
P	14		Will you develop new models or implement new model concepts?
P	15		What data science methods have you used already in other projects?
	16		What data science methods would you like to apply in DE?
P		8	What data science methods have you used in Digital Earth?
P	17		What is the purpose of applying these data science methods?
P		9	Which scientific goals will you achieve/have you achieved? And which data science method(s) have helped to do this?
P	18		Which data exploration tools have you used already in other projects?
P	19		Which data exploration tools would you like to apply in DE?
P	20		What requirements do you have for data management?
P	21		Do you know best practices for data management, and are you using them?
P	22		Which scientific goals do you want to achieve by combining data, methods and data products, tools and knowledge from different disciplines (Earth Sciences, data science):
P	23	10	Which scientific goals of Digital Earth are most important to you?
P		11	What are the key products and tools you have produced? These include data, model (code), tools, workflows etc. Please provide a name/names, or brief description, or repository location
P		12	Have you been harmonising and integrating methods, models and/or data from different disciplines or research fields? If yes, did the Digital Earth project help to find a solution for this task?
P	24		Which research process goals are most important to you?
P		13	Which research process goals have been most important to you?
P	25		Which measures could be appropriate to assess the success of the overall goals of Digital Earth? Please specify the 4 most important:
P		14	Which joint output have you achieved through your work in Digital Earth?
U	26		Are your digital datasets (observations, raw data, model output) findable through a DOI identifier (see https://www.doi.org)?
U	27		Are you and your co-authors using the ORCID (Open Researcher and Contributor ID; see https://orcid.org) identifiers for scientific authors?
U	28		Are you using the IGSN (International Geo Sample Number; see http://www.igsn.org) identifier for geoscientific samples?
U	29	15	Are (parts of) your digital data (observational data, model output, and samples) accessible for other users?
U	30		Are there specific repositories you are using for storing data, accessible for other users?
U	31		Is there a metadata description for these data?
U	32		Will you develop specific DE computer code and scripts for data analysis? If yes, will this be accessible for other users?
U	33		Are there technical standards (specific to your field of study) that you will apply for the data you will be developing?
U	34		Are your computer code and scripts using a formal language?
U	35		Are there Open Data and Software policies for the data, information and software tools you will be producing?
U		16	Has your computer code and scripts for modelling/analysis become available for other users?
U	36		What other principles are important? Please specify below:
U	37	17	Will results including data and software/scripts be made available for users beyond academia?
U	38		If yes, which of the following actions are taken? Usability assessment; User specific quality assessments; Tailoring of data products and methods; Guidelines for use and interpretation; Further support and services; Interface for data and knowledge of stakeholder actions; Other
P	39		Please give us any other comments, or ideas or suggestions for this questionnaire
P		18	What have been the five most important communication and information channels in Digital Earth for you? You can choose 5 at maximum
P		19	What has been the biggest obstacle for collaboration in Digital Earth? (for instance: transfer of information or data between centres)
P		20	What, in your opinion has been the biggest success of Digital Earth? (you can also give us any other comments)

1. *Categories are:
2. C = Capacities for data science
3. P = Project success and scientific progress
4. U = Usability of results