Colloquium on data science in earth observation

On November 26th 2018 I attended a colloquium on data science in earth observation. DLR Oberpfaffenhofen invited to that event. The speaker was Professor Xiaoxiang Zhu of the Remote Sensing Technology Institute (IMF) and the TU Munich.

The Copernicus satellites of the Sentinal series were mentioned at the beginning of the talk. Because of their high spacial resolution and their reliable continuous and daily output of data they are a good source for further analysis on its own or in combination with other data sources. Also the data is openly available.

Two companies offering scientific earth observation data were mentioned as well. Unfortunately I do not remember exactly in what context those two were mentioned. Compared to the for example Sentinel data you have to purchase the data those two companies offer. The first one was "Descarts Lab". They offer a demo map were you can see some data. The second one is "Orbital Insight".

An impressive animation that was shown was the shrinking of the arctic sea ice over the last decades by C. Künzer of the DFD. I can't find a link yet. As an alternative I can offer you this one.

One acronym that was dropped was ${\mathrm{AI_{4}EO}}$(https://eo4society.esa.int/wp-content/uploads/2018/09/ai4eo_v1.0.pdf). It stands for "Artificial Intelligence for Earth Observation". It's a project or better a new concept or approach for enhancing earth observation by using AI or neural networks in earth observation.

Next Prof. Zhu mentioned the link between the real world physics (and science) and the results the complex algorithms used show up. As an example she showed an animation of the Berlin main station. With the help of the TanDEM-X data it was possible to show the "breathing" of the building during the year because of thermal expansion. It works like tomographic slices to rebuild a 3D-model of the object that was scanned. With the X-Band used for TanDEM-X a theoretical resolution of around 1 m is be achievable but with the amount of data collected over the years it is possible to extract data of much higher spatial resolution ("Persistent Scatterer Interferometry").

More complex and detailed analysis with potentially higher resolution can be achieved by combining different data sets of different sensors. One aim is to use the EnMAP data. The satellite EnMAP is planned to be launched to space in 2020.

Prof. Zhu than focused on the AI respectively deep learning itself. She compared the first generation neural networks with modern ones. The number of hidden layers rose from 1 to up to 1000 layers. As I'm totally new to the topic I can't explain what that means at this point and first have to do a little research. Besides the obvious of reading the wikipedia article one can read the paper/article "Deep learning in remote sensing: a review" that Prof. Zhu et al. published in 2017.

One question Prof. Zhu asked at this point:
What makes deep learning in earth observation special?

A small wrap up about the use of deep learning in earth observation was:

• Classification of the data is just a small fraction
• In earth observation you have a well controlled data acquisition
• The data is 5 dimensional x, y, z, t,${\lambda}$- deep learning@myteam (can recall details)
• In general there are three ways or sets of algorithms to analyze the data CNN, RNN, GAN

Hyperspectral analysis was the next topic. It covers the third point above. For earth observation a lot of different wavelengths are used to extract information of the atmosphere and/or surface of the earth. Of course it is quite sensible to combine all of the data collected by the different instruments scanning the earth. As those datasets are (becoming) huge as more and more satellites in the past present and future collect data with increasingly high resolution (time and spacial) algorithms have to take over the job.

Open issues or questions that were mentioned are:

• What are novel applications only possible with AI/NN?
• Transferability
• Automated deep topology learning
• Very limited annotated data in remote sensing
• Benchmarking, fast growing number of algorithms
• Combine physics-based models with deep neural network

Some aims mentioned:

• Global Urban Mapping can be used for for goals 1/? and 11/13? of the Sustainable Development Goals of the UN
  Unfortunately I did not note the exact numbers of the goals. Most likely are 1 and 11
• Global Urban Footprint and its map
  GUF+ is planned and worked on. Unfortunately the linked paper is not open access, but here is an abstract and here a blog post by google that was part of the research team

A fascinating idea is to combine data of classical earth observation with data collected by social media platforms like twitter or google maps with it's photos made by users. The name of this idea is So2Sat (Social Media to Satellite). In combinations that means:

• radar data -> height
• hyperspectral data -> roof red
• social media mapping data -> pictures of house facades
• text messages (twitter) -> function of building

This way it is possible to automatically generate complex 3D maps with annotations about the function of certain buildings.

Prof. Zhu showed a comparison of OSM data and twitter mapping. The commercial places were accurately predicted!

Prof. Zhu also mentioned competitions in data science. Some bullet points were:

• Challenge of urbanization in most parts of the world
• Determining and classifying local climate zones are important for urban planners (look for for LCZ42)
• Currently there are 1700 cities with more than 300000 population according to UN data
• Tianchi data scientists
  Alibaba Cloud German AI Challenge 2018
• https://towardsdatascience.com/
  Data Science Competition Platform — Kaggle vs Tianchi

At the end of the colloquium there was a small Q&A. Only two questions/concerns were mentioned:

• usage of twitter data and the protection of privacy
• dual use of the data

During research for this article I stumbled over this page:
Awesome Satellite Imagery Datasets