 |
 |
 |
TEAM MEMBERS - Alexander Malinin, Ivan Lazarev, Rita Maria Cappello
Our experiment idea is to develop an AI model powerful enough to remove clouds from satellite imagery in a photorealistic and geographically accurate way through GAN image inpainting. Our hypothesis is that our model could be used to enhance and prolong the lifespan of visible-light spectrum satellites currently in orbit, without compromising image quality, accuracy or operating speed with on-board electronics. Our team had found a flaw in current visible light spectrum satellites; the fact that, considering all the environmental factors, satellites can see any given surface of the Earth only 17%[2] of the time. Our team saw that this flaw could be addressed with more powerful tools and out of the box thinking than those of the previously developed solutions. Our product aims to drastically expand the visual capabilities from the exceptionally low 17% clear to obstructed surface ratio on satellite imagery, to an ambitious best-case scenario of a completely clear sky at optimal daylight conditions. An exceptional ~3x fold increase when factoring in Earth’s day and night cycle.
Besides land prediction and cloud removal our team hypothesizes that our model could be used for the detection of anomalous geophysical features on the surface of earth. Based on the simple fact that if our model is not geographically accurate in its prediction, it would either be faulty because of insufficient programming and training, or because the geographical inaccuracy stems from an unpredictable earth feature (and thus, anomalous). We would be able to study our planet and its history more rigorously and efficiently, mapping all points of academic, natural resource, or natural disaster interest. Through the development of our experimental product we also hope to provide one-of-a-kind documentation and lessons for other young student developers and creators to follow and learn from. Our experiment comes with a high return on investment in terms of knowledge, our lessons and thorough project documentation would be made available free for the masses to learn, get inspired, and grow from.
In the beginning of our team’s formation we thought of many conceptually intriguing projects that all failed because of practicality. We realized that a huge obstacle to the implementation of ideas in the AI realm is that satellite images are largely obstructed by clouds, a problem which makes many algorithms painful to implement at best, and almost completely useless at worst. This is when the idea of a cloud removal tool came to mind as a serious potential project to solve the issue of cloud obstruction in satellite imagery going forward. In the research we did on previous winners of the AstroPi competition we took useful information about red-channel cloud isolation from team Centauri, but also researched ways other projects and teams have solved the same problem of cloud masking.
During research of other methods we had found a paper written by researchers Sanju Das, Purnendu Das & Bishwa Ranjan Roy in the beginning of 2020[3] who utilized a similar type of cloud isolation as team Centauri, choosing threshold values in various spectral tests to detect the clouds and remove them. In their paper they discussed why they did not choose the inpainting method, their reasoning cautioning our team of a few drawbacks that they noted, mainly: that this method fails to utilize sources of partial information–areas (semi-transparent clouds), that the image-to-image translation approach requires a paired dataset of cloudy images and their cloud-free counterpart, and that simulated clouds for image masking and training are insufficient due to being algorithm-based (models trained on synthetically generated pairs fail to generalize to real images). Drawing from this feedback we have derived an extra few rules and guides which the whitepaper will expand upon more in the ‘methods’ section.
Our team is aware of the difficulty of finding adequate image pairs of cloudy and cloud-free images, settling on the decision of using artificial cloud masks for our training set. Taking the cautions into account, only cloud-free satellite images at noon will be selected for our training to minimize the impact shadows cast by clouds in natural environments, and clouds will not be generated with an algorithm, but rather picked from a database of isolated red-channel cloud masks that we will generate ourselves. The concern of the model’s failure to utilize sources of partial information–areas is a very valid one, the solution to which our team has decided to choose the model to take the cloud pixel information into account in the process of inpainting, a method we will experiment on to supercharge our model’s ability.
Our initial hypothesis was that our model could be used to enhance and prolong the lifespan of visible-light spectrum satellites currently in orbit, without compromising image quality, accuracy, or operating speed with on-board electronics. The biggest problem for satellites is that their utility depends first and foremostly how much data they can capture. Depending on whether you are interested in cloud or land data, though usually the latter, there are not many things you can do to change mother nature - precipitation. Our model would be able to change that with ease, giving the user and satellite the power to regulate the presence of clouds in their chosen images or datasets. When it comes to the many mission-critical uses of satellite imagery, seconds count. In many cases the obstruction by clouds or, say, tornadoes, hinders rescue, analysis, and forecast teams. If there is no professional analyst on standby or government aid, many crucially important decisions are left to guesswork. By eliminating such a scenario permanently, organizations like FEMA would bypass paperwork, approval, and financial and time costs. The users would be able to put trust into the model knowing that its accuracy is close to the hypothetical ‘real image’.
As we mentioned before, our model is inherently training to learn how Earth is structured geographically and the patterns the structure follows. It would be taking the cloudless land around the mask as context for inpainting, meaning a mountain range around the mask likely means it continues under the obstruction of the clouds, and if the context is a large area of plains, it is likely the obstructed area is also a plain. Leveraging this to our advantage, we can use the AI model to scan the Earth’s surface and find all anomalies where the obstructed area does not follow the expected pattern of its context. This has the potential of discovering out of place geographic features for further academic study and research. Take the small town of Nördlingen, Germany as an example; it was discovered only in the 1960 that the town was actually built within a meteor crater, a fact only learned because the researchers started explicitly looking for the geographical signs of craters around the world. Similarly, our AI would be discovering such anomalies because it looks for all the potential specific possible outcomes, discovering anomalies with ease simply because it is trained on so much data that the likeliest right predictions are all that are most prevalent geographical features. The benefit of such a feature ranges from understanding the history and formation of Earth, the movement of its tectonic plates, volcano and mountain temporal classification, potentially even prediction of how areas of land would look like in the future or how they changed since the past.
Our student team, despite being very passionate on this topic, recognizes the challenges of such a large undertaking, and the risks that come with it. All of our work, data, results, code, entries, change logs, will all be made public on our team’s GitHub page. We have a dedicated writer, Rita Cappello, who will make sure all data is neatly organized and readable in the wiki of the project and that the code is properly documented. Our soon-to-be website will also host all of the data, including an interactive demo of our project using Gradio, and free material we will design for students like us to learn about how we made our model, and how they could make theirs. For data collection and inspection we will also be using WandB, a tool for logging machine learning progress, and ultimately we hope to publish a paper detailing our experiment and results.
Our model will be using Earth Engine’s public data archive of Harmonized Sentinel-2 surface-reflectance data, scale-adjusted to fit the Raspberry Pi camera resolution (m/px resolution data still needed). Our team has been granted developer access to the API, and will host code on Google Cloud TPU that will directly interact with the Earth Engine service, rendering downloading data unnecessary. It is as simple as gathering and feeding images to the model through .tfrecord files that the Earth Engine API generates for us. We will be filtering images by cloud percentage and time of day, this way we can satisfy the criteria we posited earlier in the paper. To generate the second dataset of clouds and their masks we will be filtering for Sentinel-2 images that are cloudy at different percentages; we will create individual datasets for images of different obstruction levels, mainly 10% through 90% in intervals of 10. With every RGB image there will be an algorithm that isolates the clouds through spectral thresholding and generates a definite mask that simply outlines the clouds, and a transparency values mask that will all be fed into the model for better prediction in semi-transparent clouds.
The biggest obstacle our team faces apart from the research and development of our model is its training cost. A similar model, Co-mod GAN by zsyzzsoft[3] reported that their training time for their model took 1 week on 8 NVIDIA Tesla V100 GPUs for 512x512 pixel resolution. Each Tesla V100 GPU produces 62.8 int8 TOPS[4], eight of them would generate 502.4 in8 TOPS. Google Cloud TPU offers renting of their v4 TPU pods which have similar processing power and produce 1,100 int8 TOPS at $12.88/h[5][6]. Doing the math, the processing power of 8 Tesla V100 GPUs comes at a cost of $5.88/h courtesy of Google Cloud TPU, running which for 168h (7*24) comes at a premium of an alarming $988. The $300 sign-up bonus Google offers wouldn’t cover any significant portion of it, even if all our team members were to unethically use all of our sign-up bonuses (which is against Google’s TOS). We hope to apply to Google’s TPU Research Cloud program[7] which would give us free access to TPU Cloud, but that method is not guaranteed or a safe bet. If it is possible for ESA or the Raspberry Pi foundation to fund our research and development costs in any way, we would be extremely grateful, and our project would be allowed to fully blossom and achieve its full potential. If that is not possible, we kindly ask if all of our team members could individually receive an AstroPi kit as we are not physically together and work over the internet as collaborators. This is because we all study at Minerva Virtual Academy, and there is no possibility for any of our team members to fly to the UK.
Once the experiment runs on the AstroPi aboard the ISS, we would have logged hardware data to see how heavy the model is on the hardware components, basic data like location, surface reflectance, cloud percentage, etc., and saved all the images that our model generates along with the original images. We are aware of the 3GB storage limit, which is why we will be using JPEG images for storage. This permits thousands of images to be stored in such a small space, opening up pons. We have possibilities of video creation and analysis of both the pre-processed and the post-processed version to analyze the effectiveness of the model in realistic situations where the clouds cast shadows, there is large amounts of glare, and the top-of-atmosphere reflectance is more pronounced. Obviously we will also be analyzing the posited idea of anomalous landform feature detection.
The hardest part of post-experiment analysis will be testing for how accurate the model is. Our team will need access to previous high-resolution imaging runs in the same locations over Earth that do not contain clouds and are shot where the sun is at the same location to be able to really accurately judge the model’s accuracy. The easiest way to circumvent this issue is to use Earth Engine as a comparative source of the images without clouds, adjusted to the same parameters of the initially captured images (Albeit this method is sub-optimal). We will use the model’s discriminant to compare the two images, and create derived datasets with all possible generated data (i.e. EXIF) that will all be made public for enthusiasts to learn from and use in their own projects. Depending on the outcomes of our model in the analysis phase, we will work to package our model as a legitimate product to be used by corporations, nonprofits and NGOs, and maybe even government organizations. We will post a research paper on our findings in a scientific journal, and make all the code public under the MIT license.
[1] - Acronym; Deep Convolutional Generative Adversarial Network
[2] - M. D. King, S. Platnick, W. P. Menzel, S. A. Ackerman and P. A. Hubanks, "Spatial and Temporal Distribution of Clouds Observed by MODIS Onboard the Terra and Aqua Satellites," in IEEE Transactions on Geoscience and Remote Sensing, vol. 51, no. 7, pp. 3826-3852, July 2013, doi: 10.1109/TGRS.2012.2227333.
[3] - Zhao, Shengyu κ.ά. ‘Large Scale Image Completion via Co-Modulated Generative Adversarial Networks’. International Conference on Learning Representations (ICLR). N.p., 2021. Print.
[4] - “In-Depth Comparison of Nvidia Tesla ‘Volta’ GPU Accelerators.” Microway, 9 Mar. 2022, https://www.microway.com/knowledge-center-articles/in-depth-comparison-of-nvidia-tesla-volta-gpu-accelerators/.
[5] - “Cloud TPU System Architecture Documentation” Google, Google, https://cloud.google.com/tpu/docs/system-architecture-tpu-vm.
[6] - “Cloud TPU Pricing” Google, Google, https://cloud.google.com/tpu/pricing.
[7] - “TPU Research Cloud Program.” TPU Research Cloud - About, https://sites.research.google/trc/about/.