SPF ICE: A Novel Approach to Predict the Optimal Amount of Silica to Preserve Glaciers Using Reinforcement Learning

. Glaciers cover nearly 10 percent of the earth’s surface but are melting at an inexorable rate. According to the paciﬁc standard magazine, the Arctic sea ice has lost 80 percent of its volume since 1979. Antarctica’s ’Doomsday Glacier’ is melting faster and could raise global sea levels by two feet. As three-quarters of the earth’s freshwater is stored in glaciers, its melting depletes freshwater resources for millions of people. Glaciers also play a huge role in the climate crisis. Preserving glaciers is an important and imminent solution to save our planet. Silica microspheres are promising materials to prevent glacier melting as it reﬂects most of the sun’s radiation. When spread in layers over the glacier, it can slow the rate of melt and aid in new ice formation [6]. However, if not used precisely, silica can be ineﬀective and expensive. SPF ICE is a novel method implemented to eﬀectively determine the optimal amount of silica based on glacier’s properties to prevent its depletion substantially using reinforcement learning agents and a custom OpenAI Gym environment. The environment simulates a real-world model of a glacial setting using speciﬁc data, such as the glacier’s mass balance, temperature, and average accumulation and ablation. After testing the agents during many episodes, my solution reduced glacial melting by an average of 60.40% using the optimal amount of Silica. Additionally, this solution is customizable for any type of glacier. SPF ICE is an eﬃcient and low-cost solution to curb glacier melting to preserve planet earth.


Glacial Melting
Glaciers are receding and shrinking at a rapid phase. While melting glaciers are caused by climate change, glacier melting further increases the temperatures across the globe. The phenomenon is called 'ice-albedo feedback.' This feedback arises from the simple fact that ice is more reflective than land or water surfaces. Therefore, as global ice cover decreases, the reflectivity of Earth's surface decreases, more incoming solar radiation is absorbed by the surface, and the surface warms. According to new research, the melting of glaciers as a result of climate change has even knocked the Earth off its axis [3].
Currently, the scale and speed of ice melting are extraordinary. In the summer of 2019, Arctic sea ice levels were tied for the second-lowest ever recorded. During a heatwave in June, Greenland lost nearly 60 billion tons of ice in just five days. Climate models predict that Arctic summers could be ice-free in the next 20 years, as shown in the Figure 1 below. It is the glaciers and ice sheets that are absorbing the brunt of the climate crisis. Preserving glaciers is crucial for various environmental reasons, but the most important one is that it is the source of freshwater for millions of people around the world. About three-quarters of Earth's fresh water is stored in glaciers. Therefore, glacier ice is the second-largest reservoir of water on Earth and the largest reservoir of fresh water on Earth [10]. They store water in the winter and doling it out in the summer as the ice slowly melts. Glaciers are critical to water management, fisheries, and flood prevention. With shrinking glaciers, less water will be available for nearby river systems when rainfall is low. In some parts of the world, millions of people could lose their primary water supplies. In the Pacific north-west US, if glaciers melted entirely, that could reduce the flow of certain watersheds by up to about 15% in dry months of August and September [7]. In Asia, 700 million people will face water problems by 2100 due to melting glaciers in that region. Ninety-eight percent of Andean glaciers have shrunk this century [9]. This melting of South America's glaciers and ice fields poses a threat to water supplies and agriculture from Bolivia to Chile.
Rising sea levels can also introduce new or exacerbate existing saltwater intrusion into freshwater resources [1]. Both groundwater and surface water sources are at risk in coastal cities posing challenges for drinking water treatment facilities and water resource managers. Melting glaciers are contributing to rising sea levels flooding coastal cities throughout the world. Glaciers had predictable seasonal changes, losing mass in the summer and regaining it in the winter. In recent years they are losing more than they accumulate through new snowfall, ultimately adding more water to the oceans, leading to a rise in sea level. Global mean sea level has risen about 8-9 inches since 1880, with about a third of that coming in just the last two and a half decades, and from 2018 to 2019, global sea level rose 0.24 inches (6.1 millimeters) as depicted in Figure 3.a on the left below [8].

Silicon Dioxide
Silica, silicon dioxide, is a compound of the two most abundant elements in Earth's crust: silicon and oxygen. The mass of Earth's crust is nearly 59 percent silica, and it is the main constituent of more than 95 percent of the identified rocks. Silica can reflect most of the radiation from the sun's rays, making it an optimal option to prevent glacier melting. Also, it sticks to ice and water the moment it hits the surface. When sprayed over water, the reflective sand creates a white slush that mimics the reflective properties of ice-meaning that heat from the sun can be reflected outward rather than being absorbed into the ice and sea. It is chemically unreactive, which means it is not prone to a chemical reaction. Since it is hydrophilic, it does not attract any oil-based pollutants. Sand silica can benefit the global silica cycle and ecosystems as long as its size is large enough to be deemed not harmful. Most silica microspheres average between 35-60 micrometers above the health risk threshold. This choice is also safe in desired amounts for animals and ecosystems. The Arctic Ice Project utilizes silica beads that were tested in Alaska have shown promising results. In a paper published by the American Geophysical Union, one field test reported a 15 to 20 percent increase in reflectivity due to the beads. In the Arctic, that could translate into a reduction in 1.5 degree Celsius temperature reduction, a 3-degree reduction in sea temperatures, and an increase in ice thickness up to 20 inches [5].
The Arctic project solution strategically applied in the Arctic can allow the world to buy up to 15 more years to decarbonize the economy and draw down greenhouse gas from the atmosphere [2].

Proposed Solution
The current solution of using the widespread deployment of silica beads is inefficient and expensive, costing billions of dollars. No single glacier or ice sheets are similar in mass balance, debris, density, thermal conductivity, and absorption; a single standard approach of preserving it with silica is not ideal. My approach is entirely novel as there are no solutions that reduce glacial melt using silica intelligently, taking into account all the characteristics and properties of the glacier. Furthermore, current machine learning algorithms only provide the evolution and mapping of glaciers with no solution providing intervention and treatment of glacier melting like SPF ICE. With my novel approach using reinforcement learning, an area of machine learning, the glacial melt can be efficiently reduced by determining the optimal amount of silica needed for adequate reflection of UV rays. As users enter specific properties of a glacier and additional metrics like temperature, average accumulation, and average ablation of the area, my algorithm will accurately determine the amount of silica desired to prevent the melt. Determining the amount of silica is crucial as it not only very cost-effective but also helps reduce any effects of silica on the environment. My solution will be most beneficial for the population that relies on glacial water as a source for drinking water and coastal communities threatened by rising sea levels. They can use this novel solution to accurately and quickly determine the amount of silica needed to reduce glacial melt without using thousands of time-consuming calculations. With all these use cases, SPF ICE has the potential to be beneficial in preserving the glaciers of the world.

OpenAI Gym Environment
The project's software development occurred in two phases: the construction of the custom OpenAI Gym environment and the reinforcement learning (RL) agents, Deep Q Network, and SARSA. The first component is the custom OpenAI gym environment, which simulates the real-life conditions of a glacial setting. The initialization of every OpenAI gym environment consists of an observation space, action space, and the environment's current state. In this scenario, the observation space is the observed mass balance of the glacier, and the action space is represented as a Box, an array of integers from 1-20, referring to the thickness of silica in centimeters. The current state of the environmental attributes to the annual melt rate of the glacier. Other factors defined are the season of the year, temperature, average accumulation, and average ablation. Users can enter values specific to their glacier or use predetermined values for prominent glaciers across the world like Matanuska, Mendenhall, Vatnajökull, and the Lambert glacier.
After the initialization of the environment is complete, each timestep must be defined, which aligns with the seasons of the year in this context. Since glacial conditions are not similar during fall-winter and spring-summer, the timesteps are divided into these two groups. For the fall-winter timesteps, as snowfall is more likely, the accumulation rate is added to the average melt rate. Based on the amount of silica chosen by the RL agent, a new melt rate is calculated using the conductive heat flux formula: Q c = k∆T /h d . The melt rate is then calculated by M = Q c /L f .
The reward system is calculated based on how effective the silica is in preventing additional melt. For this research, I hypothesized that silica could reduce glacial melt by greater than 50%. Therefore, if silica can reduce the melt rate by at least 50% than the current rate, a positive reward is given. However, to constrain the use of an excessive amount of silica, the size of the positive reward is inversely proportional to the thickness of silica. Using more silica yields a smaller positive reward, and using less silica returns a higher reward. If the silica cannot reduce the melt rate by at least 50% of the predicted melt rate, the silica is given a negative reward. The size of the negative reward is directly proportional to the amount of silica used. Using more silica returns a larger negative reward, and using less silica yields a smaller negative reward. In both cases, using more silica is punished more severely than using less silica. The interaction between the RL agent and the environment is depicted below.

Figure 4: Interaction between RL Agent and Environment
As the ultraviolet radiation intensity increases during the spring-summer timestep, the additional ablation melt is added to the annual melt rate. The new melt rate and the reward system are calculated using the amount of silica similar to the fall-winter time step. After several timesteps, when the glacier reaches a mass balance of zero, the environment gets reset. The environment is reinitialized with the original values.

DQN RL Agent
The second component of the software is the reinforcement learning agents: Deep Q Network and SARSA. DQN agent uses two neural networks: the main and target network. Both networks have the same architecture but use different weights to provide stability to the learning process. The neural networks map the input state to the (action, Q-value) pairs. The neural network architecture consists of five layers, including three hidden dense layers of 256 units with the ReLU activation function. The final layer has 20 units for each thickness of silica that could be applied. Using the current state as its input, DQN uses the Boltzmann policy to output the Q-values for all possible actions. The action associated with the highest Q-value is chosen. The agent's decisions or actions will affect the rewards it obtains. During each episode, Deep Q Network attempts to maximize the rewards that it receives. Figure 5 displays the neural network architecture.

SARSA RL Agent
The SARSA agent works differently from the DQN agent. It uses an onpolicy learning algorithm, where in the current state (S), it chooses the best possible action (A) and receives a reward (R). It arrives in a new state (S1) and takes action (A1) in that state, creating the tuple (S, A, R, S1, A1). The Q values represent the possible reward in the next time step after taking the chosen action in the current state, plus the discounted future reward received in the next state. The Q-values are updated based on the action A1 taken in state S1. SARSA also attempts to maximize the rewards that it receives.

Results
After testing the reinforcement learning agents on several episodes, they significantly reduced the melt rate of the glacier. Below are results for comparing both agents to the current melt rate and their rewards to the most optimal policy using the following hyperparameters. These hyperparameters are representative of cirque glaciers, small glaciers found in bowl-shaped depressions found near mountains.
Total Mass of Glacier = 2000 Tonnes Average Accumulation = 10 Tonnes Average Ablation = 25 Tonnes Thermal Conductivity of Silica = 1.11 Latent Heat of Energy for Ice = 334 Figure 6.a compares the average melt of the glacier applying silica using the SARSA agent (blue) compared to the average melt rate without it (red) across 100 episodes. Each episode in the environment represents a season of the year. As evident in Figure 6.b, using silica significantly reduced the melt rate of the glacier between 58.85% and 71.5%. Similar to the SARSA Agent, Deep Q Network significantly reduced the melt rate of the glacier. Figure 7.a compares the average melt of the glacier applying silica using the DQN agent compared to the average melt rate without it. Figure 7.b presents evidence that silica decreases the glacier's melt rate between 58.75% and 70.35%. It is essential to compare their rewards obtained during each episode with the most optimal policy to analyze the learners. The most optimal policy obtains the highest amount of reward possible in an episode. Below the best policy is indicated in red and the agents in blue. Figure 8.a displays the average reward for DQN and Figure 8.b for SARSA.

Discussion
As shown in the results section, both reinforcement learning agents reduced the glacier's melt rates substantially from the current melt rate. The SARSA agent reduced the melt rate from an average of 175.54 to 64.29 using silicon dioxide. This difference amounts to an average decrease of 63.38% in melt rate. The DQN agent achieved similar results to the SARSA agent. Deep Q Network reduced the melt rate from an average of 172.83 to 68.44, an average decrease of 60.40% in the glacial melt. Although Deep Q Network performs slightly worse than the SARSA agent, DQN had less variability in percent difference than SARSA. This statistic could be critical, especially in real-world glacial environments, where minor differences in melt could lead to severe consequences, such as rising sea levels, habitat loss, and loss of glacier stability. Using the more consistent agent, DQN, to determine the correct amounts of silica will be more beneficial.
For further analysis of the RL agents, their reward per episode can be compared to the most optimal policy. The most policy achieves the highest amount of reward per episode. Therefore, it is imperative that both the DQN and SARSA agents have policies that align most closely to the optimal policy. When comparing the RL agents to the most optimal policy, Deep Q Network performs better than SARSA. DQN can better maximize the amount of reward that it receives than SARSA. All of the graphs indicate that DQN is better suited in determining the correct amount of silica to reduce glacial melt across the world.
I am currently working on gathering and testing more real-time glacier data from National Snow and Ice Data Center to augment my modeling to provide the best performance in larger-scale field tests. This data set consists of glacier regime parameters observed between 1945 and 2003. Data include annual mass balances, ablation, accumulation, and equilibrium-line altitude of mountain and subpolar glaciers outside the two major ice sheets. All available sources of information, such as publications, archived data, and personal communications, have been collected and include time series of more than 300 glaciers. Data have been digitized and quality checked [4]. The data collected will be fine-tuned based on the year, categorized based on glaciers' specific properties such as average mass balance, accumulation, and ablation, and then inputted into the OpenAI Gym environment. Using these values, the RL agent will be tested over several episodes.
The values defined in the SPF ICE's OpenAI Gym are exact. However, this is not typical in the real-world glacial setting. According to a normal distribution, a range of values could be provided for the defined values to solve this issue. Therefore, the values are close to the exact values, but there will be more variability and randomness when defined to more closely model changes in a glacial environment.
Using the results obtained through this research, researchers will now be able to quickly and effectively prevent glacial melt in a timely manner. This approach is completely novel as there are no solutions that reduce glacial melting using reinforcement learning and simulations with OpenAI Gym. Furthermore, current machine learning algorithms only provide the evolution and mapping of glaciers with no solution providing intervention and treatment of glacier melting like SPF ICE.

Conclusion
SPF ICE is a revolutionary and novel solution to preserve glaciers worldwide using reinforcement learning and OpenAI Gym to determine the optimal amount of silica needed for adequate reflection of UV rays based on the season, temperature, and mass balance of the glacier without using thousands of time-consuming calculations. Determining the right amount of silica is crucial as it is cost-effective and helps reduce any adverse effects of silica on the natural environment. SPF ICE is customizable for any glacier as users can enter data specific to their glaciers, such as average mass balance, accumulation, and ablation. It is scalable to easily add more prevention techniques for glacial melting, modifying only specific formulas and values defined in the OpenAI Gym environment. With its immense benefits, this accurate and effective solution can be helpful for people around the world that depend heavily on glacial freshwater, prevent flooding of coastal cities, and helps reduce extreme weather events. Because it is cost-effective and easy to implement, it is feasible for any country, including underdeveloped countries, to implement this solution. SPF ICE is a unique high-tech solution to prevent glacier melting that is crucial in preserving planet earth.