Development of Marine Debris Detection Software Model

Abstract

Marine debris is one of the major causes of Earth’s pollution, especially for the ocean ecosystem. Detecting marine debris effectively would help evaluate the distribution of marine debris around the Earth, providing better insight into ways to reduce marine debris and also may create more public/government attention to the ocean ecosystem pollution problem. To explore a new approach for detecting marine debris in the ocean from satellites, two different methods were developed and tested. One method is using openCV to detect marine debris by finding edges. This method converts the picture into another form where it can find the edges of objects better. Then, it uses functions from OpenCV such as ’Canny’ to get the outline and figure out where the debris is. This method had about 75.0 percent accuracy. Another method is using a pre-trained AI model called Language-Segment Anything, and it finds debris by, when it gets input of threshold values as well as text value of what object it is looking for, it finds the debris by applying various digital masks and comparing with pictures in the large data set it has studied. For example, there are 100,000 pictures of debris that the model has studied, and it compares the picture given to the studied materials. If it can find similarities, it identifies where the objects are. This method performed better, having a 91.3 percent accuracy. The precision-recall curve was used to evaluate the model’s performance, and the larger than area under the curve (which would look more like a square), the better the model is. The visual representation shows a closely-square-shaped graph.

Introduction

Background

Marine debris refers to the various unnatural objects that are wasted and found in the ocean. These items, including plastics, metals, rubber, paper, textiles, abandoned fishing equipment, and even discarded vessels, are abandoned into the ocean daily, making marine debris a pervasive pollution issue affecting oceans globally. Human-made materials last for thousands of years and can-not be decomposed fast enough to cause less harm to the ocean ecosystem. Marine debris is found in all corners of the ocean, from remote coastlines to the Arctic ice and the deepest parts of the ocean floor.

Overview

In order to reduce marine debris in the ocean, it requires the step of finding the marine debris first. Since there are limitations to finding marine debris by human eyes with the concerns of cost and time, no matter what methods are used, there is a limitation to do with humans. There is a possible solution to it: using computer programs. Since there are many computer programs that distinguish certain objects from pictures, providing programs for pictures of the ocean taken from space by satellite could be a good idea. Since satellites are already available to take precise, high-quality pictures of the Earth, the only challenge was a detection program. This paper introduces three methods tested: color detection, object detection, and a pre-trained AI model. A created CubeSat that contains a camera and a computer takes a picture of the ocean and detects a certain object by color difference or finding the edges of every object. Another method is a pre-trained AI model based on ’language segment anything’ that finds certain given objects in a picture.

Research Motivations

The main goal of this research is to make more people aware of a serious envi-ronmental problem: ocean pollution from all the garbage that ends up in the water, and also to suggest probable programs that ultimately would help reduce aquatic trash by the fusion of AI and marine debris detection model.

Research Questions

How can we design and develop an advanced artificial intelligence (AI) program capable of remotely detecting and identifying marine debris and trash from satellite imagery, thereby contributing to a more comprehensive understand-ing of marine pollution and aiding in the formulation of effective mitigation strategies?

Significance of the Study

The significance of this dissertation is that it introduces ways to make the en-vironment better with specific methods and studies researched and tested.

Literature Review

Negatives and positive outcomes of marine debris in the ocean

Agency-2023¹ defines ”Aquatic trash” and explains its effects on the Earth. ”Aquatic trash” refers to garbage that contaminates waterways due to im-proper disposal, often originating from land-based activities. This marine de-bris, including plastics and cigarette butts, is carried by rain and wind into storm drains, water bodies, and the ocean. Once in the environment, it can spread through wind, stormwater, and rivers, becoming marine debris. The impacts of marine debris are extensive, affecting water quality, endangering wildlife, and polluting recreational spaces. Plastic waste is a major concern due to its persistence and harm to ecosystems. It can break down into harmful mi-croplastics, posing threats to aquatic life and the food chain. The economic and social implications include cleanup costs, health risks, and negative impacts on tourism and fisheries. The article underscores the importance of proper waste management to address this urgent issue.

(Ferries 2023)² shows negatives of marine debris in the ocean. Here are some negatives of marine debris:

• There is an estimated 75 to 199 million tons of plastic waste in the ocean, and 33 billion pounds of plastic waste enters the ocean annually.
• 100 million marine animals die from plastic waste every year.
• The marine garbage patch is twice the surface area of Texas; it outnumbers sea life there 6 to 1

While the negatives follow with marine debris, here are the positives that come from reducing marine debris.

(NOAA 2023)³ introduces positive economic and environmental outcomes of reducing marine debris in the ocean.

The reduction in marine debris can create diverse economic benefits in tourism, fisheries, restaurants, and food service businesses, and the benefits are estimated to be over 7 billion dollars in the US. The individuals who work for the food services industry near the ocean can gain over an extra 6000 dollars annually.

Current methods of detecting marine debris in the ocean

(Service 2023)⁴ discusses the current challenges of detecting marine debris in the ocean.

Satellite-based detection of marine plastic litter is in the research stage; operational monitoring isn’t feasible currently. Identifying individual plastic pieces by satellite is challenging. Dr. Lauren Biermann’s work, using Sentinel-2 satellites, identified floating aggregations, including macroplastics.

Previous trial of detecting marine debris in the ocean

Park et al had research about detecting marine debris in the ocean from space by a satellite. They have built a prototype of CubeSat, a miniaturized satellite consitsts of 10 cm cubes, that takes pictures of ocean-like environment and distinguish marine debris. The CubeSat was able to detect marine debris from the ocean by tracing out the outlines of the debris. They have sent the CubeSat to the MIT NSBE BWSI Build a CubeSat Challenge, and have won first place with their CubeSat and marine debris detection program. Park, the team leader, was interested in improving the detection program, and he deeply researched the marine debris detection program.

AI application in detecting marine debris in the ocean

(Erika 2023)⁵ discusses a research team that applied an AI system to detect marine debris from pictures of oceans.

The Research Team employs AI to identify optimal plastic cleanup locations. Their AI software, coupled with GPS-tagged images, efficiently maps floating plastic. This aids cleanup planning in areas with uneven plastic distribution. GoPro cameras on ships, combined with AI, detected over 400 plastic items. AI object detection relies on labeled images for training. A dataset of over 18,000 images was used. They were able to detect plastic items that size over 50cm.

However, this method is limited to the area it covers and the speed since the pictures are taken from the ground and need ships to travel. However, the method that will be discussed (gathering data from satellite) has more flexibility with those limitations, potentially a better method in terms of time, money, and efficiency.

Methods

Overview

To find a software program that evaluates very well the detection of marine de-bris in pictures of the ocean, we discovered two methods: a program that detects marine debris by finding edges (CubeSat Model), and an AI program based on Language Segment Anything that identifies marine debris on the pictures. The two models tested a given data set (pictures of the ocean) to identify if the pic-tures contained marine debris or not, and they showed a difference in accuracy. The Language Segment Anything-based AI model showed significantly higher accuracy than the CubeSat model, and we identified it as a model that fits into this task for the given data set.

Data

The data set (pictures of the top view of the ocean either containing marine debris or not) is collected from the internet by hand (which causes non-random bias). We chose those pictures since the Satellite or CubeSat could only view the top view of the ocean through their camera. The 12 different images (six marine debris in the ocean[Positive] and six ocean-only pictures[Negative]) collected are being tested on the two models. The dataset is very small due to the limitations of accessing the top ocean view pictures, so this will not be a perfect representation of the model. Nevertheless, it would be mentioned again later that it was able to give us general model performance. The availability of more pictures of the ocean taken from satellites would enable a better evaluation of the model.

Here are some of the data sets collected:

CubeSat Model

To have a software program that detects marine debris in the ocean from satel-lites, making a program that acts with a physical CubeSat device would be a good experiment. The program uses Python Open CV, a library of software functions for computer vision(mainly with visualized image files), to do its de-tection process.

The Program first reads image files from the computer, and it compresses the images to fewer pixels, which is in a smaller size. Then, the program applies a smooth filter to remove sharp points, which the points could make the detection less accurate.

The smoothing process is done by cv2.blur and cv2.erode functions; the blur function helps to prevent wrong edges from being detected, and the erode function makes all found in the image edges bigger – helps to identify edges of marine debris. After the filtering phase, it uses cv2.Canny to find edges found in the picture, which are identified as edges of marine debris.

The Canny function also makes edges in color of white and others color of black. Once it has found edges, it counts the number of pixels that are non-black, which is white for this case, and if the number counted is greater than 3000, it prints that it has found marine debris, and if the number counted is less than 3000, it prints it has not found marine debris.

Here is the Pseudocode of the CubeSat model below:

• Bring image file
• Compresses the image to have fewer pixels and a smaller size
• Apply a smooth filter to remove sharp points
• Apply edge detection software to the smooth images(cite canny)
• Count the length of the edges
• If more than 3000, print ”Plastic has been detected” Else less than 3000, print ”Plastic has not been detected”

Language Segment Anything (LSA) model

This is the GitHub link to the LSA model program.

Language-Segment Anything(LSA) is an open-source AI software target objects in images. Language Segment Anything is built on the segment-anything and the GroundingDINO detection model, and the model has features that are easy to learn and use, and also effective for object detection for images.

The LSA takes three different inputs: text, image, and threshold val-ues. For this research, the text input is the name of a target object that needs to be distinguished from, which is ”marine debris.” The reason why we chose ”marine debris” is because it showed the highest values than the text value of ”trash” and ”trash in the ocean,” which means ”marine debris” is the better setup for the LSA model to evaluate.

Image input is the picture data set, and the thresholds, containing box and text. Box threshold is a minimum value that determines each pixel’s containing of trash; for example: the lower box threshold could possibly identify the edges of the trash, which might result in identifying too many unnecessary parts as trash, while the higher box threshold could possibly hold only the trash parts in the trash, but might not include edges of the trash.

Text threshold is a value that determines how accurate of an object the model would identify as the text prompt. For example, if the text promo is an apple, and the text threshold is minimal, it would only look for objects that look as similar to an apple as possible. However, for larger text threshold values, the model would likely identify apples more often since it does not require as many similarity features as larger text threshold values.

Threshold values can range between 0.0 and 1.0, and I have settled that the box threshold is set to be 0.3, and the text threshold holds a value of 0.25. These are the default values from LSA model creators, and as we have tested different threshold values, we did not see significant changes in the results.

The output of the LSA is the following two: an image that shows the filtered image that has a mask around the target object and a value between 0 and 1 that shows the possibility that the found object is the target object. The values are what I need for the evaluation of predictions.

The created program is based on LSA, a pre-trained AI model that classifies target objects in images. The program is separated into four parts: setting up the variables and libraries for the model, importing the pictures into a folder and extracting the picture names, detecting marine debris in the pictures with reporting probability values, and showing results with a matrix to evaluate each result.

The program created is available in a Git Hub Library, and here is the link.

The first part, setting up the model, is importing libraries that are needed for the LSA model and evaluation program, and the libraries are PIL, lang-sam, matplotlib, sklearn, Keras, and numpy. The libraries are PIL, an image management toolkit; lang-sam, a pre-trained model that segments images based on texts; matplotlib, a GUI toolkit; sklearn, a machine learning library for Python; Keras, a library that provides an interface for artificial neural networks, and numpy, a library for multi-dimensional arrays and matrices, also supports mathematical methods to be applied on arrays and lists.

The second part, importing the pictures and extracting the names, creates a list that contains all the file names of the data set. Then, it goes through a folder corresponding to the computer directory given and extracts the names of all files. The extracted names are saved in a list to be processed later.

The last part, detecting marine debris in the pictures and report-ing probability values, starts off with a code that saves the name of the file as a variable and the length of the name as another variable.

The step required to process detection only for imagine files (in this case, png files), because there could be other types of unnecessary files that exist in the folder, which could be a non-image file that can create an error in the program. Then, it identifies if the file name starts with an ”o” or a ”t.” Files starting with an ”o” are False pictures, pictures containing only the ocean, and files starting with a ”t” are Positive pictures, pictures containing marine debris with the ocean.

For the case of ”o,” it adds a value of [0], which stands for negative, to a list that saves all the actual answers of the data set, and the list is called y-true. For the case of ”t,” it adds a value of [1], which stands for positive, to the same list: y-true.

As the detection program goes through all data sets, the y-true list would contain 0s and 1s. Then, it starts to find an object defined as a text prompt given by the user, which is ”marine debris.” For this case, it prints out a value between 0.0 (no marine debris) and 1.0 (yes marine debris), and saves the value in a list called y-score.

Then, it reformats the y-score list to be compared with the y-true value eventually, but it first converts all values of the y-score to the y-pred. When converting from y-score to y-pred, it stores ”True” or ”False” depending on whether the value is greater than 0.5 or not. In the end, the model has generated a y-pred list containing ”True” and ”False” values, and a y-true list containing ”0” and ”1” values.

Describe how the evaluation works

In order to evaluate the results of each model, graphical data representing results would be a good option. I found a matrix called the confusion matrix, which would be a good graphical representation of the evaluation of the performance of the two models, the CubeSat model and the LSA model.

The confusion matrix is a matrix that represents the prediction summary in a matrix, and it also represents how many correct or wrong predictions it has made. I created a code that visualizes a confusion matrix on a computer screen. The confusion matrix is based on the values of the y-true(answers) list and the y-pred(prediction) list.

There are four different answers that can be made out of each evaluation: False Positive Rate (FTR), False Negative Rate (FNR), True Positive Rate (TPR), and True Negative Rate (TNR).

FTR means the model evaluated true, which means it has identified marine debris, while the picture does not contain marine debris. FNR means the model evaluated False, which means it has not identified marine debris, while the picture contains marine debris.

The two cases above are identified as errors or mistakes in the model. TPR means the model correctly identified marine debris when it actually exists in the picture, and TNR means the model has not found marine debris when there actually is no marine debris in the picture.

The two cases are the results that we want to get, and the more we get TPR and TNR, the better the model for the data set(pictures) tested. In the end, the confusion matrix shows the number of each of the four results(FTR, FNR, TPR, TNR) that occurred in a classification session.

Results

Prediction results for CubeSat model

The confusion matrix of CubeSat (Figure 3) shows that the CubeSat model had some misses in detecting marine debris in ocean pictures. The CubeSat model had one miss where it says it has found marine debris when there is actually no marine debris in the picture and two misses where it says it has not detected any marine debris when there is marine debris in the picture. This represents that the CubeSat model had 83.3 percent accuracy in stating the true negative ’it has not found marine debris while there is no marine debris in the picture, and 66.7 percent accuracy in stating the true positive ’it has found marine debris while there is marine debris existing in the picture’. In summary, we can conclude that the confusion matrix of CubeSat model showed an accuracy of 75.0 percent in detecting detecting marine debris in ocean pictures.

Prediction results for Language Segment Anything model

As shown in the confusion matrix (Figure 4), it shows that the AI model based on language segment anything was able to successfully identify marine debris in the ocean when the marine debris was present in the ocean pictures. It showed 100 percent accuracy in detecting true positives (identifying marine debris in ocean pictures when the picture contained marine debris in it), and 83.3 percent accuracy in detecting true negatives (identifying as it has not detected marine debris while it actually does not exist in the picture). In summary, the confusion matrix of the LSA model showed an accuracy of 91.6 percent in detecting marine debris in ocean pictures.

Here is another representation of the model performance, which is a precision-recall curve. The curve shows the tradeoff between precision and recall for different thresholds, and here is the graphical representation of the LSA model.

Discussion

Comparison between the CubeSat and Language Segment Anything models

Predicted Accuracy performance The CubeSat model showed 83.3 percent accuracy in identifying correctly in positive cases, where the pictures contain marine debris, while the Language Segment Anything model showed 100. per-cent accuracy in identifying correctly in the positive case, which is higher than the CubeSat model by 16.7 percent. For the negative cases where the pic-ture does not contain marine debris, the CubeSat model showed 66.6 percent accuracy, while the Language Segment Anything model showed 83.3 percent accuracy, which is also higher than the CubeSat model by 16.7 percent. This represents that the Language Segment Anything model is a better model for detecting marine debris in the ocean pictures in the sample provided.

COMPUTATIONAL EFFICIENCY The CubeSat model took less than 5 seconds per picture, which is less than 5 MB, but the LSA model took nearly 50 seconds for the same pictures. However, the expectation of computational speed does not require a fast process, and the computer used is just a personal laptop; the LSA model can be satisfied in real-life applications in terms of computational speed requirements.

FLEXIBILITY Since the LSA model can be used to identify objects other than marine debris by changing the text prompt (Ex, marine debris -¿ cars), the flexibility of the model is almost infinite on the Earth as long as it has pre-studied(pre-stored picture library) with sufficient amount of dataset. The CubeSat model requires variable and setting changes to identify other objects, which has significantly less flexibility compared to the LSA model.

Limitation

There were and are going to be some limitations that limit the models’ accuracies in identifying marine debris. The limitations are primarily three main things: a small data set, computational speed, and the actual environment.

Small Data: Since there were only six pictures of the ocean and six pictures of marine debris in the ocean, the data set was too small to make an effective conclusion. We are also not sure what would be a sufficient amount and data type to have a close enough reasonable evaluation of the model since there could be many various looks of the ocean that may affect the results, which was not able to be identified in this research (limitation in data collection without any authorities)

Computational speed: The CubeSat had a Raspberry Pi computer, and computational speed was enough for the small data set, as well as for the lap-top(Macbook Air M2) running a program based on the Language Segment Any-thing model. However, in real-life situations, the satellites need to compute huge image files at a very fast rate, since typical satellites travel around the Earth every 90 minutes. Computing very high-quality pictures constantly at a fast rate would require computing devices with fast processing speeds. Geosynchronous

Actual Environment: Also, the research results would have been more reliable if we had tested our CubeSat and models on the actual satellite, which would make the data set consist of actual pictures taken from satellites. There could be many different variables that can affect the results, such as the brightness of the picture, the shape of the ocean in the picture, clouds covering the ocean, and more.

Future Directions

One of the best solutions for the limitations is to hold this research project on a company or organization with experts. They would reference this research to have ideas of ’better’ and ’efficient’ ways to approach the goal, and they would get sufficient resources and support from the companies and even the government. One suggestive method is to stick with the method of the AI model detecting marine debris in the pictures of the ocean, but have the model trained with a larger data set with pictures of the ocean (both containing marine debris and not containing). The more datasets that have been trained to the model, the better the accuracy and possible processing time it may take to detect objects. The reason why I suggest the AI model is that they eventually would get better at identifying marine debris in the ocean as they study more pictures, and the trained model could also be used in other AI models in the future, contributing huge-scale AI software. There is another suggestive way: use videos of the ocean as a data set rather than photos.

Long-term vision

There are huge benefits humans and the ecosystem of the Earth could obtain if the program has been improved and is being used in satellites to detect marine debris in the ocean. First, we can get data on the distribution of marine debris in the ocean around the Earth and can suggest solutions to remove them based on the data given, which could be more accurate when based on a good data set. Second, the idea of giving ’easier removing marine debris in the ocean’ would drive more people into the environmental problem, which would eventually contribute to the reduction in marine debris in the ocean. Lastly, the reduction in marine debris brings economic benefits for humans. As mentioned in Hoshaw 2009⁶, the reduction in marine debris contributes to producing 217 million dollars spent in tourism communities and over 3,700 jobs, increasing 33.5 million dollars in revenue in Fisheries, and saving an average of 6,000 dollars for restaurants and food service businesses annually.

Conclusions

In this study, two different methods for detecting marine debris in ocean pictures were evaluated: the CubeSat model and the Language Segment Anything (LSA) model. The study has identified that the LSA model had an accuracy of 91.6 percent, which was 16.6 percent higher than the CubeSat model. Since there were limitations mostly on the data set, the reliability, and the amount, if the model was given pictures of the ocean directly from satellites, the model would have been able to train with more data sets and evaluate results better. If all the limitations are improved and the program is deployed worldwide, we can expect a decrease in marine debris over time by detecting and evaluating marine debris in the ocean to help clean the Earth’s oceans. However, several limitations were identified in this research. The most significant limitation was the small data set, consisting of only six ocean pictures and six marine debris pictures. This limits the ability to draw effective conclusions about model performance. Also, computational speed could be a major constraint for satellite-based future applications, since real satellites need to process large image files at high speeds. Expanding the dataset with a broader variety of ocean images, both with and without marine debris, can improve model accuracy. The long-term vision for this research is to develop a highly accurate and efficient AI-based system for detecting marine debris from satellite imagery, which could have numerous benefits, including improved data for marine debris distribution and economic advantages in various sectors. Finally, the sharing of a possible solution to a better environment hopes to inform readers about the current issue and promote an improved solution to be further researched.

References

1. Agency, United States Environmental Protection (2023). Learn About Aquatic Trash. [↩]
2. Ferries, Condor (2023). 100+ Ocean Pollution Statistics Facts 2023 — con-dorferries.co.uk. https : / / www . condorferries . co . uk / marine – ocean – pollution-statistics-facts. [↩]
3. NOAA (2023). The Economic Benefits of Marine Debris Prevention and Re-moval — ORamp;Rapos;s Marine Debris Program — blog.marinedebris.noaa.gov. https://blog.marinedebris.noaa.gov/economic- benefits- marine-debris-prevention-and-removal. [↩]
4. Service, Coopernicus Marine (2023). Detecting Plastic Pollution — CMEMS— marine.copernicus.eu. https://marine.copernicus.eu/explainers/ phenomena-threats/plastic-pollution/detecting-plastic-pollution. [↩]
5. Erika (2023). Using AI to Monitor Plastic Density in the Ocean • Updates • The Ocean Cleanup — theoceancleanup.com. https://theoceancleanup. com/updates/using- artificial- intelligence- to- monitor- plastic-density-in-the-ocean/. [↩]
6. Hoshaw, Lindsey (2009). “Afloat in the ocean, expanding islands of trash”. In:New York Times 9. [↩]