Sara Beery arrived at MIT as an assistant professor within the Institute’s Department of Electrical Engineering and Computer Science (EECS) keen to address ecological issues. She has shaped her academic career around the chance to leverage her knowledge in computer vision, machine learning, and data science to confront tangible problems in conservation and sustainability. Beery was attracted to the Institute’s dedication to “computing for the planet,” and embarked on a mission to implement her techniques for global-scale environmental and biodiversity surveillance.
In the Pacific Northwest, salmon profoundly influence the vitality of their ecosystems, and their intricate reproductive requirements have captivated Beery’s interest. Annually, millions of salmon initiate their migration to reproduce. Their expedition commences in freshwater streams where the embryos hatch. Young salmon fry (newly hatched salmon) journey to the ocean, where they spend several years developing into adulthood. Upon reaching maturity, the salmon return to the streams of their origin to spawn, ensuring the survival of their species by laying their eggs in the gravel of the stream beds. Both male and female salmon perish shortly after providing the river habitat with the subsequent generation of salmon.
During their migration, salmon sustain a diverse array of organisms within the ecosystems they traverse. For instance, salmon transport nutrients such as carbon and nitrogen from the ocean upstream, making these resources more accessible in those ecosystems. Furthermore, salmon are vital in numerous predator-prey dynamics: They serve as sustenance for various predators, including bears, wolves, and birds, while simultaneously helping regulate other populations, such as insects, through predation. After their spawning process, the decaying salmon corpses contribute valuable nutrients back to the surrounding environment. The migration of salmon not only supports their own species but is essential for the overall health of the rivers and oceans they inhabit.
Concurrently, salmon populations are significant from both economic and cultural perspectives in the region. Commercial and recreational salmon fishing industries play a crucial role in the local economy. Additionally, for many Indigenous communities in the Pacific Northwest, salmon possess considerable cultural significance, being integral to their diets, customs, and ceremonies.
Monitoring salmon migration
Rising human activity, including overfishing and hydropower expansion, in conjunction with habitat destruction and climate change, has severely impacted salmon populations in the area. Consequently, effective oversight and management of salmon fisheries are vital to preserve balance among competing ecological, cultural, and human interests. Precisely counting salmon during their seasonal migration to their birthplace to spawn is crucial for monitoring vulnerable populations, evaluating the effectiveness of recovery initiatives, directing fishing season regulations, and aiding in the oversight of both commercial and recreational fisheries. Accurate population figures empower decision-makers to employ optimal strategies to protect the ecosystem’s health while addressing human requirements. Monitoring salmon migration is a demanding and inefficient endeavor.
Beery is presently spearheading a research initiative that endeavors to enhance salmon monitoring through advanced computer vision techniques. This project aligns with Beery’s broader research focus, which centers on the interdisciplinary intersection of artificial intelligence, the natural world, and sustainability. Its significance to fisheries management made it an excellent candidate for funding from MIT’s Abdul Latif Jameel Water and Food Systems Lab (J-WAFS). Beery’s 2023 J-WAFS seed grant was the first research funding she received since becoming part of the MIT faculty.
Historically, monitoring efforts depended on humans to manually count salmon from riverbanks using their eyesight. In recent decades, underwater sonar systems have been introduced to assist in counting the salmon. These sonar systems function like underwater video cameras, but differ as they utilize acoustics instead of light sensors to detect the presence of a fish. Employing this method necessitates personnel to establish a tent along the river to count salmon based on the output from a sonar camera connected to a laptop. Although this system represents an advancement over visual monitoring, it remains heavily reliant on human effort and is labor-intensive and time-consuming.
Automating salmon monitoring is essential for more effective management of salmon fisheries. “We require these technological resources,” asserts Beery. “We cannot keep pace with the demand for monitoring, comprehending, and studying these incredibly complex ecosystems without some degree of automation.”
To automate the counting of migrating salmon populations in the Pacific Northwest, the project team, including Justin Kay, a PhD candidate in EECS, has been gathering data in the form of video footage from sonar cameras across various rivers. The team annotates a portion of the data to train the computer vision system to autonomously detect and quantify the fish during their migration. Kay explains how the model counts each migrating fish: “The computer vision algorithm is programmed to identify a fish in the frame, outline it, and then monitor it over time. If a fish is identified on one end of the screen and exits on the opposite end, we count it as moving upstream.” In rivers where the team has developed training data for the system, it has delivered impressive results, with only 3 to 5 percent counting error. This error rate is significantly lower than the target of no more than 10 percent established by the team and collaborating stakeholders.
Testing and deployment: Balancing human effort and use of automation
The researchers’ technology is currently being utilized to monitor salmon migration on the recently restored Klamath River. Four dams along the river were recently removed, marking the most extensive dam removal project in U.S. history. The removal followed a campaign exceeding 20 years led by Klamath tribes in partnership with scientists, environmental organizations, and commercial fishermen. Following the dam demolition, 240 miles of river now flow freely, and nearly 800 square miles of habitat have become accessible to salmon. Beery observes the almost immediate revival of salmon populations in the Klamath River: “It seems that within just eight days after the dam was taken down, they began to observe salmon actually migrating upstream beyond the dam.” Collaborating with California Trout, the team is actively processing new data to adapt and develop a tailored model that can subsequently be deployed to assist with counting the newly migrating salmon.
A challenge faced by the system is training the model to accurately count the fish in unfamiliar environments with variations such as riverbed features, water clarity, and lighting conditions. These elements can significantly change how the fish appear in the sonar camera output and may confuse the computer model. When implemented in new rivers with no prior data collection, like the Klamath, the system’s performance deteriorates, leading to a notable increase in the margin of error to 15-20 percent.
The researchers devised an automatic adaptation algorithm within the system to address this challenge and create a scalable solution that can be deployed to any site without human involvement. This self-calibrating technology functions to automatically adjust to the new conditions and environment, enabling accurate counts of migrating fish. In tests, the automatic adaptation algorithm was capable of reducing the counting error to the range of 10 to 15 percent. The enhancement in counting accuracy due to the self-calibrating feature is significant.means that the technology is nearer to being implementable in new areas with minimal extra human involvement.
Facilitating immediate management with the “Fishbox”
Another obstacle encountered by the research group was the creation of an effective data framework. To operate the computer vision system, the footage generated by sonar cameras must be transmitted through the cloud or by physically mailing hard drives from a river site to the laboratory. These approaches have significant limitations: a cloud-based method is hindered by inadequate internet access in secluded river locations, while transporting the data introduces delays.
Rather than depending on these techniques, the team has introduced an energy-efficient computer, termed the “Fishbox,” which can be utilized in the field for processing tasks. The Fishbox comprises a compact, lightweight computer equipped with optimized software, allowing fishery managers to connect it to their current laptops and sonar cameras. Consequently, the system can implement salmon counting models directly at the sonar locations without necessitating internet access. This empowers managers to make decisions on an hourly basis, fostering more agile, immediate management of salmon populations.
Community engagement
The team is also striving to foster a community focused on monitoring salmon fisheries management in the Pacific Northwest. “It’s remarkably thrilling to have stakeholders who are keen to access [our technology] as we refine it and to establish a tighter integration and collaboration with them,” remarks Beery. “Especially when working on food and water systems, direct collaboration is essential to facilitate impact, ensuring that what you create genuinely meets the needs of the individuals and organizations you are intended to assist.”
In June of this year, Beery’s lab arranged a workshop in Seattle that brought together non-profit organizations, tribes, and state and federal fish and wildlife departments to converse about utilizing automated sonar systems for monitoring and managing salmon populations. Kay points out that the workshop was an “excellent opportunity for everyone to share various methods of using sonar and to explore how the automated processes we’re developing could integrate into that workflow.” The conversation continues through a shared Slack channel established by the team, engaging over 50 participants. Bringing this group together is a notable accomplishment, as many of these organizations would have otherwise lacked the chance to collaborate.
Looking ahead
As the team persistently refines the computer vision system, hones their technology, and interacts with a broad range of stakeholders—from Indigenous communities to fishery managers—the project is on the brink of making remarkable advancements in the efficiency and precision of salmon monitoring and management in the area. And as Beery progresses the efforts of her MIT group, the J-WAFS seed grant is aiding in keeping issues like fisheries management within her focus.
“The existence of the J-WAFS seed grant here at MIT has allowed us to continue pursuing this project since our relocation,” Beery notes, adding, “it also broadened the project’s scope and helped us sustain active collaboration on what I believe to be a highly significant and impactful project.”
As J-WAFS celebrates its 10th anniversary this year, the initiative seeks to persist in supporting and motivating MIT faculty to engage in pioneering projects that aim to enhance understanding and develop practical solutions addressing global challenges in water and food systems.