New Technologies for Environmental Impact Assessments in the Deep Sea

Current Activities:

• Standardisation: work continues of the development of an ISO standard for Marine environment impact assessment (MEIA) – performance specifications for in situ image-based surveys in deep seafloor environments.
• New technologies:
  • Work continues on the development of software to bridge the gap between high quality video data collected in situ and currently available cloud-based image annotation tools, and on easily implementable protocols for meta-barcoding analyses of deep sea meiofauna. See Past Activities below for related fieldwork.
  • Work is ongoing to develop tools for the quantification of microplastics in sediments (see Publications below in Past Activities)
• A sub-group of DOSI members, led by Dr Amy Baco from Florida State University, are working on: “A community consensus on designating Vulnerable Marine Ecosystems (VMEs) from imagery”. Under UNGA resolution 61/105, management of fisheries in areas beyond national jurisdiction requires identification of vulnerable marine ecosystems (VMEs). Criteria to designate a VME include uniqueness, functional significance, fragility, structural complexity, and certain life history traits. Currently the only quantitative way to assess VME locations is to use fisheries trawl bycatch data. Besides potentially destroying the VME in gathering these data and method caveats, the threshold for designating a VME from trawls varies among FAO regions. Imagery data from scientific surveys is a less destructive approach, however there currently is not a framework for designating VMEs from images. Thus, the goal of this project is to bring together a large international team to establish first pass consensus guidelines across regions for designating VMEs from images.

Past Activities (in reverse chronological order):

Meetings:

• 19 July 2023, OpenVision AI marine scientist working group meeting.
• 17 May 2023, OpenVision AI marine scientist working group meeting.
• 23 February 2023, Fathomnet workshop.
• 22 September 2022, Fathomnet taxonomy working group meeting.
• 29 June 2022, Advisory panel meeting with Japan Organization for Metals and Energy Security (JOGMEC) regarding Environmental Impact Assessments of Deep Sea mining (manganese crust).
• 1-2 April 2022, Fathomnet workshop.

Fieldwork:

• 7-8 September 2023, cruise on the T/V Hokuto in Suruga Bay to further test the shadowgraphy-based camera system prototype and collect training data for machine learning pipelines.
• 22-23 August 2023, cruise on the T/V Hokuto in Suruga Bay to further test the shadowgraphy-based camera system prototype and collect training data for machine learning pipelines.
• 5-6 July 2023, cruise on the T/V Hokuto in Suruga Bay to further test the shadowgraphy-based camera system prototype and collect training data for machine learning pipelines.
• 26-28 June 2023, cruise on the T/V Hokuto in Suruga Bay to further test the shadowgraphy-based camera system prototype and collect training data for machine learning pipelines.
• 20 May – 11 June 2023, cruise on the R/V Mirai (MR23-03) was carried out. The updated/optimized shadowgraphy-based camera system prototype was tested during CTD rosette casts in Suruga Bay and the Myojin caldera to quantify the marine snow and turbidity field.
• 30 April – 13 May 2023, ROV cruise with King Abdullah University of Science and Technology (KAUST), Saudia Arabia. Single-housing stereo 4K camera system deployed on KAUST ROV.  Real-time video capture and organism annotation system deployed and tested. Targeted deep ocean surveys of the already warm Red Sea as a proxy for a warming global ocean.
• 18 – 26 March 2023, customization of realtime data logging and video annotation system for KAUST ROV, King Abdullah University of Science and Technology (KAUST), Saudia Arabia.
• 3 September – 20 December 2022 cruise on the R/V Kaimei (cruise KM22-15) was carried out to further develop a science payload for submersible platforms of opportunity. During this cruise a single-housing stereo 4K camera system with battery housing was tested for 3D reconstructions of the seafloor and quantification, including sizing, of fish and macroplankton. A newly developed shadowgraphy-based camera system prototype was also tested during CTD rosette casts to quantify the marine snow and turbidity field.
• 1 February – 14 February 2022, ROV cruise with King Abdullah University of Science and Technology (KAUST), Saudia Arabia. Single-housing stereo 4K camera system deployed on KAUST ROV. DOSI WG members involved remotely. Targeted deep ocean surveys of the already warm Red Sea as a proxy for a warming global ocean.
• 11 November-19 December 2021, second Environmental Impact Assessment cruise to Clarion-Clipperton Zone for Deep Green/The Metals Company to do world’s first EIA of midwater. Several new technologies developed through the work of DOSI’s Technology Working Group were employed.
• 22 October – 2 November 2021 cruise on the R/V Kaimei (cruise KM21-08) was carried out to further develop a science payload for submersible platforms of opportunity. During this cruise a single-housing stereo 2.7K camera system was tested for 3D reconstructions of the seafloor and quantification, including sizing, of fish and macroplankton.
• 11 March-22 April 2021, Environmental Impact Assessment cruise to Clarion-Clipperton Zone for Deep Green/The Metals Company to do world’s first EIA of midwater. Several new technologies developed through the work of DOSI’s Technology Working Group were employed. Lindsay took part remotely due to Covid19.
• 12-25 December 2020 cruise on R/V Kaimei to test the following equipment:
  •   – RamaCam – A combined holographic camera and Raman spectrometer for measuring particle morphologies and constituents with a view to detecting, characterizing and quantifying suspended microplastic particles in the deep ocean’s midwaters.
  •   – a 3D shadowgraph camera system to produce image-based machine learning training sets of deep ocean plankton
• 22-28 February 2020 cruise on the R/V Yokosuka (cruise YK20-E02) was carried out to further develop a science payload for submersible platforms of opportunity, incorporating an 8K video camera, two HD cameras, two NTSC cameras, a line laser, three pairs of LED stick lights, CTDs, altimeter, and compass. On board testing, further development and optimization of a video annotation system incorporating Aphia IDs for taxonomy was also done during this cruise.
• May/June 2017: New technologies for surveying vent plumes and larval distributions around areas targeted for Deep Sea mining were tested during an expedition to the Okinawa Trough, Japan. A holographic camera capable of capturing nine holograms per second was deployed in combination with a re-engineered digital Continuous Plankton Recorder, in order to capture the fine-scale distributions of larvae around vents to help in efforts to model larval dispersal between sites and assess connectivity. These technologies were deployed on a towed camera array with an Autonomous Visual Plankton Recorder (VPR) and a prototype 4K VPR imaging the same water parcel.
• Boreal summer 2016: An intercalibration experiment comparing the datasets collected with an Autonomous Visual Plankton Recorder (VPR), an Underwater Video Profiler (UVP) and a GoPro in a pressure housing with controlled lighting was carried out during the Hidden Ocean cruise to the Arctic Ocean

Publications:

• Sangekar, M., Friedman, A., Hidaka, M., Hosono, T., Lindsay. D. (2023) SquidJam: A Video Annotation Ecosystem. 2023 IEEE Underwater Technology (UT23) 1-5. DOI: 10.1109/UT49729.2023.10103435
• Takahashi, T., Liu, Z., Thevar, T., Burns, N., Sangekar, M., Lindsay. D., Watson, J., Thornton, B. (2023) RamaCam: autonomous in-situ monitoring system of marine particles by combining holography and Raman spectroscopy. 2023 IEEE Underwater Technology (UT23) 1-5. DOI: 10.1109/UT49729.2023.10103388
• Liu, Z., Giering, S., Takahashi, T., Thevar, T., Takeuchi, M., Burns, N., Thornton, B., Watson, J., Lindsay. D. (2023) Advanced subsea imaging technique of digital holography: in situ measurement of marine microscale plankton and particles. 2023 IEEE Underwater Technology (UT23) 1-9. DOI: 10.1109/UT49729.2023.10103440
• Mofokeng, R.P., Faltynkova, A., Alfonso, M.B., Boujmil, I., de Carvalho, I., Lunzalu, K., Zanuri, N.B.M., Nyadjro, E.S., Puskic, P.S., Lindsay, D.J., Willis, K., Adyel, T.M., Serra-Gonçalves, C., Zolich, A., Eriksen, T.S., Evans, H-C., Gabriel, D., Hajbane, S., Suaria, G., Law, K.L., Lobelle, D.  (2023) The future of ocean plastics: designing diverse collaboration frameworks. ICES Journal of Marine Science  1-12. DOI: 10.1093/icesjms/fsad055
• Takahashi, T., Liu, Z., Thevar, T., Burns, N., Lindsay. D., Watson, J., Mahajan, S., Yukioka, S., Tanaka, S., Nagai, Y., Thornton, B. (2023) Multimodal Image and Spectral Feature Learning for Efficient Analysis of Water-Suspended Particles. Optics Express 31(5): 7492-7504. doi.org/10.1364/OE.47087
• Lindsay, D.J. (2022) Imaging surveys of pelagic communities. in Report of the workshop on enhancing image-based biodiversity assessments to advance deep-sea taxonomy. International Seabed Authority, 32. ISBN 978-976-8313-22-5
• Irisson, J-O., Ayata, S-D., Lindsay, D.J., Karp-Boss, L. and Stemmann, L. (2022) Machine Learning for the Study of Plankton and Marine Snow from Images. Annual Review of Marine Science 14: 277-301. doi.org/10.1146/annurev-marine-041921-013023
• Liu, Z., Thevar, T., Takahashi, T., Burns, N., Yamada, T., Sangekar, M., Lindsay, D., Watson, J. and Thornton, B. (2021) Unsupervised feature learning and clustering of particles imaged in raw holograms using an autoencoder. Journal of the Optical Society of America A 38(10): 1-11. doi.org/10.1364/JOSAA.424271
• Verhaegen, G., Cimoli, E., & Lindsay, D. J. (2021). Life beneath the ice: jellyfish and ctenophores from the Ross Sea, Antarctica, with an image-based training set for machine learning. Biodiversity Data Journal, 9, e69374. https://doi.org/10.3897/BDJ.9.e69374
• Liu, Z., Takahashi, T., Lindsay. D., Thevar, T., Sangekar, M., Watanabe, H.K., Burns, N., Watson, J. and Thornton, B. (2021) Digital in-line holography for large-volume analysis of vertical motion of microscale marine plankton and other particles. IEEE Journal of Oceanic Engineering, 46(4): 1248-1260, 2020JOE003166. Doi: 10.1109/JOE.2021.3066788
• Kitahashi, T., Nakajima, R., Nomaki, H., Tsuchiya, M., Yabuki, A., Yamaguchi, S., Zhu, C., Kanaya, Y., Lindsay, D.J., Chiba, S., and Fujikura, K. (2021) Development of robust models for rapid classification of microplastic polymer types based on infrared hyperspectral images. Analytical Methods 13: 2215-2222. DOI: 10.1039/D1AY00110H
• Liu, Z., Takahashi, T., Thevar, T., Lindsay. D., Burns, N., Watson, J. and Thornton, B. (2020) Comparison of Region Proposal Methods for Marine Holograms. Global Oceans 2020: Singapore – U.S. Gulf Coast, pp. 1-5. DOI: 10.1109/IEEECONF38699.2020.9389216

July 2021: The following were published as international standards by ISO committee (TC8/SC13), through a working group DL was a member of, and are expected to be used for environmental impact assessments for seabed mining.

ISO 23731: Long-term in-situ imaged-based surveys in deep-sea environments

This is a standard for procedures and settings for long-term in-situ camera observations with subsea instruments. This standard is applicable to both submersible probes and seabed-mounted instruments, and can also be applied to video observation methods using the platform like Japan’s domestically produced “Edokko Mark 1″* series. By standardizing the observation method, we were able to facilitate comparative studies with observation results from different locations. Available here.

ISO23732: General protocol for observation of meiofaunal community

This is a standard for efficient procedures to determine the abundance, morphology and community composition of small benthic organisms living in marine sediments. In conventional survey methods, individuals are sorted by hand while being observed under a microscope. This standard establishes a rapid and efficient survey procedure by combining sample preparation with FlowCam (particle image analyzer) and metagenomics. Available here.

• A chapter on new techniques for standardization of Environmental Impact Assessments was published:
  • Furushima, Y., Yamakita, T., Miwa, T., Lindsay, D., Fukushima, T., Shirayama, Y. (2019) New Techniques for Standardization of Environmental Impact Assessment. in Environmental Issues of Deep-Sea Mining (ed. Sharma R), pages 275-313. ISBN: 978-3-030-12695-7, DOI:10.1007/978-3-030-12696-4_11
• A review on estimating particle fluxes using optical instruments was also published. Such instruments and methodology are valuable tools for Environmental Impact Assessments (EIAs) in the deep sea, particularly with respect to sediment discharge in the midwater during mining operations:
  • Giering, S.L.C., Cavan, E.L., Basedow, S.L., Briggs, N., Burd, A.B., Darroch, L., Guidi, L., Irisson, J-O., Iversen, M.H., Kiko, R., Lindsay, D.J., Marcolin, C.R., McDonnell, A.MP., Möller, K.O., Passow, U., Trull, T. and Waite A.M. (2020) Sinking organic particles in the ocean – flux estimates from in situ optical devices. Frontiers in Marine Science 6: 834. doi: 10.3389/fmars.2019.00834
• High-resolution holographic imaging has been combined with the chemical information available from Raman spectroscopy to create a sensing method that could one day be used to detect, identify and measure the distribution of particles found in the ocean in situ without any sample preparation or manipulation. The demonstrated approach, published in the journal Applied Optics, paves the way for long term, global scale analysis of marine particulates and microplastics using large networks of observation platforms, such as floats or gliders. The technique would also be invaluable for carrying out Environmental Impact Assessments (EIAs) in the deep sea, particularly with respect to sediment discharge in the midwater during mining operations:
  • Takahashi, T., Liu, Z., Thevar, T., Burns, N., Mahajan, S., Lindsay. D., Watson, J. and Thornton, B. (2020) Identification of microplastics in a large volume by integrated holography and Raman spectroscopy. Applied Optics 59(17): 5073-5078. doi: 10.1364/AO.393643
  • Read at Phys.org here
• Work has also been ongoing on developing tools for the quantification of microplastics in sediments.
  • Nakajima, R., Tsuchiya, M., Lindsay, D.J., Kitahashi, T., Fujikura, K., Fukushima, T. (2019) A new small device made of glass for separating microplastics from marine and freshwater sediments. PeerJ 10/2019; 7(19):e7915, DOI:10.7717/peerj.7915
  • Nakajima, R., Lindsay, D.J., Tsuchiya, M., Matsui, R., Kitahashi, T., Fujikura, K., Fukushima, T. (2019) A small, stainless-steel sieve optimized for laboratory beaker-based extraction of microplastics from environmental samples. MethodsX 07/2019; 6, DOI:10.1016/j.mex.2019.07.012
• Publications on the dissemination of technologies for EIAs in the deep sea:
  • Lindsay DJ, Hidaka M, Watanabe HK, Yahagi T, Grossmann MM, Mori M, Yamamoto H. Baseline data on planktonic communities inside and outside submarine calderas: survey methods and preliminary results. 2019 KIOST International Symposium on Hydrothermal Ecosystem and Indian Ocean. Busan, Korea, 4 November 2019.
  • Lindsay DJ. State of the art in marine monitoring technologies: high-performance imaging surveys. France-Japan Workshop on Subsea Observatory in the South Pacific and its surrounding ocean: scientific frontiers and technical challenges, Noumea, New Caledonia, 19 September 2019. (Oral)

Workshops:

• 19 September 2023, Takahashi, T., Liu, Z., Bourdakos, K., Thevar, T., Burns, N., Lindsay. D., Thornton, B., Mahajan, S. (2022) Fast classification of microplastics by hyphenated-Raman techniques. 7th International Marine Debris Conference (7IMDC), Busan, Republic of Korea, Oral.
• 21 March 2023, Lindsay, D.J., Sangekar, M.N. (2023) Remotely Operated Vehicle Survey System Design and Development: from Alpha to Omega. ROV Development Workshop, KAUST, Saudi Arabia, Invited Speaker.
• 21 March 2023, Sangekar, M.N., Lindsay, D.J.  (2023) Development of Tools for Science in the ROV Ecosystem. ROV Development Workshop, KAUST, Saudi Arabia, Invited Speaker.
• 29 November 2022, Watanabe, H., Lindsay, D.J. (2022) Image-based analysis of plankton/meiofauna collections. Inception workshop of the Sustainable Seabed Knowledge Initiative, Busan, Republic of Korea, Oral.
• 08 March 2023, Sangekar, M., Friedman, A., Hidaka, M., Hosono, T., Lindsay, D.J. (2023) SquidJam: A video annotation ecosystem. International Symposium on Underwater Technology (UT23), Tokyo, Japan, Oral. Holography: In-situ Measurement of Marine Particles and Data Processing. International Symposium on Underwater Technology (UT23), Tokyo, Japan. Presentation.
• 08 March 2023, Liu, Z., Giering, S., Takahashi, T., Thevar, T., Takeuchi, M., Burns, N., Thornton, B., Watson, J., Lindsay. D. (2023) Advanced Submersible Imaging Technique of Digital Holography: In-situ Measurement of Marine Particles and Data Processing. International Symposium on Underwater Technology (UT23), Tokyo, Japan, Oral.
• 03 October 2022, Sangekar, M., Friedman, A., Hidaka, M., Hosono, T., Lindsay, D.J. (2022) RESTful API plugins for SquidJam – a video annotation ecosystem for multi-camera datasets. Fourth Marine Imaging Workshop. Brest, France, Oral.
• 03 October 2022, Lindsay, D.J., Friedman, A., Sangekar, M., Bergman, L.A., Montenegro, J., Bachtel, T., Seid, C., Thuesen, E., Hidaka, M., Hosono, T.,  Drazen, J. (2022) SquidJam – a video annotation tool for the analysis of challenging video datasets: A case study on a midwater Environmental Impact Assessment in the Clarion-Clipperton Zone. Fourth Marine Imaging Workshop. Brest, France, Poster.
• 03 October 2022, Friedman, A., Monk, J., Williams, S., Pizarro, O., Lindsay, D.J. (2022) SQUIDLE+: online tool for managing, exploring, annotating and understanding marine imagery. Fourth Marine Imaging Workshop. Brest, France, Poster.
• 12 October 2021, Lindsay DJ. Imaging surveys of pelagic communities – Image acquisition: needs and requirements for biological image surveys and laboratory studies. International Seabed Authority Workshop on Enhancing Image-based Biodiversity Assessments to Advance Deep-Sea Taxonomy.
• 14 September 2021, Lindsay DJ, Hidaka M, Sangekar M, Hosono T, Matsuoka D, Sugiyama D, Friedman A, Hopcroft R. Deep Learning and Deep Arctic Jellies: a case study pipelining real-time ROV video annotations, post-cruise analysis, data dissemination and production of machine-learning training sets. 16th Deep Sea Biology Symposium. Brest, France. Poster.
• 16 September 2021, Sangekar M, Schroder S-M, Kiko R, Hidaka M, Watanabe H, Irisson J-O, Lindsay D. Dealing with Diversity: Data-driven unsupervised clustering on diverse deep-sea image collections. 16th Deep Sea Biology Symposium. Brest, France.

7 July 2021, Japan-India Workshop including discussions on technology sharing for plastic pollution surveys from coast to deep ocean.

• 3 March, 7 April, 14 May, 3 June, 9 July, Plankton image sorting online workshops
• 17 February 2021, France-Japan workshop on Deep Sea Observatory for South Pacific to scope applicable technologies.
• 22-24 February 2021, Saudi Arabia-Japan workshop including discussions on technology sharing for deep ocean surveys of the already warm Red Sea as a proxy for a warming global ocean.
• 2nd EcoDeep Workshop, March 2017, Tokyo, Japan.
• Identifying technologies needed for Midwater Exploration and surveys, including vent plume work, were discussed during a workshop at the Ocean Sciences meeting in Hawaii.
• Technologies for imaged-based, environmentally friendly surveys were discussed during the Marine Imaging Workshop in Kiel, Germany in February 2017. http://marine-imaging-workshop.com/
• A first workshop aiming to identify and review current and emerging technologies relating to underwater image acquisition, real-time and post-cruise image annotation, data sharing and archiving was convened in November 2015 in JAMSTEC and was funded by JAMSTEC, the Schmidt Ocean Institute and the Australian Embassy in Tokyo. See DOSI New Technologies WG Marine Imaging Workshop Report – December 2015 for further detail.
• Outcomes of the DOSI workshop were presented at the Marine Video Workshop, March 2016, Rhode Island, USA.