Skip to main content

Open Soil Spectral Library

About

Access source code

DOI

“Man’s most human characteristic is not his ability to learn, which he shares with many other species, but his ability to teach and store what others have developed and taught him.” Margaret Mead, Culture and Commitment: The New Relationships Between the Generations in the 1970s

Soil Spectroscopy for Global Good

SoilSpec4GG is a USDA-funded Food and Agriculture Cyberinformatics Tools Coordinated Innovation Network NIFA Award #2020-67021-32467 project. It brings together soil scientists, spectroscopists, informaticians, data scientists and software engineers to overcome some of the current bottlenecks preventing wider and more efficient use of soil spectroscopy. A series of working groups will be formed to address topics including calibration transfer, model choice, outreach & demonstration, and use of spectroscopy to inform global carbon cycle modeling. For more info refer to: https://soilspectroscopy.org/.

R tutorials and software developed to implement OSSL is available via: https://github.com/soilspectroscopy.

Soil spectroscopy for global good project works with other global initiatives including the FAO Global Soil Partnership and the IEEE P4005 Standards and Protocols for Soil Spectroscopy Working Group.

What is soil spectroscopy?

Soil spectroscopy is the measurement of light absorption when light in the visible, near infrared or mid infrared (Vis–NIR–MIR) regions of the electromagnetic spectrum is applied to a soil surface. The proportion of the incident radiation reflected by soil is sensed through Vis–NIR–MIR reflectance spectroscopy. These characteristic spectra (see Fig. below) can then be used to estimate numerous soil attributes including: particle size distribution, mineral and organic compounds, and water.

Schematic explanation of the soil spectroscopy. For more info see: https://soilspectroscopy.org/.

Figure 0.1: Schematic explanation of the soil spectroscopy. For more info see: https://soilspectroscopy.org/.

Example of spectral signatures for large number of VisNIR scans (KSSL).

Figure 0.2: Example of spectral signatures for large number of VisNIR scans (KSSL).

Open Soil Spectral Library

DOI

The Open Soil Spectral Library (OSSL) is a suite of datasets, web-services, software, and tutorials. It includes (see also https://github.com/soilspectroscopy):

MIR spectral diversity

MIR locations

VisNIR locations

Importing new datasets to OSSL

The OSSL database has been prepared following the OSSL schema which is available at ossl-imports.

The github repository ossl-imports has all the importing codes for preparing and integrating new datasets into the OSSL. The folder ossl-imports/dataset contains all the datasets separated by its name/code, with an internal README.Rmd file describing in detail the steps necessary for preparing and standardizing the files into the OSSL format.

The original files are placed on an internal server to avoid the storage of big files in the github repository, but some of them can be found on persistent online repositories if they have a public license. Other original datasets that do not have a public license or are shared to this project with some restrictions will no be shared publicly. As we keep the original files in a local repository to run the importing notebooks (each ossl-imports/dataset/../README.Rmd), this operation can only be run at our internal server.

New contributors are encouraged to host their files on a public repository and draft the importing step on github through pull requests. The final checks and consolidation, however, will be concluded on the internal server.

The README.Rmd files follow a basic structure. Each one has a description of the dataset at the top of the notebook which links to the dataset source and references. An extended description is provided in this book.

For preparing the import, the following subsections are defined:

  • Basic description.
  • Soil site information.
  • Soil laboratory (wet chemistry) data
  • Mid-infrared (MIR) spectroscopy data (optional).
  • Visible-near-infrared (VisNIR) spectroscopy data (optional).
  • Quality control.
  • References.

Each subsection has its own reading and processing functions, and the outputs can be joined using shared id columns. The output files share the same name and pattern but can be retrieved with different folder names, which are binded together in a final stage. The outputs are named as ossl_soilsite_<version>.qs, ossl_soillab_<version>.qs, ossl_mir_<version>.qs, ossl_visnir_<version>.qs. The version number is adjusted for recurring updates.

Soil properties of interest

The contrasting methods used for analytically determining (wet chemistry) a given soil property has been a subject of internal discussion in this project. Some global initiatives have been facing this same issue in their soil databases but there still no clear or full consensus on how to harmonize those different methods. This has been a topic of great discussion and research development at the Global Soil Partnership’s Global Soil Laboratory Network (GLOSOLAN).

In order to maximize transparency, for now, we have decided to produce two different levels for the OSSL database. Level 0 takes into account the original methods employed in each dataset but tries to initially fit them to two reference lists: KSSL Guidance – Laboratory Methods and Manuals and ISO standards. A copy of the KSSL procedures and coding scheme is archived in ossl-imports.

If a reference method does not fall in any previous method, then we create a new variable sharing at least a common property and unit. A final harmonization takes place in the OSSL Level 1, where those common properties sharing different methods are converted to a target method using some publicly available transformation rule, or in the worst scenario, they are naively binded or kept separated to produce its specific model. All the implementations are documented in the ossl-import/ossl_level0_to_level1_soillab_harmonization.csv repository.

In addition, GLOSOLAN’s Standard Operating Procedures (SOPs) list four groups of soil variables of interest to international soil spectroscopy projects:

Soil chemical variables:

  • pH,
  • Carbon,
  • Phosphorous,
  • Potassium,
  • Nitrogen,
  • Exchangeable cations and CEC,
  • Extractable microelements,
  • Trace and major element analyses,
  • Gypsum,
  • Electrical conductivity and total soluble salt content,
  • Soluble sulfate and chloride analysis,
  • Special analysis for peats, mineral and organic soils, agriculture and forest,

Soil physical variables:

  • Bulk density,
  • Coarse fragments,
  • Particle-size distribution,
  • Water retention curve,
  • Porosity,
  • Hydraulic conductivity function,
  • Aggregate stability,
  • Moisture content,

Soil biological variables:

  • Microbial biomass,
  • Soil Respiration,
  • Enzyme activity,
  • Microbial identification,

Soil contaminants:

  • Heavy metal elements: As, Hg, Cu, Cd, Pb and similar,
  • Other soil pollutants,

Contributing data

We encourage public and private entities to help this project and share SSL data. The following four modes of data sharing are especially encouraged:

  1. Publish your data open acces by releasing it under a Creative Commons license (CC-BY, CC-BY-SA)
    or the Open Data Commons Open Database License (ODbL). This data can then directly imported into the OSSL.
  2. Donate a small part (e.g. 5%) of your data (release under CC-BY, CC-BY-SA and/or ODbL). This data can then be directly imported into the OSSL.
  3. Allow SoilSpectroscopy.org project direct access to your data so that we can perform data mining and then release ONLY results under some Open Data license.
  4. Use OSSL data to produce new derivative products, then share them through own infrastructures OR contact us for providing hosting support.

We can sign professional Data Sharing Agreements with data producers that specify in detail how will the data will be used. Our primary interest is in enabling research, sharing and use of models (calibration and prediction) and collaboration of groups across borders.

We take especial care that your data is secured, encrypted where necessary, and kept safely, closely following our privacy policy and terms of use.

Contributing documentation

Please feel free to contribute to this technical documentation. Check the GitHub repository for more detailed instructions.

Information outdated or missing? Please open an issue or best do a correction in the text and then make a pull request.

Contributors

If you’ve contributed to this manual, add your name, Twitter handle, ORCID or blog link below:

Jonathan Sanderman, Tomislav Hengl, Katherine Todd-Brown, Leandro L. Parente, Wanderson de Sousa Mendes, Dellena Bloom, José Lucas Safanelli, Henning Teickner.

Acknowledgments

Open Soil Spectral Library was possible due to the contributions by public and private organizations. Listed based on the date of import:

We are also grateful to Wanderson de Sousa Mendes for the help with initial screening of the data and the development of the initial R code for processing soil spectroscopy data.

For more advanced uses of the soil spectral libraries we advise to contact the original data producers especially to get help with using, extending and improving the original SSL data.

We are also grateful to USDA National Institute of Food and Agriculture #2020-67021-32467 for providing funding for this project.

Disclaimer

Whilst utmost care has been taken by the Soil Spectroscopy project and data authors while collecting and compiling the data, the data is provided “as is”. Woodwell Climate Research Center, University of Florida, OpenGeoHub foundation and its suppliers and licensors hereby disclaim all warranties of any kind, express or implied, including, without limitation, the warranties of merchantability, fitness for a particular purpose and non-infringement. Neither Woodwell Climate Research Center, University of Florida, OpenGeoHub foundation nor its suppliers and licensors, makes any warranty that the Website will be error free or that access thereto will be continuous or uninterrupted. You understand that you download from, or otherwise obtain content or services through, the Website at your own discretion and risk.

In no event shall the data authors, the Soil Spectroscopy project, or relevant funding agencies be liable for any actual, incidental or consequential damages arising from use of the data. By using the Soil Spectroscopy project data, the user expressly acknowledges that the Data may contain some nonconformities, defects, or errors. No warranty is given that the data will meet the user’s needs or expectations or that all nonconformities, defects, or errors can or will be corrected. The user should always verify actual data; therefore the user bears all responsibility in determining whether the data is fit for the user’s intended use.

This document is under construction. If you notice an error or outdated information, please submit a correction/pull request or open an issue.

This is a community project. No profits are being made from building and serving Open Spectral Library. If you would like to become a sponsor of the project, please contact us via: https://soilspectroscopy.org/contact/.

License

This website/book and attached software is free to use, and is licensed under the MIT License. The OSSL training data and models, if not otherwise indicated, are available either under the Creative Commons Attribution 4.0 International CC-BY and/or CC-BY-SA license / Open Data Commons Open Database License (ODbL) v1.0.

Suggested literature

Some other connected publications and initiatives describing collation, import and use of soil spectroscopy data: