6 Spectra model prediction

We used only the raw spectra and three spectra transformations from the 15 spectra produced in the FTIR part. This transformation are improving the quality of the prediction model (Ludwig et al. 2023). In total we had four spectra:

  • Raw spectra.
  • SG-2-11 the Savitzky-Golay with a polynomial order of 2 and a window size of 11.
  • Moving average of 11. -SNV-SG standard normal variate transformation on the Savitzky-Golay with a polynomial order of 2 and a window size of 11.

6.1 Model implementation

We implemented the prediction model in Python language. A sanity check was performed to assess the conformity of the data set Sanity_Check_code.

Density plot of pH
Density plot of pH

We split the data into an 85% training set and a 15% testing set.

As EC values had outliers we removed all samples with values higher than 400 µS/cm: 53802 (827 µS/cm), 53871 (857 µS/cm), 53872 (932 µS/cm), 53902 (718 µS/cm), 53938 (447 µS/cm), 53954 (426 µS/cm) and 55852 (636 µS/cm).

All the properties were predicted using the Cubist model under Python environment Cubist_Kfold_code. A 5 time stratified k-fold was used with a total of 9 combinations for the hyperparameter. The stratified k-fold helps have a higher diversity of sampling in the training and testing set.

Table 6.1: Hyperparamters of the Cubist model.
n_rules n_committees
20 5
30 5
40 5
20 10
30 10
40 10
20 15
30 15
40 15

6.2 Model evaluation

6.2.1 All spectra tranformations metrics

(#tab:df raw show)Cubist model evaluation metrics on raw spectra.
Metric ME R2 RMSE RPIQ
pH -0.1130862 0.4401298 0.1479898 1.2423942
CaCO3 -1.3252205 0.8882458 4.1193275 2.3192395
Nt -0.0210327 0.6229612 0.0543926 1.4595850
Ct -0.2208946 0.8913525 0.5595367 2.3885632
Corg -0.0612059 0.5264148 0.6699846 0.9218613
EC -30.6617828 0.1141583 66.3535662 0.9523699
Sand -6.3994920 0.7674734 7.3599863 2.3624346
Silt 3.2742514 0.2136148 9.3070178 0.8782774
Clay 3.5598760 0.6930271 7.9695160 2.3912120
MWD -0.0183279 0.4745772 0.0539841 1.2472350

\[\\[1.5cm]\]

(#tab:df moving show)Cubist model evaluation metrics on mooving average 11 transformed spectra.
Metric ME R2 RMSE RPIQ
pH -0.0822356 0.4621653 0.1450483 1.221859
CaCO3 -0.0115416 0.8820737 4.2315510 2.389729
Nt -0.0207909 0.6334873 0.0536280 1.410630
Ct -0.0333944 0.9101929 0.5087145 2.228431
Corg -0.0276244 0.5112682 0.6806143 1.087405
EC -16.8330812 0.1914366 63.3932824 1.114983
Sand -6.9387918 0.7336180 7.8775860 2.300951
Silt 0.4148447 0.1086026 9.9089712 0.875066
Clay 6.7544251 0.7609876 7.0322156 2.343124
MWD -0.0284105 0.4724332 0.0540942 1.331620

\[\\[1.5cm]\]

(#tab:df SG show)Cubist model evaluation metrics on SG transformed spectra.
Metric ME R2 RMSE RPIQ
pH 0.0139322 0.4081296 0.1521604 0.9712110
CaCO3 -0.0466458 0.9012224 3.8727870 2.7690478
Nt 0.0136756 0.3532399 0.0712392 0.4787131
Ct -0.0795040 0.8415988 0.6756130 1.8099295
Corg -0.0751953 0.2343362 0.8518925 0.3591207
EC 37.7198879 0.0463005 68.8481029 0.9539687
Sand -2.6170488 0.7616948 7.4508777 2.4868337
Silt -4.7215761 0.3247059 8.6246080 0.9129198
Clay -3.5665841 0.7175530 7.6445239 2.4276119
MWD 0.0299509 0.4394541 0.0557593 1.1867736

\[\\[1.5cm]\]

(#tab:df SNV show)Cubist model evaluation metrics on SNV-SG transformed spectra.
Metric ME R2 RMSE RPIQ
pH -0.1487360 0.3990151 0.1533275 0.8891613
CaCO3 -1.1519722 0.8888660 4.1078803 2.7618729
Nt -0.0046014 0.8257789 0.0369741 1.7119021
Ct 0.0367057 0.9123964 0.5024348 2.7212452
Corg 0.1736385 0.6126784 0.6059009 1.1525632
EC -5.9358389 0.2942953 59.2239976 0.9349021
Sand 2.0420108 0.8106039 6.6424182 2.6565763
Silt 0.1929490 0.3783928 8.2746742 0.9772474
Clay -1.7827777 0.6928479 7.9718426 2.2699483
MWD -0.0022866 0.4592173 0.0547675 1.1640412

6.2.2 Selected transformed spectra

Regarding the results of the predictions, we selected for each variables the following spectra:

  • pH Raw spectra.
  • CaCO\(_3\) SG.2.11 spectra.
  • Nt SNV-SG spectra.
  • Ct SNV-SG spectra.
  • SOC SNV-SG spectra.
  • EC SNV-SG spectra.
  • Sand SNV-SG spectra.
  • Silt SNV-SG spectra.
  • Clay SG.2.11 spectra.
  • MWD Raw spectra.

## Warning: Removed 39 rows containing missing values or values outside the
## scale range (`geom_point()`).

6.3 Predicted values

6.3.1 Full predicted values table

The full table of 2022 and 2023 samples predicted is visible below or can be found on the deposit as Mir_spectra_prediction file or at : Bellat, Mathias; Glissmann, Benjamin; Rentschler, Tobias; Sconzo, Paola; Pfälzner, Peter; Brifkany, Bekas; Scholten, Thomas (2024): “Soil properties predicted on mid-infrared (MIR) spectroscopy measurements in North-Western Kurdistan region, Iraq [dataset]”. PANGAEA, https://doi.org/10.1594/PANGAEA.973700

## Rows: 532 Columns: 18
## ── Column specification ────────────────────────────────────────
## Delimiter: ";"
## chr   (1): Event label
## dbl  (16): Lab label, Latitude, Longitude, Depth, top, Depth, bot, Weight, p...
## dttm  (1): Date
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

We removed the sample with ID 55864 which is an outlier.

6.3.2 Boxplot of the predicted values

\[\\[1.5cm]\]

\[\\[1.5cm]\]

\[\\[1.5cm]\]

\[\\[1.5cm]\]

\[\\[1.5cm]\]

6.3.3 Texture of the predicted samples

We ploted the soil texture according to (WRB 2006) classification.

\[\\[1.5cm]\]

\[\\[1.5cm]\]

\[\\[1.5cm]\]

\[\\[1.5cm]\]

\[\\[1.5cm]\]

6.4 Observed vs. predicted values

## Rows: 108 Columns: 29
## ── Column specification ────────────────────────────────────────
## Delimiter: ";"
## chr   (3): Event label, Nt, St
## dbl  (25): Lab label, Latitude, Longitude, Depth, top, Depth, bot, pH, CaCO3...
## dttm  (1): Date/Time
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

## Warning: Removed 7 rows containing non-finite outside the scale range
## (`stat_density()`).

## Warning: Removed 34 rows containing non-finite outside the scale range
## (`stat_density()`).

## Warning: Removed 2 rows containing non-finite outside the scale range
## (`stat_density()`).

## Warning: Removed 2 rows containing non-finite outside the scale range
## (`stat_density()`).

## Warning: Removed 2 rows containing non-finite outside the scale range
## (`stat_density()`).

## Warning: Removed 2 rows containing non-finite outside the scale range
## (`stat_density()`).

6.5 Spatial ploting

6.5.1 0 - 10 cm depth

6.5.2 10 - 30 cm depth

6.5.3 30 - 50 cm depth

6.5.4 50 - 70 cm depth

6.5.5 70 - 100 cm depth

References

Ludwig, Bernard, Isabel Greenberg, Michael Vohland, and Kerstin Michel. 2023. “Optimised Use of Data Fusion and Memory‐based Learning with an Austrian Soil Library for Predictions with Infrared Data.” European Journal of Soil Science 74 (4): e13394. https://doi.org/10.1111/ejss.13394.
WRB, IUSS Working Group. 2006. Guidelines for Soil Description. 4th ed. Rome: Food; Agriculture Organization of the United Nations.