Hyper-spectral sensors take measurements in the narrow contiguous bands across the electromagnetic spectrum. Usually, the goal is to detect a certain object or a component of the medium with unique spectral signatures. In particular, the hyper-spectral measurements are used in atmospheric remote sensing to detect trace gases. To improve the efficiency of hyper-spectral processing algorithms, data reduction methods are applied. This paper outlines the dimensionality reduction techniques in the context of hyper-spectral remote sensing of the atmosphere. The dimensionality reduction excludes redundant information from the data and currently is the integral part of high-performance radiation transfer models. In this survey, it is shown how the principal component analysis can be applied for spectral radiance modelling and retrieval of atmospheric constituents, thereby speeding up the data processing by orders of magnitude. The discussed techniques are generic and can be readily applied for solving atmospheric as well as material science problems. 1. D.G. Loyola, S. Gimeno Garcia, R. Lutz, A. Argyrouli, F. Romahn, R.J.D. Spurr, M. Pedergnana, A. Doicu, V. Molina Garca, and O. Schussler. The operational cloud retrieval algorithms from TROPOMI on board Sentinel?5 precursor. Atmospheric Measurement Techniques, 2018,11(1), pp.409-427.
2. C.A. Lee, S.D. Gasster, A. Plaza, C. Chang, and B. Huang. Recent developments in high performance computing for remote sensing: A review. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 20114(3), pp.508-527.
3. J.P. Veefkind, I. Aben, K. McMullan, H. Forster, J. de Vries, G. Otter, J. Claas, H.J. Eskes, J.F. de Haan, Q. Kleipool, and et al. TROPOMI on the ESA Sentinel?5 Precursor: A GMES mission for global observations of the atmospheric composition for climate, air quality and ozone layer applications. Remote Sensing of Environment, 2012, 120, pp.70-83.
4. Y. Ma, H. Wu, L. Wang, B. Huang, R. Ranjan, A. Zomaya, and W. Jie. Remote sensing big data computing: Challenges and opportunities. Future Generation Computer Systems, 2015, 51, pp.47-60.
5. P. Liu. A survey of remote-sensing big data. Frontiers in Environmental Science, 2015, 3, 45p.
6. V.M. Roozendael, R. Spurr, D. Loyola, C. Lerot, D. Balis, J.C. Lambert, W. Zimmer, J. Gent, J. Van Geffen, M.E. Koukouli, J. Granville, A. Doicu, C. Fayt, and C. Zehner. Sixteen years of GOME/ERS2 total ozone data: the new direct-fitting GOME Data Processor (GDP) Version 5: I. algorithm description. J Geophys Res: Atmospheres, 2012,117: D03305, pp.1-18.
7. V. Natraj. A review of fast radiative transfer techniques. In A.A. Kokhanovsky, editor, Light scattering reviews, volume 8, pages 475-504. Springer Berlin Heidelberg, 2013.
8. D. Efremenko, A. Doicu, D. Loyola, and T. Trautmann. Acceleration techniques for the discrete ordinate method. J Quant Spectrosc Radiat Transfer, 2013,114, pp.73-81.
9. D.S. Efremenko, D. Loyola, A. Doicu, and T. Trautmann. Data-intensive computing in radiative transfer modelling. In P. Soille and P.G. Marchetti, editors, Proc. of the 2016 conference on Big Data from Space (BiDS16), Santa Cruz de Tenerife, Spain, 2016, pp. 188-191.
10. V.P. Budak, G.A. Kaloshin, O.V. Shagalov, and V.S. Zheltov. Numerical modeling of the radiative transfer in a turbid medium using the synthetic iteration. Opt. Express, 23(15): A829, 2015.
11. D.S. Efremenko, D.G. Loyola, A. Doicu, and R.J.D. Spurr. Multi-core-CPU and GPU-accelerated radiative transfer models based on the discrete ordinate method. Computer Physics Communications, 2014, 185(12), pp.3079-3089.
12. D.S. Efremenko, D. Loyola, R.J.D. Spurr, and A. Doicu. Acceleration of radiative transfer model calculations for the retrieval of trace gases under cloudy conditions. J Quant Spectrosc Radiat Transfer, 2014,135, pp.58-65.
13. R.J.D. Spurr. VLIDORT: A linearized pseudospherical vector discrete ordinate radiative transfer code for forward model and retrieval studies in multilayer multiple scattering media. J Quant Spectrosc Radiat Transfer, 2006, 102(2), pp.316-342.
14. R.J.D. Spurr, T.P. Kurosu, and K.V. Chance. A linearized discrete ordinate radiative transfer model for atmospheric remote-sensing retrieval. J Quant Spectrosc Radiat Transfer, 2001, 68(6), pp.689-735.
15. K. Pearson. On lines and planes of closest fit to systems of points in space. Phil Mag, 19012, 6, pp.559-572.
16. R. D. Morris, A. Kottas, M. Taddy, R. Furfaro, and B.D. Ganapol. A statistical framework for the sensitivity analysis of radiative transfer models. IEEE Transactions on Geoscience and Remote Sensing, 2008,46(12), pp.4062-4074.
17. V.A. Ambartzumyan. The effect of the absorption lines on the radiative equilibrium of the outer layers of the stars. Publ. Obs. Astron. Univ. Leningrad, 1936, 6, pp.7-18.
18. R.M. Goody, R. West, L. Chen, and D. Crisp. The correlated k-method for radiation calculations in nonhomogeneous atmosphere. J Quant Spectrosc Radiat Transfer, 1989, 42, 6, pp.539-550. v 19. W.J. Wiscombe and J.W. Evans. Exponential-sum fitting of radiative transmission functions. J Comput Phys, 1997, 24, 4, pp.416-444.
20. R. West, D. Crisp, and L. Chen. Mapping transformations for broadband atmospheric radiation calculation. J Quant Spectrosc Radiat Transfer, 1990, 43,3, pp.191-199.
21. E. Boesche, P. Stammes, R. Preusker, R. Bennartz, W. Knap, and J. Fischer. Polarization of skylight in the O2A band: effects of aerosol properties. Applied Optics, 2008, 47, 19, p. 3467.
22. Christiane Helling and Uffe Grae Jorgensen. Optimizing the opacity sampling method. Astronomy and Astrophysics, 1998, 337, pp.477-486.
23. Frederic Andre, Longfeng Hou, Maxime Roger, and Rodolphe Vaillon. The multispectral gas radiation modeling: A new theoretical framework based on a multidimensional approach to k-distribution methods. Journal of Quantitative Spectroscopy and Radiative Transfer, 2014,147, pp.178-195.
24. Boris A. Fomin. A k-distribution technique for radiative transfer simulation in inhomogeneous atmosphere: 1. FKDM, fast k-distribution model for the longwave. Journal of Geophysical Research, 109(D2), 2004.
25. Boris Fomin. A k-distribution technique for radiative transfer simulation in inhomogeneous atmosphere: 2. FKDM, fast k-distribution model for the shortwave. Journal of Geophysical Research, 110(D2), 2005.
26. V. A. Falaleeva and B.A. Fomin. Overcoming spectroscopic challenges in direct problems of satellite sounding of the atmosphere. Atmospheric and Oceanic Optics, 2017, 30(1), pp.1-6.
27. B.A. Fomin. Effective interpolation technique for line-by-line calculations of radiation absorption in gases. Journal of Quantitative Spectroscopy and Radiative Transfer, 1995, 53(6), pp.663-669.
28. S. Najmabadi, P. Offenhauser, M. Hamann, G. Jajnabalkya, F. Hempert, C. Glass, and S. Simon. Analyzing the effect and performance of lossy compression on aeroacoustic simulation of gas injector. Computation, 2017, 5(4):24.
29. S.T. Roweis and L.K. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 2000, 290(22), pp.2323-2326.
30. M.A. Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE Journal, 1991, 37(2), pp.233-243.
31. A.N. Gorban, B. Kegl, D.C. Wunsch, and A.Y. Zinovyev, editors. Principal Manifolds for Data Visualization and Dimension Reduction. Springer Berlin Heidelberg, 2008.
32. I.K. Fodor. A Survey of Dimension Reduction Techniques. Office of Scientific and Technical Information (OSTI), 2002.
33. L.J.P. van der Maaten, E.O. Postma, and H.J. van den Herik. Dimensionality Reduction: A Comparative Review. Tilburg University Technical Report, TiCCTR2009- 005, 2009.
34. D.S. Efremenko, A. Doicu, D. Loyola, and T. Trautmann. Optical property dimensionality reduction techniques for accelerated radiative transfer performance: Application to remote sensing total ozone re trievals. J Quant Spectrosc Radiat Transfer, 2014, 133, pp.128-135.
35. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011,12, pp.2825-2830.
36. Robert H. MacArthur. On the relative abundance of bird species. Proc Natl Acad Sci USA, 1957, 43(3), pp.293-295.
37. X Liu, W.L. Smith, D.K. Zhou, and A. Larar. Principal component-based radiative transfer model for hyperspectral sensors: theoretical concept. Applied Optics, 2006, 45(1), pp.201-208.
38. D. G. Loyola, M. Pedergnana, and S. Gimeno Garcia. Smart sampling and incremental function learning for very large high dimensional data. Neural Networks, 2016, 78, pp.75-87.
39. J. H. Halton. Algorithm 247: Radical-inverse quasi-random point sequence. Commun ACM, 1964, 7(12), pp,701-702.
40. M. Matricardi. A principal component based version of the RTTOV fast radiative transfer model. Quarterly Journal of the Royal Meteorological Society, 2010, 136, pp.1823-1835.
41. P. D. Hurley, S. Oliver, D. Farrah, L. Wang, and A. Efstathiou. Principal component analysis and radiative transfer modelling of Spitzer Infrared spectrograph spectra of ultraluminous infrared galaxies. Monthly Notices of the Royal Astronomical Society, 2012, 424(3), pp.2069-2078.
42. A. Hollstein and R. Lindstrot. Fast reconstruction of hyperspectral radiative transfer simulations by using small spectral subsets: application to the oxygen A band. Atmospheric Measurement Techniques, 2014, 7(2), pp.599-607.
43. U. Platt. Differential optical absorption spectroscopy (DOAS). Chem Anal Series, 1994, 127, pp.27-83.
44. U. Platt and J. Stutz. Differential Optical Absorption Spectroscopy: Principles and Applications. Springer- Verlag, Berlin, Heidelberg, 2008.
45. G.W. Kattawar, A.T. Young, and T.J. Humphreys. Inelastic scattering in planetary atmospheres. I. The Ring effect, without aerosols. Astrophys J, 1981, 243, pp.1049-1057.
46. M. Vountas, V.V. Rozanov, and J.P. Burrows. Ring effect: Impact of rotational Raman scattering on radiative transfer in earthТs atmosphere. J Quant Spectrosc Radiat Transfer, 1998, 60(6), pp.943-961.
47. James Slusser, Kyle Hammond, Arve Kylling, Knut Stamnes, Lori Perliski, Arne Dahlback, Donald Anderson, and Robert DeMajistre. Comparison of air mass computations. Journal of Geophysical Research: Atmospheres, 1996,101(D5), pp. 9315-9321.
48. Lori M. Perliski and Susan Solomon. On the evaluation of air mass factors for atmospheric near-ultraviolet and visible absorption spectroscopy. Journal of Geophysical Research, 1993, 98(D6), p.10363.
49. Can Li, Joanna Joiner, Nickolay A. Krotkov, and Pawan K. Bhartia. A fast and sensitive new satellite SO2 retrieval algorithm based on principal component analysis: Application to the ozone monitoring instrument. Geophysical Research Letters, 2013, 40(23), pp. 6314-6318.
50. R.J.D. Spurr. LIDORT and VLIDORT. Linearized pseudo-spherical scalar and vector discrete ordinate radiative transfer models for use in remote sensing retrieval problems. In A.A. Kokhanovsky, editor, Light scattering reviews, 2008, V. 3, pp. 229-275.
51. A. Doicu and T. Trautmann. Two linearization methods for atmospheric remote sensing. J Quant Spectrosc Radiat Transfer, 2009, 110(8), pp. 477-490.
52. P.F. Levelt, G.H.J. van den Oord, M.R. Dobber, A. Malkki, Huib Visser, Johan de Vries, P. Stammes, J.O.V. Lundell, and H. Saari. The ozone monitoring instrument. IEEE Transactions on Geoscience and Remote Sensing, 2006, 44(5), pp.1093-1101.
53. A.N. Tikhonov and V.Y. Arsenin. Solution of Ill- Posed Problems. Winston, 1977.
54. C.D. Rodgers. Inverse methods for atmospheric sounding: Theory and Practice. Wolrd Scientific Publishing, 2000.
55. A. Doicu, T. Trautmann, and F. Schreier. Numerical Regularization for Atmospheric Inverse Problems. Springer, 2010.
56. D. S. Efremenko, O. Schussler, A. Doicu, and D. Loyola. A stochastic cloud model for cloud and ozone retrievals from UV measurements. J Quant Spectrosc Radiat Transfer, November 2016,184, pp.167-179.
57. J. Xu, O. Schussler, D.G. Loyola Rodriguez, F. Romahn, and A. Doicu. A novel ozone profile shape retrieval using full-physics inverse learning machine (FP-ILM). IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2017, 10(12), pp.5442-5457.
58. G. Hughes. On the mean accuracy of statistical pattern recognizers. IEEE Trans. Inform. Theory, 1968, 14(1), pp.55-63.
59. W.J. Blackwell and F.W. Chen. Neural Networks in Atmospheric Remote Sensing. Lexington, 2009.
60. John Shawe-Taylor and Nello Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, New York, NY, USA, 2004.
61. G. Camps-Valls, J. Munoz-Mari, L. Gomez- Chova, L. Guanter, and X. Calbet. Nonlinear statistical retrieval of atmospheric profiles from MetOp-IASI and MTG-IRS infrared sounding data. IEEE Transactions on Geoscience and Remote Sensing, May 2012 50(5), pp.1759-1769.
62. Roman Rosipal and Nicole Kramer. Overview and recent advances in partial least squares. In Subspace, Latent Structure and Feature Selection, pages 34-51. Springer Science + Business Media, 2006.
63. Peter D. Wentzell and Lorenzo Vega Montoto. Comparison of principal components regression and partial least squares regression through generic simulations of complex mixtures. Chemometrics and Intelligent Laboratory Systems, 2003, 65(2), pp.257-279.
64. Saikat Maitra and Jun Yan. Principal component analysis and partial least squares: Two dimension reduction techniques for regression. In Discussion Papers: 2008 Discussion Paper Program - Applying Multivariate Statistical Models, 2008, pp.79-90, Casualty actuarial society.
65. H. Hotelling. The most predictable criterion. Journal of Educational Psychology, 1935, 26, pp.139-142.
66. D. Wilks. Statistical Methods in the Atmospheric Sciences, 3rd Edition. New York: Elsevier, 2011.
67. D.S. Efremenko, D.G. Loyola R., P. Hedelt, and R.J.D. Spurr. Volcanic SO2 plume height retrieval from UV sensors using a full-physics inverse learning machine algorithm. International Journal of Remote Sensing, 2017, 38(sup1), pp.1-27.
68. Pascal Hedelt, Dmitry S. Efremenko, Diego G. Loyola, Robert Spurr, and Lieven Clarisse. SO2 layer height retrieval from Sentinel?5 Precursor/ TROPOMI using FP_ILM. Atmospheric Measurement Techniques Discussions, Feb. 2019, pp.1-23.
69. M.Yu. Kataev, A.K. Lukyanov, and A.A. Bekerov. Modification of the empirical orthogonal functions method for solving the inverse task of retrieving of the CO2 total content from satellite data. Journal of Siberian Federal University. Engineering & Technologies, 2018,11(1), pp.77-85.
70. M.Y. Kataev and A.K. Lukyanov. Empirical orthogonal functions and its modification in the task of retrieving of the total amount CO2 and CH4 with help of satellite Fourier transform spectrometer GOSAT (TANSO-FTS). In G.G. Matvienko and O.A. Romanovskii, editors, 22nd International Symposium on Atmospheric and Ocean Optics: Atmospheric Physics. SPIEIntl Soc Optical Eng, 2016.
71. V. Natraj, X. Jiang, R.L. Shia, X. Huang, J.S. Margolis, and Y.L. Yung. Application of the principal component analysis to high spectral resolution radiative transfer: A case study of the O2A-band. J Quant Spectrosc Radiat Transfer, 2005, 95(4), pp.539-556.
72. I.E. Gordon, L.S. Rothman, C. Hill, R.V. Kochanov, Y. Tan, P.F. Bernath, M. Birk, V. Boudon, A. Campargue, K.V. Chance, B.J. Drouin, J.- M. Flaud, R.R. Gamache, J.T. Hodges, D. Jacquemart, V.I. Perevalov, A. Perrin, K.P. Shine, M.-A.H. Smith, J. Tennyson, G.C. Toon, H. Tran, V.G. Tyuterev, A. Barbe, A.G. Csaszar, V.M. Devi, T. Furtenbacher, J.J. Harrison, J.-M. Hartmann, A. Jolly, T.J. Johnson, T. Karman, I. Kleiner, A.A. Kyuberis, J. Loos, O.M. Lyulin, S.T. Massie, S.N. Mikhailenko, N. Moazzen-Ahmadi, H.S.P. Muller, O.V. Naumenko, A.V. Nikitin, O.L. Polyansky, M. Rey, M. Rotger, S.W. Sharpe, K. Sung, E. Starikova, S.A. Tashkun, J. Vander Auwera, G. Wagner, J. Wilzewski, P. Wciso, S. Yu, and E.J. Zak. The HITRAN2016 molecular spectroscopic database. Journal of Quantitative Spectroscopy and Radiative Transfer, 2017, 203, pp.3-69.
73. P. Somkuti, H. Boesch, V. Natraj, and P. Kopparla. Application of a PCA-based fast radiative transfer model to XCO2 retrievals in the shortwave infrared. Journal of Geophysical Research: Atmospheres, 122(19), pp.10,477-10,496.
74. V. Natraj, R.L. Shia, and Y.L. Yung. On the use of principal component analysis to speed up radiative transfer calculations. J Quant Spectrosc Radiat Transfer, 2010, 111(5), pp.810-816.
75. P. Kopparla, V. Natraj, R. Spurr, R. Shia, D. Crisp, and Y.L. Yung. A fast and accurate PCA based radiative transfer model: Extension to the broadband shortwave region. Journal of Quantitative Spectroscopy and Radiative Transfer, 2016, 173, pp. 65-71.
76. Ana del Aguila, Dmitry Efremenko, Victor Molina Garcia, and Jian Xu. Analysis of two dimensionality reduction techniques for fast simulation of the spectral radiances in the Hartley-Huggins band. Atmosphere, Mar. 2019,10(3), p.142.
77. Y. Timofeyev, A. Polyakov, H. Steele, and M. Newchurch. Optimal eigenanalysis for the treatment of aerosols in the retrieval of atmospheric composition from transmission measurements. Appl. Opt., July 2003,42 (15), p.2635.
78. G. Mie. Beitraege zur optik trueber medien, speziell kolloidaler metalloesungen. Annalen der Physik, 1908, 330(3), pp.377-445.
79. A. V. Polyakov, Yu.M. Timofeev, and Ya.A. Virolainen. Using artificial neural networks in the temperature and humidity sounding of the atmosphere. Izvestiya, Atmospheric and Oceanic Physics, 2014, 50(3), pp.330-336.
80. Michael T. Eismann, Joseph Meola, and Russell C. Hardie. Hyperspectral change detection in the presence of diurnal and seasonal variations. IEEE Transactions on Geoscience and Remote Sensing, 2008, 46(1), pp. 237-249.
81. Dandan Ma, Yuan Yuan, and Qi Wang. Hyperspectral anomaly detection via discriminative feature learning with multiple-dictionary sparse representation. Remote Sensing, May 2018,10(5), p.745.
82. David B Brough, Daniel Wheeler, and Surya R. Kalidindi. Materials knowledge systems in python - a data science framework for accelerated development of hierarchical materials. Integrating Materials and Manufacturing Innovation, Mar.2017, 6(1), pp. 36-53.
83. Sai Kiranmayee Samudrala, Prasanna Venkataraman Balachandran, Jaroslaw Zola, Krishna Rajan, and Baskar Ganapathysubramanian. A software framework for data dimensionality reduction: application to chemical crystallography. Integrating Materials and Manufacturing Innovation, 3(1), Jun. 2014.
84. Pilar B. Garca-Allende, Olga M. Conde, Ana M. Cubillas, Cesar Jauregui, and Jose M. Lopez- Higuera. New raw material discrimination system based on a spatial optical spectroscopy technique. Sensors and Actuators A: Physical, Apr. 2007, 135(2), pp. 605-612.
85. Juan Xing, Cedric Bravo, Pal T. Jancsok, Herman Ramon, and Josse De Baerdemaeker. Detecting bruises on Сgolden deliciousТ apples using hyperspectral imaging with multiple wavebands. Biosystems Engineering, Jan 2005, 90(1), pp.27-36.
86. V. Molina Garcia, S. Sasi, D.S. Efremenko, A. Doicu, and D. Loyola. Radiative transfer models for retrieval of cloud parameters from EPIC/DSCOVR measurements. Journal of Quantitative Spectroscopy and Radiative Transfer, 2018, 213, pp. 228-240.
87. V.P. AfanasТev, V.P. Budak, D.S. Efremenko, and P.S. Kaplya. Application of the photometric theory of the radiance field in the problems of electron scattering. Light & Engineering, 2019, 27(2), pp. 88-96.
88. F. Vasefi, N. MacKinnon, and D.L. Farkas. Hyperspectral and multispectral imaging in dermatology. In Imaging in Dermatology, Elsevier, 2016, pp. 187-201.

• Previous article
• Next article

• passive remote sensing
• hyper-spectral data
• principal component analysis
• full-physics machine learning
• trace gas retrieval