Гиперспектральные датчики проводят измерения в узких сопряжённых полосах спектра электромагнитного излучения. Целью при этом обычно является обнаружение определённого объекта или элемента среды, имеющих присущие только им спектральные характеристики. В частности, гиперспектральные измерения применяются при дистанционном зондировании атмосферы для выявления малых газовых компонент. Для улучшения эффективности алгоритма обработки гиперспектральных данных были использованы методы уменьшения количества данных. В статье описаны методы снижения размерности применительно к гиперспектральному дистанционному зондированию атмосферы. При снижении размерности происходит исключение из данных избыточной информации, и в настоящее время снижение размерности является неотъемлемой частью высокопроизводительных моделей переноса излучения. В обзоре описано, как можно использовать метод главных компонент 2 для моделирования спектрального распределения энергетической яркости и определения составляющих атмосферы, ускоряя тем самым на порядки скорость обработки данных. Представленные методы являются обобщёнными, и их можно непосредственно использовать для решения как атмосферных задач, так и задач из других связанных с материаловедением областей знаний. [1]. D.G. Loyola, S. Gimeno Garcia, R. Lutz, A. Argyrouli, F. Romahn, R.J.D. Spurr, M. Pedergnana, A. Doicu, V. Molina Garcá, and O. Schüssler. The operational cloud retrieval algorithms from TROPOMI on board Sentinel‑5 precursor. Atmospheric Measurement Techniques, 11(1):409–427, 2018.
[2]. C.A. Lee, S.D. Gasster, A. Plaza, C. Chang, and B. Huang. Recent developments in high performance computing for remote sensing: A review. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 4(3):508–527, 2011.
[3]. J.P. Veefkind, I. Aben, K. McMullan, H. Forster, J. de Vries, G. Otter, J. Claas, H.J. Eskes, J.F. de Haan, Q. Kleipool, and et al. TROPOMI on the ESA Sentinel‑5 Precursor: A GMES mission for global observations of the atmospheric composition for climate, air quality and ozone layer applications. Remote Sensing of Environment, 120:70–83, 2012.
[4]. Y. Ma, H. Wu, L. Wang, B. Huang, R. Ranjan, A. Zomaya, and W. Jie. Remote sensing big data computing: Challenges and opportunities. Future Generation Computer Systems, 51:47–60, 2015.
[5]. P. Liu. A survey of remote-sensing big data. Frontiers in Environmental Science, 3:45, 2015.
[6]. V.M. Roozendael, R. Spurr, D. Loyola, C. Lerot, D. Balis, J.C. Lambert, W. Zimmer, J. Gent, J. Van Geffen, M.E. Koukouli, J. Granville, A. Doicu, C. Fayt, and C. Zehner. Sixteen years of GOME/ERS2 total ozone data: the new direct-fitting GOME Data Processor (GDP) Version 5: I. algorithm description. J Geophys Res: Atmospheres, 117: D03305(1–18), 2012.
[7]. V. Natraj. A review of fast radiative transfer techniques. In A.A. Kokhanovsky, editor, Light scattering reviews, volume 8, pages 475–504. Springer Berlin Heidelberg, 2013.
[8]. D. Efremenko, A. Doicu, D. Loyola, and T. Trautmann. Acceleration techniques for the discrete ordinate method. J Quant Spectrosc Radiat Transfer, 114:73–81, 2013.
[9]. D.S. Efremenko, D. Loyola, A. Doicu, and T. Trautmann. Data-intensive computing in radiative transfer modelling. In P. Soille and P.G. Marchetti, editors, Proc. of the 2016 conference on Big Data from Space (BiDS16), pages 188–191, Santa Cruz de Tenerife, Spain, 2016.
[10]. V.P. Budak, G.A. Kaloshin, O.V. Shagalov, and V.S. Zheltov. Numerical modeling of the radiative transfer in a turbid medium using the synthetic iteration. Opt. Express, 23(15): A829, 2015.
[11]. D.S. Efremenko, D.G. Loyola, A. Doicu, and R.J.D. Spurr. Multi-core-CPU and GPU-accelerated radiative transfer models based on the discrete ordinate method. Computer Physics Communications, 185(12):3079–3089, 2014.
[12]. D.S. Efremenko, D. Loyola, R.J.D. Spurr, and A. Doicu. Acceleration of radiative transfer model calculations for the retrieval of trace gases under cloudy conditions. J Quant Spectrosc Radiat Transfer, 135:58–65, 2014.
[13]. R.J.D. Spurr. VLIDORT: A linearized pseudo-spherical vector discrete ordinate radiative transfer code for forward model and retrieval studies in multilayer multiple scattering media. J Quant Spectrosc Radiat Transfer, 102(2):316–342, 2006.
[14]. R.J.D. Spurr, T.P. Kurosu, and K.V. Chance. A linearized discrete ordinate radiative transfer model for atmospheric remotesensing retrieval. J Quant Spectrosc Radiat Transfer, 68(6):689–735, 2001.
[15]. K. Pearson. On lines and planes of closest fit to systems of points in space. Phil Mag, 2(6):559–572, 1901. [16]. R. D. Morris, A. Kottas, M. Taddy, R. Furfaro, and B.D. Ganapol. A statistical framework for the sensitivity analysis of radiative transfer models. IEEE Transactions on Geoscience and Remote Sensing, 46(12):4062–4074, 2008.
[17]. V.A. Ambartzumyan. The effect of the absorption lines on the radiative equilibrium of the outer layers of the stars. Publ. Obs. Astron. Univ. Leningrad, 6:7–18, 1936.
[18]. R.M. Goody, R. West, L. Chen, and D. Crisp. The correlated k-method for radiation calculations in nonhomogeneous atmosphere. J Quant Spectrosc Radiat Transfer, 42(6):539–550, 1989.
[19]. W.J. Wiscombe and J.W. Evans. Exponentialsum fitting of radiative transmission functions. J Comput Phys, 24(4):416– 444, 1997.
[20]. R. West, D. Crisp, and L. Chen. Mapping transformations for broadband atmospheric radiation calculation. J Quant Spectrosc Radiat Transfer, 43(3):191–199, 1990.
[21]. E. Boesche, P. Stammes, R. Preusker, R. Bennartz, W. Knap, and J. Fischer. Polarization of skylight in the O2A band: effects of aerosol properties. Applied Optics, 47(19):3467, 2008.
[22]. Christiane Helling and Uffe Grae Jorgensen. Optimizing the opacity sampling method. Astronomy and Astrophysics, 337:477–486, 1998.
[23]. Frédéric André, Longfeng Hou, Maxime Roger, and Rodolphe Vaillon. The multispectral gas radiation modeling: A new theoretical framework based on a multidimensional approach to k-distribution methods. Journal of Quantitative Spectroscopy and Radiative Transfer, 147:178–195, nov 2014.
[24]. Boris A. Fomin. A k-distribution technique for radiative transfer simulation in inhomogeneous atmosphere: 1. FKDM, fast k-distribution model for the longwave. Journal of Geophysical Research, 109(D2), 2004.
[25]. Boris Fomin. A k-distribution technique for radiative transfer simulation in inhomogeneous atmosphere: 2. FKDM, fast k-distribution model for the shortwave. Journal of Geophysical Research, 110(D2), 2005.
[26]. V. A. Falaleeva and B.A. Fomin. Overcoming spectroscopic challenges in direct problems of satellite sounding of the atmosphere. Atmospheric and Oceanic Optics, 30(1):1–6, jan 2017.
[27]. B.A. Fomin. Effective interpolation technique for line-by-line calculations of radiation absorption in gases. Journal of Quantitative Spectroscopy and Radiative Transfer, 53(6):663–669, jun 1995.
[28]. S. Najmabadi, P. Offenhäuser, M. Hamann, G. Jajnabalkya, F. Hempert, C. Glass, and S. Simon. Analyzing the effect and performance of lossy compression on aeroacoustic simulation of gas injector. Computation, 5(4):24, 2017.
[29]. S.T. Roweis and L.K. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(22):2323–2326, 2000.
[30]. M.A. Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE Journal, 37(2):233–243, 1991.
[31]. A.N. Gorban, B. Kégl, D.C. Wunsch, and A.Y. Zinovyev, editors. Principal Manifolds for Data Visualization and Dimension Reduction. Springer Berlin Heidelberg, 2008.
[32]. I.K. Fodor. A Survey of Dimension Reduction Techniques. Office of Scientific and Technical Information (OSTI), 2002.
[33]. L.J.P. van der Maaten, E.O. Postma, and H.J. van den Herik. Dimensionality Reduction: A Comparative Review. Tilburg University Technical Report, TiCC-TR2009–005, 2009.
[34]. D.S. Efremenko, A. Doicu, D. Loyola, and T. Trautmann. Optical property dimensionality reduction techniques for accelerated radiative transfer performance: Application to remote sensing total ozone retrievals. J Quant Spectrosc Radiat Transfer, 133:128–135, 2014.
[35]. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
[36]. Robert H. MacArthur. On the relative abundance of bird species. Proc Natl Acad Sci USA, 43(3):293–295, 1957.
[37]. X Liu, W.L. Smith, D.K. Zhou, and A. Larar. Principal component-based radiative transfer model for hyperspectral sensors: theoretical concept. Applied Optics, 45(1):201–208, 2006.
[38]. D. G. Loyola, M. Pedergnana, and S. Gimeno Garcia. Smart sampling and incremental function learning for very large high dimensional data. Neural Networks, 78:75–87, 2016.
[39]. J. H. Halton. Algorithm 247: Radical-inverse quasi-random point sequence. Commun ACM, 7(12):701–702, 1964.
[40]. M. Matricardi. A principal component based version of the RTTOV fast radiative transfer model. Quarterly Journal of the Royal Meteorological Society, 136:1823–1835, 2010.
[41]. P. D. Hurley, S. Oliver, D. Farrah, L. Wang, and A. Efstathiou. Principal component analysis and radiative transfer modelling of Spitzer Infrared spectrograph spectra of ultraluminous infrared galaxies. Monthly Notices of the Royal Astronomical Society, 424(3):2069–2078, 2012.
[42]. A. Hollstein and R. Lindstrot. Fast reconstruction of hyperspectral radiative transfer simulations by using small spectral subsets: application to the oxygen A band. Atmospheric Measurement Techniques, 7(2):599–607, 2014.
[43]. U. Platt. Differential optical absorption spectroscopy (DOAS). Chem Anal Series, 127:27–83, 1994.
[44]. U. Platt and J. Stutz. Differential Optical Absorption Spectroscopy: Principles and Applications. Springer-Verlag, Berlin, Heidelberg, 2008.
[45]. G.W. Kattawar, A.T. Young, and T.J. Humphreys. Inelastic scattering in planetary atmospheres. I. The Ring effect, without aerosols. Astrophys J, 243:1049–1057, 1981.
[46]. M. Vountas, V.V. Rozanov, and J.P. Burrows. Ring effect: Impact of rotational Raman scattering on radiative transfer in earth’s atmosphere. J Quant Spectrosc Radiat Transfer, 60(6):943–961, 1998.
[47]. James Slusser, Kyle Hammond, Arve Kylling, Knut Stamnes, Lori Perliski, Arne Dahlback, Donald Anderson, and Robert De-Majistre. Comparison of air mass computations. Journal of Geophysical Research: Atmospheres, 101(D5):9315–9321, 1996.
[48]. Lori M. Perliski and Susan Solomon. On the evaluation of air mass factors for atmospheric near-ultraviolet and visible absorption spectroscopy. Journal of Geophysical Research, 98(D6):10363, 1993.
[49]. Can Li, Joanna Joiner, Nickolay A. Krotkov, and Pawan K. Bhartia. A fast and sensitive new satellite SO2 retrieval algorithm based on principal component analysis: Application to the ozone monitoring instrument. Geophysical Research Letters, 40(23):6314–6318, 2013.
[50]. R.J.D. Spurr. LIDORT and VLIDORT. Linearized pseudo-spherical scalar and vector discrete ordinate radiative transfer models for use in remote sensing retrieval problems. In A.A. Kokhanovsky, editor, Light scattering reviews, volume 3, pages 229–275. 2008.
[51]. A. Doicu and T. Trautmann. Two linearization methods for atmospheric remote sensing. J Quant Spectrosc Radiat Transfer, 110(8):477–490, 2009.
[52]. P.F. Levelt, G.H.J. van den Oord, M.R. Dobber, A. Malkki, Huib Visser, Johan de Vries, P. Stammes, J.O.V. Lundell, and H. Saari. The ozone monitoring instrument. IEEE Transactions on Geoscience and Remote Sensing, 44(5):1093–1101, 2006.
[53]. A.N. Tikhonov and V.Y. Arsenin. Solution of Ill-Posed Problems. Winston, 1977.
[54]. C.D. Rodgers. Inverse methods for atmospheric sounding: Theory and Practice. Wolrd Scientific Publishing, 2000.
[55]. A. Doicu, T. Trautmann, and F. Schreier. Numerical Regularization for Atmospheric Inverse Problems. Springer, 2010.
[56]. D. S. Efremenko, O. Schüssler, A. Doicu, and D. Loyola. A stochastic cloud model for cloud and ozone retrievals from UV measurements. J Quant Spectrosc Radiat Transfer, 184:167–179, November 2016.
[57]. J. Xu, O. Schussler, D.G. Loyola Rodriguez, F. Romahn, and A. Doicu. A novel ozone profile shape retrieval using full-physics inverse learning machine (FP-ILM). IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10(12):5442–5457, 2017.
[58]. G. Hughes. On the mean accuracy of statistical pattern recognizers. IEEE Trans. Inform. Theory, 14(1):55–63, 1968.
[59]. W.J. Blackwell and F.W. Chen. Neural Networks in Atmospheric Remote Sensing. Lexington, 2009.
[60]. John Shawe-Taylor and Nello Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, New York, NY, USA, 2004.
[61]. G. Camps-Valls, J. Munoz-Mari, L. Gomez-Chova, L. Guanter, and X. Calbet. Nonlinear statistical retrieval of atmospheric profiles from MetOp-IASI and MTG-IRS infrared sounding data. IEEE Transactions on Geoscience and Remote Sensing, 50(5):1759– 1769, May 2012.
[62]. Roman Rosipal and Nicole Krämer. Overview and recent advances in partial least squares. In Subspace, Latent Structure and Feature Selection, pages 34–51. Springer Science + Business Media, 2006.
[63]. Peter D. Wentzell and Lorenzo Vega Montoto. Comparison of principal components regression and partial least squares regression through generic simulations of complex mixtures. Chemometrics and Intelligent Laboratory Systems, 65(2):257–279, 2003.
[64]. Saikat Maitra and Jun Yan. Principal component analysis and partial least squares: Two dimension reduction techniques for regression. In Discussion Papers: 2008 Discussion Paper Program – Applying Multivariate Statistical Models, pages 79–90. Casualty actuarial society, 2008.
[65]. H. Hotelling. The most predictable criterion. Journal of Educational Psychology, 26:139–142, 1935.
[66]. D. Wilks. Statistical Methods in the Atmospheric Sciences, 3rd Edition. New York: Elsevier, 2011.
[67]. D.S. Efremenko, D.G. Loyola R., P. Hedelt, and R.J.D. Spurr. Volcanic SO2 plume height retrieval from UV sensors using a full-physics inverse learning machine algorithm. International Journal of Remote Sensing, 38(sup1):1–27, 2017.
[68]. Pascal Hedelt, Dmitry S. Efremenko, Diego G. Loyola, Robert Spurr, and Lieven Clarisse. SO2 layer height retrieval from Sentinel‑5 Precursor/TROPOMI using FP_ILM. Atmospheric Measurement Techniques Discussions, pages 1–23, feb 2019.
[69]. M.Yu. Kataev, A.K. Lukyanov, and A.A. Bekerov. Modification of the empirical orthogonal functions method for solving the inverse task of retrieving of the CO2 total content from satellite data. Journal of Siberian Federal University. Engineering & Technologies, 11(1):77–85, 2018.
[70]. M.Y. Kataev and A.K. Lukyanov. Empirical orthogonal functions and its modification in the task of retrieving of the total amount CO2 and CH4 with help of satellite Fourier transform spectrometer GOSAT (TANSO-FTS). In G.G. Matvienko and O.A. Romanovskii, editors, 22nd International Symposium on Atmospheric and Ocean Optics: Atmospheric Physics. SPIE-Intl Soc Optical Eng, 2016.
[71]. V. Natraj, X. Jiang, R.L. Shia, X. Huang, J.S. Margolis, and Y.L. Yung. Application of the principal component analysis to high spectral resolution radiative transfer: A case study of the O2A-band. J Quant Spectrosc Radiat Transfer, 95(4):539–556, 2005.
[72]. I.E. Gordon, L.S. Rothman, C. Hill, R.V. Kochanov, Y. Tan, P.F. Bernath, M. Birk, V. Boudon, A. Campargue, K.V. Chance, B.J. Drouin, J.-M. Flaud, R.R. Gamache, J.T. Hodges, D. Jacquemart, V.I. Perevalov, A. Perrin, K.P. Shine, M.-A.H. Smith, J. Tennyson, G.C. Toon, H. Tran, V.G. Tyuterev, A. Barbe, A.G. Császár, V.M. Devi, T. Furtenbacher, J.J. Harrison, J.-M. Hartmann, A. Jolly, T.J. Johnson, T. Karman, I. Kleiner, A.A. Kyuberis, J. Loos, O.M. Lyulin, S.T. Massie, S.N. Mikhailenko, N. Moazzen-Ahmadi, H.S.P. Müller, O.V. Naumenko, A.V. Nikitin, O.L. Polyansky, M. Rey, M. Rotger, S.W. Sharpe, K. Sung, E. Starikova, S.A. Tashkun, J. Vander Auwera, G. Wagner, J. Wilzewski, P. Wciso, S. Yu, and E.J. Zak. The HITRAN2016 molecular spectroscopic database. Journal of Quantitative Spectroscopy and Radiative Transfer, 203:3–69, 2017.
[73]. P. Somkuti, H. Boesch, V. Natraj, and P. Kopparla. Application of a PCA-based fast radiative transfer model to XCO2 retrievals in the shortwave infrared. Journal of Geophysical Research: Atmospheres, 122(19):10,477– 10,496.
[74]. V. Natraj, R.L. Shia, and Y.L. Yung. On the use of principal component analysis to speed up radiative transfer calculations. J Quant Spectrosc Radiat Transfer, 111(5):810– 816, 2010.
[75]. P. Kopparla, V. Natraj, R. Spurr, R. Shia, D. Crisp, and Y.L. Yung. A fast and accurate PCA based radiative transfer model: Extension to the broadband shortwave region. Journal of Quantitative Spectroscopy and Radiative Transfer, 173:65–71, 2016.
[76]. Ana del Águila, Dmitry Efremenko, Víctor Molina García, and Jian Xu. Analysis of two dimensionality reduction techniques for fast simulation of the spectral radiances in the Hartley-Huggins band. Atmosphere, 10(3):142, mar 2019.
[77]. Y. Timofeyev, A. Polyakov, H. Steele, and M. Newchurch. Optimal eigenanalysis for the treatment of aerosols in the retrieval of atmospheric composition from transmission measurements. Appl. Opt., 42(15):2635, July 2003.
[78]. G. Mie. Beitraege zur optik trueber medien, speziell kolloidaler metalloesungen. Annalen der Physik, 330(3):377–445, 1908.
[79]. A. V. Polyakov, Yu.M. Timofeev, and Ya.A. Virolainen. Using artificial neural networks in the temperature and humidity sounding of the atmosphere. Izvestiya, Atmospheric and Oceanic Physics, 50(3):330–336, 2014.
[80]. Michael T. Eismann, Joseph Meola, and Russell C. Hardie. Hyperspectral change detection in the presence of diurnal and seasonal variations. IEEE Transactions on Geoscience and Remote Sensing, 46(1):237–249, 2008.
[81]. Dandan Ma, Yuan Yuan, and Qi Wang. Hyperspectral anomaly detection via discriminative feature learning with multiple-dictionary sparse representation. Remote Sensing, 10(5):745, may 2018.
[82]. David B Brough, Daniel Wheeler, and Surya R. Kalidindi. Materials knowledge systems in python – a data science framework for accelerated development of hierarchical materials. Integrating Materials and Manufacturing Innovation, 6(1):36–53, mar 2017.
[83]. Sai Kiranmayee Samudrala, Prasanna Venkataraman Balachandran, Jaroslaw Zola, Krishna Rajan, and Baskar Ganapathysubramanian. A software framework for data dimensionality reduction: application to chemical crystallography. Integrating Materials and Manufacturing Innovation, 3(1), jun 2014.
[84] . Pilar B. Garcá - Allende, Olga M. Conde, Ana M. Cubillas, César Jáuregui, and José M. López-Higuera. New raw material discrimination system based on a spatial optical spectroscopy technique. Sensors and Actuators A: Physical, 135(2):605–612, apr 2007.
[85] . Juan Xing , Cédric Bravo, Pál T. Jancsók, Herman Ramon, and Josse De Baerdemaeker. Detecting bruises on ‘golden delicious’ apples using hyperspectral imaging with multiple wavebands. Biosystems Engineering, 90(1):27–36, jan 2005.
[86]. V. Molina García, S. Sasi, D.S. Efremenko, A. Doicu, and D. Loyola. Radiative transfer models for retrieval of cloud parameters from EPIC/DSCOVR measurements. Journal of Quantitative Spectroscopy and Radiative Transfer, 213:228–240, 2018.
[87]. Афанасьев В.П., Будак В.П., Ефременко Д.С., Капля П.С. Применение фотометрической теории светового поля в задачах рассеяния электронов // Светотехника.– 2018.– № 5. – С. 44–50.
[88]. F. Vasefi, N. MacKinnon, and D.L. Farkas. Hyperspectral and multispectral imaging in dermatology. In Imaging in Dermatology, pages 187–201. Elsevier, 2016.

• Предыдущая статья
• Следующая статья

• пассивное дистанционное зондирование
• гиперспектральные данные
• метод главных компонент
• физически полное машинное самообучение
• определение малых газовых компонент