Abstract:

Progress on the UN Sustainable Development Goals (SDGs) is hampered by a persistent lack of data regarding key social, environmental, and economic indicators, particularly in developing countries. For example, data on poverty and slavery, the first and eighth of seventeen SDGs respectively, are both spatially sparse and infrequently collected in developing countries due to the high cost of surveys. Other such examples include taxation, population census, vaccine coverage and electricity demand forecasting. Use of satellite imagery provides an opportunity to automate these surveys, however, while considering large geographical area, several inherent complexities in satellite imagery make automated detection a challenging task. These include high amount of computation and storage as well as variation in images over time and space such as structural, environmental and sensor variations.

The main goal of this dissertation is to develop accurate and compute-efficient learning approaches for huge volumes of Earth's images. This study also addresses the problem of scalability of the model for automatic interpretation of satellite imagery covering a large geographical extent. For this purpose, we develop several deep learning strategies using spatial convolution, 3D convolution and graph convolution. In particular, we design four deep learning architectures: 1) Tiny-Inception-ResNet-v2, 2) Kiln-Net, 3) 3D Residual Network and 4) STAG-NN for spatial and spatio-temporal classification and detection for high resolution imagery. We also develop two large-scale classification benchmark datasets from open-source remote sensing imagery of South Asia. The first spatial dataset, named Asia14, consisting of 14,000 Digital Globe RGB images and 14 classes including brick kilns, houses, roads, tennis courts, grass, farms, sparse forest, dense forest, orchards, parking lots, parks, oil tanks, and barren lands. The second spatio-temporal dataset, named 2C2D, consists of four key transition classes: construction, destruction, cultivation, and harvesting.

Land-use and Land-cover (LULC) is one of the essential steps in many remote sensing-based surveys. Initially, we developed a general purpose and compute efficient architecture for the problem of LULC classification and tested it on Asia14 dataset with 11 different classes. We purpose and employ Inception-ResNet inspired deep learning architecture called Tiny-Inception-ResNet-v2 which provide 63.8% reduction in parameters, and 2.6% increase in classification accuracy. We then used it for the localization of brick kilns to help eliminate poverty and bonded labour within the "Brick-Kiln-Belt" of South Asia. To incorporate intra-class variations over large-scale spatial analysis, we propose a coarse-to-fine strategy consisting of an inexpensive classifier and a detector that work in tandem to achieve high accuracy at a low computational cost. More specifically, we propose a two-stage gated neural network architecture called Kiln-Net. At the first stage, imagery is classified using the ResNet-152 model which filters out over 99% of irrelevant data. At the second stage, a YOLOv3-based object detector is applied to find the precise location of each brick kiln in the candidate regions.

To incorporate temporal variations over large-scale analysis, we then introduce a scalable method to predict socio-economic indicators using our spatio-temporal dataset 2C2D. Our approach improves existing techniques in three ways; forgoing hand-crafted features traditionally used in the remote sensing community, introducing a novel 3D spatio-temporal input technique and incorporating 3D convolutional kernels to explicitly model the spatio-temporal structure of the data and automatically learn useful features. We evaluate the efficacy of our approach on data from three different cities from different South Asian countries. Despite the significant computational overhead, 3D convolution networks fail to address the problems of non-Euclidean domains. We, therefore, develop Spatio-Temporal driven Attention Graph Neural Network (STAG-NN) to learn the interactions between dense spatial and sparse temporal data. Performance of the STAG-NN is evaluated for spatial and spatio-temporal feature learning over large scale analysis. We also demonstrate the comprehensive comparative analysis (in terms of accuracy and compute cost) of proposed techniques with state-of-the-art deep learning methods.

Our proposed solution will enable regional monitoring and evaluation mechanisms for the SDGs and will help governments target their interventions with the precision that was previously unheard of. Here we only demonstrate its application for the geo-localization of brick kilns and the detection of spatio-temporal transition classes e.g., construction, destruction, cultivation and harvesting. This research will also allow governments to monitor illegal kiln activities and implement strategies to help prevent famine and emancipate individuals trapped within institutions of slavery and support humanitarian efforts.

List of Publications:

M. A. Bhimra, Usman Nazir, and Murtaza Taj. "Using 3d Residual Network For Spatio-Temporal Analysis Of Remote Sensing Data." IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019. Link: https://ieeexplore.ieee.org/document/8682286

Usman Nazir, et al. "Tiny-Inception-ResNet-v2: Using Deep Learning For Eliminating Bonded Labors Of Brick Kilns In South Asia." IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. 2019. Link: https://arxiv.org/abs/1907.05552

Usman Nazir, et al. "Kiln-net: A gated neural network for detection of brick kilns in South Asia." IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (JSTARS) 13 (2020): 3251-3262. Link: https://ieeexplore.ieee.org/document/9115879

Usman Nazir, Muhammad Awais Ather, and Murtaza Taj. "Detection of Illegal Kiln Activity During SMOG Period." International Conference on Robotics and Automation in Industry (ICRAI) 2023.  Link: https://arxiv.org/abs/2301.07521

M Sohail, Usman Nazir, et al. “Analyzing the knock-on Impacts of 2022 Floods on Rabi 2023 Using Remote Sensing and Field Surveys” International Conference on Integrated Flood Management under Changing Climate Scenario (IFMCC), 2023.

Usman Nazir, et al. “Improved flood mapping using fusion of Sentinel-1, Sentinel-2 and Landsat-9 datasets” International Conference on Integrated Flood Management under Changing Climate Scenario (IFMCC), 2023.

Usman Nazir, et al. “Mitigating climate and health impact of small-scale kiln industry using multi-spectral classifier and deep learning”  ICLR 2023 Workshop: Tackling climate change with AI. Link: https://arxiv.org/abs/2303.11654

Usman Nazir, et al. “Spatio-Temporal Driven Attention Graph Neural Network with Block Adjacency Matrix (STAG-NN-BA)” ACM Conference On Knowledge Discovery And Data Mining (SIGKDD), 2023. (submitted)

Website: https://usmanweb.github.io/

Google Scholar: https://scholar.google.com.pk/citations?user=EKpcuAoAAAAJ&hl=en

Final Defense Committee (FDC):

1. Dr. Murtaza Taj (Advisor)
2. Dr. Momin Uppal (Member)
3. Dr. Zubair Khalid (Member)
4. Prof. Mian Muhammad Awais  (Member)
5. Dr. Mohsen Ali (External Examiner)