Progress on the UN Sustainable Development Goals (SDGs) is hampered by a persistent lack of data regarding key social, environmental, and economic indicators, particularly in developing countries. For example, data on poverty and slavery, the first and eighth of seventeen SDGs respectively, are both spatially sparse and infrequently collected in developing countries due to the high cost of surveys. Governments the world over, therefore, have committed once again to ending poverty and modern slavery in all its forms by 2025. They are, however, faced with a significant challenge: lack of access to reliable, up-to-date, and actionable data on these activities. To address this, we developed two annotated datasets from open-source remote sensing imagery of South Asia. The first spatial dataset, named Asia14, consisting of 14,000 Digital Globe RGB images and 14 classes including brick kilns, houses, roads, tennis courts, grass, farms, sparse forest, dense forest, orchards, parking lots, parks, oil tanks, and barren lands. The second Spatio-temporal dataset, named 2C2D, consists of four key transition classes namely construction, destruction, cultivation, and de-cultivation. To incorporate intra-class variations over large-scale analysis, we propose a coarse-to-fine strategy consisting of an inexpensive classifier and a detector that work in tandem to achieve high accuracy at a low computational cost. More specifically, we propose a two-stage gated neural network architecture called Kiln-Net. At the first stage, imagery is classified using the ResNet-152 model which filters out over 99% of irrelevant data. At the second stage, a YOLOv3-based object detector is applied to find the precise location of each brick kiln in the candidate regions. To incorporate temporal variations over large-scale analysis, we then introduce a scalable method to predict socio-economic indicators using our spatio-temporal dataset 2C2D. Our approach improves existing techniques in three ways; forgoing hand-crafted features traditionally used in the remote sensing community, introducing a novel 3D spatio-temporal input technique, and incorporating 3D convolutional kernels to explicitly model the Spatio-temporal structure of the data and automatically learn useful features. We evaluate the efficacy of our approach on data from three different cities namely Aleppo, Kathmandu, and Lahore.