The inescapable problem of increased data traffic in wireless communication systems has been the subject of interest to researchers in the wireless community. The exponential growth in data traffic volume, increase in the number of connected devices, capacity and efficiency have led to the development of next-generation wireless technologies, known as 5G technology. A novel technical solution and a paradigm shift to achieve the spectral efficiency required by 5G standards are to incorporate a large number of antennas in the system called massive multiple-input multiple-output (MIMO). Massive MIMO systems scale up the capacity by the number of transmit and receive antennas. Furthermore, a large segment of underutilized spectra is available at millimeter Wave (mmWave) band, which are suitable for 5G communication systems as they occupy large bandwidth. However, at these high frequencies, a large number of antennas are required for a reasonable link signal-to-noise ratio (SNR). In addition to beamforming gain, large antenna arrays also enable precoding to support multiple users, multiple streams, improved spectral efficiency, and increased data rates.

However, realizing mmWave massive MIMO systems in practice involves several key challenges. Traditional MIMO processing, which is performed digitally at baseband, cannot be directly extended to future massive MIMO 5G systems. The optimal fully digital systems are costly and power-hungry, as a separate radio frequency (RF) chain required for each antenna element where an RF chain includes analog to digital converter, down-converter, and a low-noise amplifier.

This dissertation is directed towards a unified framework for energy and spectral efficient (EE-SE) systems, which simultaneously employ techniques from array processing for massive MIMO at mmWave frequency band. The Saleh-Valenzuela 3D channel model capturing the characteristics of the azimuth and elevation dimensions for the mmWave massive MIMO systems. This strategy where azimuth and elevation dimensions are jointly beamformed is sometimes referred to as full dimension (FD) MIMO, and is an active area of research in standardization. Spectral efficiency (SE), energy efficiency (EE), sum rate, and SNR are the key performance metrics considered in this thesis. This study consists of two parts: the first part discusses the spatial correlation properties of mmWave massive MIMO channel in the presence of different object/scatterers, and the second part involves symbol optimization while ensuring maximum SE and EE, and is also related to the design of the hybrid precoder based on the channel characteristics for single (SU) and multi-user (MU) scenarios. To carry out the first analysis, the analytical expression for the generalized spatial correlation function is derived by leveraging the mathematical convenience of the spherical harmonic transform (SHT) using spherical harmonic expansion of plane waves. Since due to the compact structure of large-scale antenna arrays and the angular distribution of the scatterer, the correlation among antenna elements increases. The increased spatial correlation affects the wireless link performance and must be thoroughly investigated.

In the second part of the thesis, different hybrid precoding algorithm designs are considered for SU and MU cases. The performance of the hybrid system is examined by changing the multiplexing of data streams that are transmitted by optimizing the symbols under different SNR values. The multiple data streams and RF chains for large arrays introduce inter-stream interference, degrade system SE and increase the implementation cost of the system. On the other hand, SE would be increased with an increasing SNR, if the suitable numbers of data streams are supported. A novel heuristic algorithm is proposed to determine the optimal symbols for a SU case. Finally, a relationship in terms of a trade-off between EE and SE of hybrid beamforming structures at mmWaves is discussed. The simulation results are generated using MATLAB. The performance of different linear hybrid precoding schemes is then compared for multi-user in the random and closely-spaced locations. The analysis shows an extra RF chain may be needed for the closely-spaced user placement to get the performance close to randomly located users. A best optimal RF digital precoding combination for hybrid structure is also suggested. It is believed that the study and the results provided in this thesis are applicable to the analysis of future 5G mmWave massive MIMO systems.