The Facial Recognition System is used to identify the persons based upon the facial data as the marker. There are many approaches to building such a recognition system. One of the traditional methods is Local Binary Pattern(LBP). In this method, we extract the superficial features from the original image. The problem with this method is that it is not very robust. Another approach is deep-learning based. Deep-learning based methods exploits much more powerful feature present in the image, and they are much more effective than the traditional algorithms such as the one mentioned before. The robustness we achieve due to the Deep learning based algorithms comes at a rather high computational complexity, which costs us high latency in our system. Designs targeted for real-time systems can't afford that increment in latency due to high time-complexity of such algorithms. To cater to this latency issue, we mostly use GP-GPU. Usage of GPU might resolve the latency issues, but they consume a lot of power as well, making it unsuitable for the its applications in handheld mobile phones and embedded computing devices. However, to achieve low latency and low consumption of electrical power FPGAs is a more suitable approach. The purpose of this work is to get low latency implementations of the facial recognition system with lower power consumption.