Abstract
Running complex neural networks for image classification and object detection requires huge amounts of hard drives, GPU and CPU computation and energy consumption. Running this network on the cloud which runs on servers will already have large amounts of space, GPU which will never be a problem. When it comes to running this neural network on smartphones is more challenging as smartphones are still having limited space, computation power and energy. To make this neural network to run on smartphones deep compression techniques can be applied which will prune parameters, quantize weights, but this technique has drawbacks like large CNNs accuracy being lowered, l1 and l2 regularization requires more iterations. The Google brain team had worked on CNN for smartphones called MNasNet. This MNasNet uses a neural architecture search using reinforcement learning with a factorized hierarchical search space to design CNN for smartphones. This MNasNet considers both accuracy and inference latency when designing CNN. The key contribution was to find the proper mobile CNN model that objectives to achieve high accuracy and high speed. When training the MNasNet from scratch for extracting complex features that can be used in generating predictions for smartphones is a difficult job. We are using knowledge distillation which improves the performance of the MNasNet model on mobile devices. We train the complex and large network which can extract important features and produce better predictions, we call this as teacher model. Then with the help of the teacher model, we try to train smaller MNasNet which can replicate the results of the teacher model. In this teacher and student model, we used teacher assistant which is a medium model that helps in mediating the larger model and smaller model. We were using the CIFAR-10 dataset.