Globally health consciousness is increasing among people nowadays leading to taking a balanced diet with nutritious food. People are showing interest in selecting required nutritious food from various food applications. Classifying food images is a challenging area in smart nutrition and kitchen-based applications. Food Image Classification models using Deep Learning produce prominent results with high computational resources. Most of the existing applications in this area are mobile applications to interact with common people. Hence deploying current Deep Learning models into mobile kind of devices grab the attention of researchers. To address this issue, we propose a Lightweight model using Binary Convolutional Neural Network (BCNN) for classifying food images. Our experimental results show a significant improvement compared to the competitive models.