The recent widespread increase of the Mpox (formerly monkeypox) virus infections in the South Asian and African countries has raised concerns among medical professionals regarding the potential emergence of another pandemic in those regions. With the number of available test kits surpassing the count of positive/probable cases, there is a pressing need to develop a robust and lightweight classifier model to alleviate the burden of physical testing kits and expedite the detection process. The existing state-of-the-art primarily focuses on achieving high accuracy in modeling Mpox without considering factors such as modeling suitability, real-time inferencing, and adaptability to resource-constrained CPU-only mobile devices. In this research, we propose a novel lightweight binarized DarkNet53 model, referred to as BinaryDNet53, which is approximately ∼ 20× more computationally efficient and ∼ 2× more power-efficient than the current state-of-the-art. This model demonstrates smooth detection capabilities when deployed on small hand-held or embedded devices. Our work introduces large-margin feature learning and weighted loss calculation to enhance results, particularly on complex samples. We conduct experiments using the latest MSLD v2.0 dataset, showcasing the superiority of the proposed model over state-of-the-art models based on classification and computational metrics, including Watt power consumption, required memory, and GFLOPS.