Identifying breast cancer lesions with a portable diffuse optical tomography (DOT) device can improve early detection while avoiding otherwise unnecessarily invasive, ionizing, and more expensive modalities such as CT, as well as enabling pre-screening efficiency. Critical to this capability is not just the identification of lesions but rather the complex problem of discriminating between malignant and benign lesions. To accurately capture the highly heterogeneous tissue of a cancer lesion embedded in healthy breast tissue with non-invasive DOT, multiple frequencies can be combined to optimize signal penetration and reduce sensitivity to noise. However, these frequency responses can overlap, capture common information, and correlate, potentially confounding reconstruction and downstream end tasks. We show that an orthogonal fusion loss of multi-frequency DOT can improve reconstruction. More importantly, the orthogonal fusion leads to more accurate end-to-end identification of malignant versus benign lesions, illustrating its regularization properties in the multi-frequency input space. While the deployment of portable DOT probes requires a severely constrained computational budget, we show that our raw-to-task model, for direct prediction of the end task from signal, significantly reduces computational complexity without sacrificing accuracy, enabling a high real-time throughput, desiredin medical settings. Furthermore, our results indicate that image reconstruction is not necessary for unbiased classiication of lesions with a balanced accuracy of 77% and 66% on the synthetic dataset and clinical dataset, espectively, using the raw-to-task model. Code is available at https: //github.com/sfu-mial/FuseNet