Identifying breast cancer lesions with a portable diffuse optical tomography (DOT) device improves early detection, while avoiding otherwise unnecessarily invasive, ionizing, and expensive modalities such as CT, as well as enabling first line of care treatment efficacy. Critical to this capability is not just identification of lesions, but rather the complex problem of discriminating between malignant and benign lesions. To accurately capture the highly heterogeneous tissue of a cancer lesion embedded in healthy breast tissue with non-invasive DOT, multiple frequencies can be combined to optimize signal penetration and reduce sensitivity to noise. However, these frequency responses can overlap, capture common information, and correlate, potentially confounding reconstruction and downstream end tasks. We show that an orthogonal fusion loss of multi-frequency DOT can improve reconstruction. More importantly, the orthogonal fusion leads to more accurate end-to-end identification of malignant versus benign lesions, illustrating its regularization properties on the multi-frequency input space. With the line-of-care deployment of portable DOT probes requiring a severely constrained computational budget, we show that our raw-to-task model, for direct prediction of end task from signal, significantly reduces computational complexity without sacrificing accuracy, enabling lower latency and higher, real-time throughput in medical settings.