Federated learning (FL) has emerged as a machine learning (ML) paradigm for distributed training without requiring trainers to share private data. Coevally, unmanned aerial vehicles (UAVs) become sufficiently powerful that they can contribute to training sophisticated ML models. Direct application of the conventional FL framework to a UAV swarm, however, could result in unnecessarily high communication and computation complexity. This paper aims to address this challenge by proposing a scalable and clustered FL framework for such large UAV swarms. Specifically, we develop a clustering scheme based on which the UAV-based wireless network is partitioned into different clusters coordinated by corresponding cluster-head UAVs forming a connected graph. While the cluster-head UAVs coordinate the ML model updates as in the conventional FL framework, we propose two intercluster model aggregation strategies to produce the final global model in each training round. Extensive numerical studies demonstrate the convergence and desirable trade-offs between training performance and communication efficiency of our proposed framework.