Cell localization is still facing two unresolved challenges: 1) the large variability in cell size and shape, coupled with the heterogeneous intensity distribution of lightly stained cells, presents the bottleneck limiting accurate cell localization; 2) existing cell location map is unreasonable, which loses cell scale information and significantly affects the accuracy of cell counting. To address the challenges from the cell shape and heterogeneous intensity distribution, we propose a novel gradient-aware and shape-adaptive Difference-Deformable Convolution (DDConv), which can focus on the gradient information of lightly stained cells for overcoming the challenge of heterogeneous intensity distribution and meanwhile adaptively adjust the shape of the convolutional kernel for overcoming the challenge of the large variability in cell shape. Moreover, to solve the challenge of unreasonable location map, we propose a new cell location map, called Pseudo-Scale Instance (PSI) map. Our PSI map enables adaptively computing the scale information and associating it with each cell’s annotation, which addresses the unreasonable challenges in existing cell location map and advanced makes the model sensitive to the size of cells. We analyze and evaluate DDConv and PSI on two challenging cell localization tasks. Compared with existing methods, our proposed method has significantly improved the localization performance, setting a new benchmark for cell localization tasks.