Over 20,000 fluorescence lifetime images from 10 patients were collected using a fibre-based custom fluorescence lifetime imaging endomicroscopy (FLIM) system. During the data collection, various measuring conditions were applied, including exposure time, optical wavelength, and lifetime extraction approaches to obtain diverse results rich in spatial and spectral resolution. The data for further processing was chosen with exposure time of 6 and 20 ns, excitation bands of 490-570 and 594-764 nm, and RLD. In addition, there are some images with sizes different than 128x128. In order to avoid any artificial errors on the lifetime images during the processing, only the lifetime images with 128x128 resolution were selected. After the selection, there were 10,155 and 11,363 frames of cancer and normal tissues respectively, and each frame contained one intensity and one corresponding lifetime image.