With the development of robot technology and the arrival of industry 4.0 era, society pays more attention to col- laboration and interaction between human and robots. However, safety is still main concern in the development of human-robot collaboration. In this paper, a novel real-time collision avoidance approach for mobile manipulator is proposed by considering the motion status of the human, which includes the relative minimum distance and velocity (both magnitude and direction) between the robot and the human. The distance and velocity of the human hand are first estimated online using a vision sensor, and then defined as danger factors in the potential function of the potential field. The novel potential function proposed in this paper considers not only the safety problem, but also the efficient problem, i.e., the manipulator can make smart control decision to avoid the collision according to the relative velocity in case of the cross over. To overcome the local minimum problem and choose a best motion direction, we propose a motion sampling mechanism for motion planning. For each sample, the robot calculates the potential function to evaluate the safety and efficiency, and chooses a direction which is best for avoidance. We finally demonstrate our idea on a real mobile manipulator platform in a simulated co-worker environment.