In this paper, we propose WearVis, a generic framework for human-machine hybrid recognition based on wearable visual computing. The chief objective of WearViS is to help human complete the complex recognition task and let user flexibly interact with the visual assistant system. The framework process can be described in four steps. First, the behavior of human’s attention drivers WearVis search region of interest. Second, WearVis selects the most abundant features in image sequence. Then, WearVis detects and localizes task-related interest objects. Finally, WearVis select the natural interaction strategy according to the priority of recognition task. We describe the goals and architecture of WearViS, a prototype implementation that validates its design.