Handover management with Quality of Service (QoS) provision is the major challenge for the seamless mobility across heterogeneous wireless networks. In this paper, we propose the context-aware vertical handoff (VHO) management architecture with QoS provision, based on the extension of IEEE 802.21. In this proposed architecture, context information is collection from all layers of the protocol stack both mobile node side and network side as most as possible. Context-aware Adaptive Analytic Hierarchy Process Method (CAAHP) method is chosen to identify the most suitable network for the optimization of handover decision mechanism. For QoS provision, QoS Management Module is carefully designed to provide the various functionalities, especially, Dynamic QoS negotiation mechanism and Dynamic QoS Mapping mechanism to provide the QoS service continuity and adaptation. Finally, the Mobile-initiated Handover scheme integrating with our proposed architecture is detailedly illustrated through the messages and procedures.