A functional polyhedral oligomeric silsesquioxane (POSS) with eight vinyl groups was used to fabricate a POSS/bismaleimide (BMI) nanocomposite as a potential matrix for RTM processing, and its blend with a thermoplastic polyether-ether ketone (PEK-C) was prepared. A self-designed optical microscope with a hot stage has been performed to monitor phase separation process of the thermoplastic-thermoset blend. The phase separation and final morphology of POSS/PEK-C/BMI system were characterized in the research. SEM pictures of POSS/BMI nanocomposite exhibit that the vinyl POSS has significant improvement in toughness of the materials, and its eight vinyl groups lead to chemical bonding interface with BMI resin in the nanocomposite. Results of optical microscope indicate that the corporation of POSS in PEK-C/BMI blend postponed phase separation process. The morphologies show that POSS/PEK-C/BMI systems have a similar phase separation structure with PEK-C/BMI blend.