The demand to replace Fe-V steel with Fe-Nb steel is evolving because of high costs of raw vanadium material. For the mass production of Fe-Nb steel, the most critical barrier is a poor impact toughness comparing with that of Fe-V steel. This study covers a microstructural investigation for ferrite grain size to explain the strength and toughness results as a function of V and Nb contents. The steel samples were made of three different compositions, i.e., Fe-V steel (Fe-0.05V-0.001Nb), Fe-V-Nb steel (Fe-0.014V-0.03Nb), and Fe-Nb steel (Fe-0.003V-0.033Nb). Rolling temperature to initiate was 1150°C for the all experiments. However, rolling temperature to finish was set differently for two conditions; 950°C and 860°C. The rolling to 860°C decreased the grain size for the ferrite phase and increased the impact toughness rather than the case of 950°C. The Fe-V-Nb steel exhibited similar value of the impact toughness with that for the Fe-V steel because of the low rolling temperature to finish, i.e., 860°C. The whole replace of V with Nb decreased the impact toughness significantly, however some extent of V content remained with Nb content showed the comparable toughness to the Fe-V steel by optimizing the controlled rolling process.