The self energy of geometrically necessary dislocations (GNDs) in single crystals is considered to inevitably introduce a higher-order stress as the work conjugate to slip gradient. It is pointed out that this higher-order stress changes stepwise in response to in-plane slip gradient and thus explicitly influences the initial yielding of polycrystals. The self energy of GNDs is then incorporated into the strain gradient plasticity theory of Gurtin (2002). The resulting theory is applied to 2D and 3D model crystal grains of diameter D, leading to a D-1-dependent term with a coefficient determined by grain shape. This size effect term is verified using published experimental data of several polycrystalline metals. It is thus found that the D-1-dependent term is successful for predicting not only the grain size dependence of initial yield stress but also the dislocation cell size dependence of flow stress in the submicron to several micron range of D.