The work presented here provides a generalized structure for modeling polycrystals from micro- to nano-size range. The polycrystal structure is defined in terms of the grain core, the grain boundary and the triple junction regions with their corresponding volume fractions. Depending on the size of the crystal from micro to nano, different types of analyses are used for the respective different regions of the polycrystal. The analyses encompass local and nonlocal continuum or crystal plasticity. Depending on the physics of the region dislocation-based inelastic deformation and/or slip/separation is used to characterize the behavior of the material. The analyses incorporate interfacial energy with grain boundary sliding and grain boundary separation. Certain state variables are appropriately decomposed into energetic and dissipative components to accurately describe the size effects. This new formulation does not only provide the internal interface energies but also introduces two additional internal state variables for the internal surfaces (contact surfaces). One of these new state variables measures tangential sliding between the grain boundaries and the other measures the respective separation. Additional entropy production is introduced due to the internal subsurface and contacting surface. A multilevel Mori-Tanaka averaging scheme is introduced in order to obtain the effective properties of the heterogeneous crystalline structure and to predict the inelastic response of a nanocrystalline material. The inverse Hall-Petch effect is also demonstrated. The formulation presented here is more general, and it is not limited to either polycrystalline- or nanocrystalline-structured materials. However, for more elaborate solution of problems, a finite element approach needs to be developed.