Abstract Mesoporous silica nanoparticles (MCM‐41) with an average diameter of ∼ 20 nm were synthesized by a sol‐gel method using binary surfactant system. Polystyrene (PS) composites containing mesoporous silica nanoparticles were prepared by in situ polymerization of styrene monomers. Similar in situ polymerized PS composites were prepared based on the modified silica functionalized with methyl and vinyl groups. The effects of silylation on thermal and rheological properties of the PS/silica composites are investigated. Of particular importance is that the in situ polymerization of monomers within the mesoporous silica may trap some polymer chains, if not all, thereby affording a greater physical interaction between polymer and the porous fillers, whereas the chemical modification of silica surface promotes the polymer–filler interaction, which in turn enhances the thermal stability of composites. © 2008 Wiley Periodicals, Inc. J Appl Polym Sci, 2009