Set partitions and permutations with restrictions on the size of the blocks and cycles are important combinatorial sequences. Counting these objects lead to the sequences generalizing the classical Stirling and Bell numbers. The main focus of the present article is the analysis of combinatorial and arithmetical properties of them. The results include several combinatorial identities and recurrences as well as some properties of their $p$-adic valuations.