The adoption of new technologies such as the Internet of Things and Big Data has become one of the main challenges in the era of Industry 4.0. Industrial 4.0 systems can provide increased reconfigurability and flexibility, in particular regarding Cyber-Physical Systems that integrate advanced artificial intelligence (AI) and fast-communication systems to provide real-time decisions. However, adoption of these new technologies in real-world applications is typically hindered by the complexity introduced by a multitude of legacy and state-of-the-art hardware and software components that need to be connected and need to communicate with each other. Consulting previously published approaches and analysing three real industrial use cases, this paper proposes a novel software-and-hardware architecture design for AI-driven Industrial 4.0 systems to facilitate the transition towards such smart-connected systems.