Human action video retrieval is a useful tool for video surveillance and sports video analysis, among other applications. Previous work on image retrieval tasks has shown that latent semantic methods are an effective way to build a high-level representation of data to discover implicit relations between visual patterns, achieving a significant improvement on these tasks. The current paper evaluates the applicability of Non-Negative Matrix Factorization (NMF), a latent semantic method, on human action video retrieval. Experiments are carried out on common human action recognition datasets using state-of-the-art descriptors. We focus on evaluating the query by example approach i.e. only videos are used as queries. The performance of the method is compared against classic direct matching between video features.