Evaluating the performance and improving the usability of parallel and distributed word embedding tools

Abstract

The representation of words by means of vectors, also called Word Embeddings (WE), has been receiving great attention from the Natural Language Processing (NLP) field. WE models are able to express syntactic and semantic similarities, as well as relationships and contexts of words within a given corpus. Although the most popular implementations of WE algorithms present low scalability, there are new approaches that apply High-Performance Computing (HPC) techniques. This is an opportunity for an analysis of the main differences among the existing implementations, based on performance and scalability metrics. In this Dissertation, we present an interdisciplinary study that addresses resource utilization and performance aspects of known WE algorithms found in the literature. To improve scalability and usability we propose an integration for local and remote execution environments that contains a set of the most optimized versions. Utilizing these optimizations it is possible to achieve an average performance gain of 15x for multicores and 105x for multinodes compared to the original version. There is also a big reduction in the memory footprint compared to the most popular Python versions. Since an appropriated use of HPC environments may require expert knowledge, we also propose a parameter tuning model utilizing an Multilayer Perceptron (MLP) neural network and Simulated Annealing (SA) algorithm to suggest the best parameter setup considering the computational resources, that may be an aid for non-expert users in the usage of HPC environments.