A History-based resource manager for genome analysis workflows applications on clusters with heterogeneous nodes
Badosa, Ferran (Universitat Autònoma de Barcelona. Departament d'Arquitectura de Computadors i Sistemes Operatius)
Espinosa, Antonio (Universitat Autònoma de Barcelona. Departament d'Arquitectura de Computadors i Sistemes Operatius)
Acevedo Giménez, César Esteban (Universitat Autònoma de Barcelona. Departament d'Arquitectura de Computadors i Sistemes Operatius)
Vera Rodríguez, Gonzalo (Centre de Recerca en Agrigenòmica)
Ripoll Aracil, Ana (Universitat Autònoma de Barcelona. Departament d'Arquitectura de Computadors i Sistemes Operatius)

Date:	2019
Abstract:	Bioinformatics workflows require large amounts of resources and are commonly executed in clusters. Determining the adequate amount of resources for bioinformatics applications is a tricky matter, since the resource usage of a single application might vary substantially from one execution to the next. Resource management systems in clusters don't consider these variations and subsequent needs. As a result, the computing power offered by clusters is not harnessed properly, compromising both application performance and resource efficiency. To tackle these issues, we propose a History-Based Resource Manager for bioinformatics workflows applications running on clusters with heterogeneous nodes. The proposed resource manager features a prediction model that generates multiple performance predictions for each job under different combinations of cluster resources. Furthermore, the proposed resource manager includes a scheduling algorithm that considers the degree of multiprogramming of the nodes, scheduling combinations of applications for simultaneous same-node execution upon their compatibility. To test the proposed resource manager, we process two workloads formed by different amounts of workflows made up by common bioinformatics applications. Results prove that for the given cases, the proposed resource manager improves the performance obtained with SLURM, using First Come First Served policy. The proposal shows an average workflow makespan improvement range between 28 and 35%, averaging 32%, an average workflow efficiency improvement range between 75 and 83%, averaging 79%, and an average resource usage improvement range between 96 and 101%, averaging 99%. Furthermore, the proposed scheduling algorithm can improve the average workflow makespan by a range of values between 26 and 36%, averaging 31%, compared to Max-Min and Min-Min algorithms.
Grants:	Ministerio de Economía y Competitividad TIN2014-53234-C2-1-R
Rights:	Aquest document està subjecte a una llicència d'ús Creative Commons. Es permet la reproducció total o parcial, la distribució, la comunicació pública de l'obra i la creació d'obres derivades, fins i tot amb finalitats comercials, sempre i quan es reconegui l'autoria de l'obra original.
Language:	Anglès
Document:	Article ; recerca ; Versió publicada
Subject:	Resource manager ; Bioinformatics workflows ; Multivariate regression prediction ; Scheduling algorithm ; Resource sharing ; Slowdown ; Makespan
Published in:	International Journal of Parallel Programming, Vol. 47, Issue 2 (April 2019) , p. 317-342, ISSN 0885-7458

DOI: 10.1007/s10766-018-0600-z

26 p, 1.8 MB

The record appears in these collections:
Research literature > UAB research groups literature > Research Centres and Groups (research output) > Experimental sciences > CRAG (Centre for Research in Agricultural Genomics)
Articles > Research articles
Articles > Published articles