Communications of the ACM
Technical Perspective: Balancing At All Loads
Credit: Getty, Freepnglogos
Large-scale distributed parallel computing has become necessary for handling machine learning and other algorithms with ever-increasing complexity and data requirements. For example, Google TensorFlow can execute distributed algorithms that require thousands of computing nodes to work simultaneously. However, computing systems suffer from random fluctuations in service times. Power management, software or hardware failures, maintenance, and resource sharing are the primary causes of service time variability. Failures and maintenance are inevitable, and power management is crucial for reducing energy consumption. Multiplexing and balancing many applications over shared hardware and software enable high resource utilization. Therefore, random service time fluctuations are inherent to computing environments, as much as noise is unavoidable in communication systems.
Distributed parallel computing provides simultaneous execution of smaller tasks that make up larger computing jobs. However, because of the random fluctuations in service, some tasks take much more time to complete. We call these tasks stragglers. Large-scale systems split jobs into many tasks. Thus, even small-variance random service times will, with high probability, result in a non-negligible number of straggling tasks. Redundancy in task execution has emerged as a powerful way to curtail the overall service variability for two reasons: It attains load balancing without monitoring task execution speeds, and it disposes of the need to move data between nodes quickly.
No entries found