Abstract
We advocate a novel approach to grid computing that is based on a combination of classic” operating system level virtual machines (VMs) and middleware mechanisms to manage VMs in a distributed environment. The abstraction is that of dynamically instantiated and mobile VMs that are a combination of traditional OS processes (the VM monitors) and files (the VM state). We give qualitative arguments that justify our approach in terms of security, isolation, customization, legacy support and resource control, and we show quantitative results that demonstrate the feasibility of our approach from a performance perspective. Finally, we describe the middleware challenges implied by the approach and architecture for grid computing using virtual machines.
Introduction
The fundamental goal of grid computing is to seamlessly multiplex distributed computational resources of providers among users across wide area networks. In traditional computing environments, resources are multiplexed using the mechanisms found in typical operating systems. For instance, user accounts and time-sharing enable the multiplexing of processors, virtual memory enables the multiplexing of main memory, and file systems multiplex disk storage. These and other traditional multiplexing mechanisms assume that trust and accountability are established by a centralized administration entity. In contrast, multiplexing in a grid environment must span independent administrative domains, and cannot rely on a central authority. The level of abstraction upon which current grid middleware solutions are implemented is that of an operating system user. This approach suffers from the limitations of traditional user account models in crossing administrative domain boundaries. In practice, multiplexing at this level of abstraction makes it difficult to implement the security mechanisms that are necessary to protect the integrity of grid resources from untrusted, legacy codes run on general purpose operating systems by untrusted users. It also greatly complicates the management of accounts and file systems that are not suited for wide-area environments. Unfortunately, most applications need precisely these services. Virtual machines present the image of a dedicated raw machine to each user. This abstraction is very powerful for grid computing because users then become strongly decoupled from
a) the system software of the underlying resource, and
b) other users sharing the resource.
In terms of security, VMs ensure that an untrusted user or application can only compromise their own operating system within a virtual machine, not the computational resource (nor other VMs). In terms of administration, virtual machines allow the configuration of an entire operating system to be independent from that of the computational resource; it is possible to completely represent a VM “guest” machine by its virtual state (e.g. stored in a conventional file) and instantiate it in any VM “host”, independently of the location or the software configuration of the host.
Why Grid Computing with Classic VMs?
The high-level answer to this question is that classic virtual machines provide a new abstraction layer, with low overhead that offers functionality that greatly simplifies addressing many of the issues of grid computing.
Introduction
The fundamental goal of grid computing is to seamlessly multiplex distributed computational resources of providers among users across wide area networks. In traditional computing environments, resources are multiplexed using the mechanisms found in typical operating systems. For instance, user accounts and time-sharing enable the multiplexing of processors, virtual memory enables the multiplexing of main memory, and file systems multiplex disk storage. These and other traditional multiplexing mechanisms assume that trust and accountability are established by a centralized administration entity. In contrast, multiplexing in a grid environment must span independent administrative domains, and cannot rely on a central authority. The level of abstraction upon which current grid middleware solutions are implemented is that of an operating system user. This approach suffers from the limitations of traditional user account models in crossing administrative domain boundaries. In practice, multiplexing at this level of abstraction makes it difficult to implement the security mechanisms that are necessary to protect the integrity of grid resources from untrusted, legacy codes run on general purpose operating systems by untrusted users. It also greatly complicates the management of accounts and file systems that are not suited for wide-area environments. Unfortunately, most applications need precisely these services. Virtual machines present the image of a dedicated raw machine to each user. This abstraction is very powerful for grid computing because users then become strongly decoupled from
a) the system software of the underlying resource, and
b) other users sharing the resource.
In terms of security, VMs ensure that an untrusted user or application can only compromise their own operating system within a virtual machine, not the computational resource (nor other VMs). In terms of administration, virtual machines allow the configuration of an entire operating system to be independent from that of the computational resource; it is possible to completely represent a VM “guest” machine by its virtual state (e.g. stored in a conventional file) and instantiate it in any VM “host”, independently of the location or the software configuration of the host.
Why Grid Computing with Classic VMs?
The high-level answer to this question is that classic virtual machines provide a new abstraction layer, with low overhead that offers functionality that greatly simplifies addressing many of the issues of grid computing.
Definitions
A modern operating system uses multiprogramming, virtual memory, and file systems to share CPU, memory, and disk resources among multiple processes and users. Each process accesses the physical resources indirectly, through abstractions provided by the operating system. Contemporaneous to the development of these mechanisms was that of another resource-sharing approach, virtual machines. A virtual machine presents the view of a duplicate of the underlying physical machine to the software that runs within it, allowing multiple operating systems to run concurrently and multiplex resources of a computer - processor, memory, disk, and network. Virtual machines can be divided into two main categories: those that virtualize a complete instruction set architecture (ISA-VMs) including both user and system instructions, and those that support an application binary interface (ABI-VMs) with virtualization of system calls. An important class of virtual machines (“classic” VMs) consists of ISA-VMs that support same-ISA execution of entire operating systems. A classic virtual machine abstraction allows for great flexibility in supporting multiple operating systems and is the focus of this paper.
Advantages
Unlike conventional operating systems, classic VMs allow dynamic multiplexing of users onto physical resources at the granularity of a single user per operating system session, thereby supporting per-user VMconfiguration and isolation from other users sharing the same physical resource. In the remainder of this section we focus on a scenario where each dynamic instance of a classic VM is dedicated to a single logical user.
1. Security and isolation
2. Customization
3. Administrator privileges
4. Resource control
5. Performance considerations
The advantages of virtual machines are for naught if they can not deliver sufficient performance. Virtual machine monitors incur performance overheads when applications within a VM execute privileged instructions that must be trapped and emulated. These are typically issued by kernel code of “guest” VMs during system calls, virtual memory handling, context switches and I/O. User-level code within VMMs runs directly on hardware without translation overheads. The overall overhead incurred by VMs thus depends on system characteristics, including the processor’s ISA, the VMM architecture and implementation, and the type of workload running in the system.
Table 1. Macrobenchmark results. User, system and total times are reported for three scenarios: physical machine, VM with state in local disk, VM with state accessed via NFS-based grid virtual file system (PVFS). Overheads are calculated using execution times and the physical machine as reference. In the PVFS scenario, the physical and data servers are located at Northwestern University, while the image server is located at the University of Florida.
Figure 1. Microbenchmark results: slowdown of synthetic test task under presence of background load for twelve different scenarios.
Table 2. Average, standard deviation, minimum and maximum VM startup times. Virtual machine sessions are instantiated using globusrun (Globus 2.0 toolkit) within a LAN. Measurements have been taken across 10 samples. Time (in seconds) is measured as wall-clock execution time from the beginning to the end of the execution of globusrun.
Middleware challenges
1. Data management
Data management is a key technology for VM-based grid computing, enabling administrative decoupling of computation providers and users. Data management involves: the transfer of VM images so that a user’s virtual machine can be instantiated anywhere and migrated when necessary, and support for location-independent access to user files. With appropriate data management support, computation, state, and user data can reside in different domains.
The components of a virtual machine session are distributed across three different logical entities: image servers, which provide the capability of archiving static VM states; computation servers (or VM hosts), which provide the capability of instantiating dynamic VM images (or VM guests); and data servers, which provide the capability of storing user data. In this scenario, VM state information needs to be transferred from an image server to a VM host (where it is instantiated), and from a data server to the VM guest (where it is processed) as in Figure 2.
High performance data transfers: Fast and simple access to images and user data is critical. Current grid solutions, such as Globus [4, 1] and PBS [3] typically employ file-staging techniques to transfer files between user accounts in the absence of a common file system.
Figure 3. Architecture for a VM-based grid service. In 1- 6, a virtual machine (V4) is dynamically created by middleware front-end F on behalf of user X. This VM is dedicated to a single user. In another scenario, virtual machines V1, V2 are instantiated on P2 on behalf of a service provider S, and are multiplexed across users A, B, C and applications provided by S. The logical user account abstraction decouples access to physical resources (middleware) from access to virtual resources (end-users and services).
In the following we lay out initial software architecture for virtual machine grid computing by describing the life cycle of a VM within it.
1. A user X (or grid middleware F on their behalf) first consults an information service, querying for a VM future (a physical machine able to instantiate a dynamic VM) P that meets their needs.
2. If necessary, X also consults an information service to query for a VM image server I with a base O/S installation that meets their application needs. Alternatively, users may provide VM images of their own (e.g. a customized O/S installation).
3. The middleware then establishes a data session between the physical server P and the image server I to allow for the instantiation of a dynamic VM. This data connection can be established via explicit transfers (e.g. Grid FTP) or via implicit, on-demand transfers (e.g. a grid virtual file system, Figure 2).
4. Once the data session for image I is established, the user can negotiate with the physical machine the startup of a VM instance Vi (e.g. using Globus GRAM or SSH). The virtual machine Vi may start from a preboot (cold) state, or from a post-boot (warm) state stored as part of the image. In addition, upon startup, the VM is assigned an IP address (via DHCP, or by connecting to a virtual network).
5. Once the VM instance Vi is running and on the network, additional data sessions are established. These connect the O/S within Vi to application server A and to the user’s data server D. As previously, these sessions can be realized with explicit or implicit transfers (Figure 2).
6. The application executes in the virtual machine; if it is an interactive application, a handle is provided back to the user (e.g. a login session, or a virtual display session such as VNC)
Conclusions
Classic virtual machines support a grid computing abstraction where computation becomes decoupled from the underlying physical resources. In this model, entire computing environments can be represented as data (a large state) and physical machines can be represented as resources for instantiating data. This abstraction is powerful because it decouples the administration of computing users from the administration of resource providers. This simplifies addressing many issues in grid computing and provides a new layer at which to work. We have presented a qualitative argument for the use of virtual machines in grid computing and quantitative results that demonstrate the feasibility of this idea from a performance perspective. We then illustrated the middleware challenges that must be overcome to build grid computing on top of virtual machine monitors and described how we are addressing those challenges. Finally, we provided a description of our nascent software architecture and its integration with existing middleware to support a VM-based infrastructure for computational grids. The envisioned architecture builds upon virtual machines, applications, data and networks from which necessary resources can be provided to the services layer.
