Home‎ > ‎

Oracle Database on VMware - Best Practices

Introduction 

This Oracle Databases on VMware Best Practices Guide provides best practice guidelines for deploying Oracle databases on VMware vSphere . The recommendations in this guide are not specific to any particular set of hardware, or size and scope of any particular Oracle database implementation. The examples and considerations provide guidance, but do not represent strict design requirements.

vSphere

VMware virtualization solutions provide numerous benefits to DBA administrators. VMware virtualization creates a layer of abstraction between the resources required by an application and operating system, and the underlying hardware that provides those resources. This abstraction layer provides value for the following:

  • Consolidation – VMware technology allows multiple application servers to be consolidated onto one physical server, with little or no decrease in overall performance. 
  • Ease of provisioning – VMware virtualization encapsulates an application into an image that can be duplicated or moved, greatly reducing the cost of application provisioning and deployment. 
  • Manageability – Virtual machines can be moved from server to server with no downtime using VMware vSphere vMotion, which simplifies common operations such as hardware maintenance, and reduces planned downtime. 
  • Availability – If an unplanned hardware failure occurs, VMware vSphere High Availability (HA) restarts affected virtual machines on another host in a VMware cluster. With VMware HA you can reduce unplanned downtime and provide higher service levels to an application. VMware vSphere Fault Tolerance (FT) features zero downtime and zero data loss, providing continuous availability in the face of server hardware failures for any application running in a virtual machine.

Production Support for Oracle Databases on vSphere 

Oracle has a support statement for VMware products that is honored around the world. While there has been much public discussion about Oracle’s perceived position on support for VMware virtualization, our experience is that Oracle Support upholds its commitment to customers, including those using VMware virtualization in conjunction with Oracle products.

ESX Host Guidelines

General

Create a computing environment optimized for vSphere.

Justification - The VMware ESX or VMware ESXi host BIOS settings can be specifically adjusted to maximize compute resources (such as disabling unnecessary processes and peripherals) to Oracle databases.

Create golden images of optimized operating systems using vSphere cloning technologies.

Justification - After the operating system has been prepared with the appropriate patches and kernel settings, Oracle can be installed in a virtual machine the same way it is installed on a physical system. This speeds up the installation of a new database.

BIOS Settings

Virtualization Technology - Yes, Necessary to run 64-bit guest operating systems.

Turbo Mode - Yes, Balanced workload over unused cores.

Node Interleaving - No, Disables NUMA benefits if disabled.

VT-x, AMD-V, EPT, RVI - Yes, Hardware-based virtualization support.

C1E Halt State - No, Disable if performance is more important than saving power.

Power-Saving -  No, Disable if performance is more important than saving power.

Virus Warning - No, Disables warning messages when writing to the master boot record.

Hyperthreading  - Yes, For use with some Intel processors. Hyperthreading is always recommended with Intel’s newer Core i7 processors such as the Xeon 5500 series.

Video BIOS Cacheable - No, Not necessary for database virtual machine.

Wake On LAN -  Yes, Required for VMware vSphere Distributed Power Management feature.

Execute Disable - Yes, Required for vMotion and VMware vSphere Distributed Resource Scheduler (DRS) features.

Operating System Host Processes

VMware recommends disabling unnecessary foreground and background processes within the guest operating system.

  • Examples of unnecessary Linux processes are: anacron, apmd, atd, autofs, cups, cupsconfig, gpm, isdn, iptables, kudzu, netfs, and portmap. 
  • Examples of unnecessary Windows processes are: alerter, automatic updates, clip book, error reporting, help and support, indexing, messenger, netmeeting, remote desktop, and system restore services. 
  • For Linux installs, the database administrator (DBA) should request that the system administrator compile a monolithic kernel to load only the necessary features. Whether you intend to run Windows or Linux as the final optimized operating system, these host installs should be cloned by the VMware administrator for reuse. 
  • After the operating system has been prepared, install Oracle the same way as for a physical environment. Use the recommended kernel parameters listed in the Oracle Installation guide. Also, it is a good practice to check with Oracle Support for the latest settings to use prior to beginning the installation process.

Memory

Set memory reservations equal to the size of the Oracle SGA.

Justification - The memory reservation should be large enough to avoid kernel swapping between ESX and the guest OS because Oracle databases can be memory intensive.

Use large memory pages.

Justification - Large page support is enabled by default in ESX 3.5 and later, and is supported from Oracle 9i R2 for Linux operating systems and 10g R2 for Windows. Enable large pages in the guest OS to improve the performance of Oracle databases on vSphere

Virtual CPU

Use as few virtual CPUs (vCPUs) as possible.

Justification - If monitoring of the actual workload shows that the Oracle database is not benefitting from the increased virtual CPUs, the excess vCPUs impose scheduling constraints and can degrade overall performance of the virtual machine.

Enable hyperthreading for Intel Core i7 processors.

Justification - With the release of Intel Xeon 5500 series processors, enabling hyperthreading is recommended.

Storage Guidelines

Enable jumbo frames for IP-based storage using iSCSI and NFS.

Justification - Jumbo frames enable Ethernet frames to have a larger payload, allowing for improved performance.

Create dedicated datastores to service database workloads.

Justification - The creation of dedicated datastores for I/O-intensive databases is analogous to provisioning dedicated LUNs in the physical world. This is a typical design for a mission-critical enterprise workload. 

Use VMware vSphere VMFS for single instance Oracle database deployments.

Justification - To balance performance and manageability in a virtual environment, deploy Oracle using VMFS.

Use your storage vendor’s best practices documentation when laying out the Oracle database.

Justification - Oracle ASM cannot determine the optimal data placement or LUN selection with respect to the underlying storage infrastructure. For that reason, Oracle ASM is not a substitute for close communication between the storage administrator and the database administrator.

Storage Virtualization Concepts

VMware storage virtualization can be categorized into three layers of storage technology.

  • The storage array is the bottom layer, consisting of physical disks presented as logical disks (storage array volumes or LUNs) to the layer above. 
  • The next layer is the virtual environment occupied by vSphere. Storage array LUNs are presented to ESX/ESXi hosts as datastores and are formatted as VMFS volumes. 
  • Virtual machines consist of virtual disks that are created in the datastores and presented to the guest operating system as disks that can be partitioned and used in file systems.

Virtual Machine File System (VMFS)

VMFS is a cluster file system that provides storage virtualization optimized for virtual machines. Each virtual machine is encapsulated in a set of files and VMFS is the default storage system for these files on physical SCSI disks and partitions. VMFS allows multiple ESX/ESXi instances to access shared virtual machine storage concurrently. It also enables virtualization-based distributed infrastructure services such as vMotion, DRS, and VMware HA to operate across a cluster of ESX/ESXi hosts. 

Raw Device Mapping (RDM).

RDM allows a virtual machine to directly access a volume on the physical storage subsystem, and can be used only with Fibre Channel or iSCSI. RDM can be thought of as providing a symbolic link from a VMFS volume to a raw volume. The mapping makes volumes appear as files in a VMFS volume. The mapping file, not the raw volume, is referenced in the virtual machine configuration.

Storage Protocol Capabilities 

Type 
Boot VM 
Boot vSphere 
vMotion HA/DRS 
 VMFS 
RDM 
SRM 
Fibre Channel YesYesYesYesYesYes
iSCSI YesYesYesYesYesYes
NAS YesNoYesNoNoNo
Local StorageYesYesNoYesNoNo

Database Layout Considerations

Oracle Clustered File System (OCFS)

The Oracle Clustered File System is a POSIX-compliant shared disk cluster file system for Linux that can be used with Oracle Real Application Clusters. OCFS was the predecessor to Oracle ASM that was introduced in Oracle 10g. (Discussion of Real Application Clusters is beyond the scope of this guide.) ASM is the recommended clustering technology. Also, because ASM can also be used for single instance deployments, it provides an on-ramp to Real Application Clusters.

Consolidated or Dedicated Datastores

It is a generally accepted best practice to create a dedicated datastore if the application has a demanding I/O profile. Databases fall into this category. The creation of dedicated datastores allows DBAs to define individual service level guarantees for different applications and is analogous to provisioning dedicated LUNs in the physical world. A datastore is an abstraction of the storage tier and, therefore, it is a logical representation of the storage tier, not a physical representation of the storage tier. Creating a dedicated datastore to isolate a particular I/O workload (whether log or database files), without isolating the physical storage layer as well, does not have the desired effect on performance.

Virtual SCSI Controllers

VMware highly recommends using multiple virtual SCSI controllers for the database virtual machines or virtual machines with high I/O load. The use of multiple virtual SCSI controllers allows the execution of several parallel I/O operations inside the guest operating system. VMware also highly recommends separating the Redo/Log I/O traffic from the data file I/O traffic through separate virtual SCSI controllers. As a best practice, you can use one controller for the operating system and swap, another controller for DB Log, and one or more additional controllers for database data files (depending on the number and size of the database data files). 

VMFS versus RDM

Both VMFS and RDM volumes can provide similar transaction throughput. For more details, refer to Performance Characterization of VMFS and RDM Using a SAN (http://www.vmware.com/files/pdf/performance_char_vmfs_rdm.pdf)

VMware generally recommends VMFS, but there may be situations where RDMs are required.

VMFS 
RDM 
Volume can host many virtual machines (or can be dedicated to one virtual machine).Maps a single LUN to one virtual machine, so only one virtual machine is possible per LUN.
Increases storage utilization, provides better flexibility, easier administration and management.More LUNs are required, so it is easier to reach the LUN limit of 256 that can be presented to an ESX/ESXi host.
Can potentially support clustering software that does not issue SCSI reservations, such as Oracle Clusterware. To configure, follow the procedures given in Disabling simultaneous write protection provided by VMFS using the multi-writer flag (http://kb.vmware.com/kb/1034165).RDM might be required to leverage third party storage array-level backup and replication tools.
Oracle RAC node Live Migration.RDM volumes can help facilitate migrating physical Oracle databases to virtual machines. Alternatively, enables quick migration to physical in rare Oracle support cases.
 Required for MSCS quorum disks.

 

Networking Guidelines

Use the VMXNET family of Paravirtualized network adapters.

Justification - The Paravirtualized network adapters in the VMXNET family implement an optimized network interface that passes network traffic between the virtual machine and the physical network interface cards with minimal overhead.

Separate infrastructure traffic from virtual machine traffic for security and isolation.

Justification - Virtual machines should not see infrastructure traffic (security violation) and should not be impacted by infrastructure traffic bursts (for example, vMotion operations).

Use NIC teaming for availability and load balancing.

Justification - NIC teams can share the load of traffic among some or all of its members, or provide passive failover in the event of a hardware failure or a network outage.

Comments