Hello everyone, it’s been a while that I don’t post anything here, so I am bringing this post about Exadata with OVM to take the webs off the blog and to contribute to the Oracle Developers Community Appreciation Day that Tim Hall from Oracle-Base organizes every year #ThanksODC 🙂
The intention here is to enable the VM Domains to increase their CPUs dynamically in this Exadata environment.
Changing the VM configuration file in the Exadata Compute Node
First of all you need to change the VM configuration file.
There are 2 CPU related parameters in this file. The cpu parameter is to provide the initial number of CPUs and the max_cpus is the maximum count of CPUs that can be assigned to the VM.
In the current scenario we have cpu and max_cpus parameters set to the same number of 12. So in order to be able to increase the CPU cores without the need of rebooting the VM we need to have max_cpus parameter with a bigger value than the cpu parameter.
Let’s set the max_cpus to 68 and shutdown the VM to have this vm.cfg file read again. Bare in mind that only rebooting the VM is not enough as the file will not be reread during the reboot.
Let’s check the current configuration in place for the VM:
[root@exadb3 ~]# xm list -l vm5.loredata.com.br |grep cpus (vcpus 12) (cpus (online_vcpus 12)
vcpus and online_cpus are the same so we cannot increase CPU cores online.
First thing, change max_cpus to 68 in the vm.cfg parameter file:
[root@exadb3 ~]# vi /EXAVMIMAGES/GuestImages/vm5.loredata.com.br/vm.cfg
acpi = 1
apic = 1
pae = 1
builder = 'hvm'
kernel = '/usr/lib/xen/boot/hvmloader'
device_model = '/usr/lib/xen/bin/qemu-dm'
# To make VMs with more than 12 vCPUs work on exadata server
# 1: Processor Info and Feature Bits
# This returns the CPU's stepping, model, and family information in EAX (also called the signature of a CPU),
# feature flags in EDX and ECX, and additional feature info in EBX.
# The format of the information in EAX is as follows:
# 3:0 - Stepping
# 7:4 - Model
# 11:8 - Family
# 13:12 - Processor Type
# 19:16 - Extended Model
# 27:20 - Extended Family
# Each register has 32 bits with 31st bit in the left end and 0 bit in the right end.
# edx register:
# 12 bit - Memory Type Range Registers. Force to 0 set uncached access mode to memory ranges.
# Each successive character represent a lesser-significant bit:
# '1' -> force the corresponding bit to 1
# '0' -> force to 0
# 'x' -> Get a safe value (pass through and mask with the default policy)
# 'k' -> pass through the host bit value
# 's' -> as 'k' but preserve across save/restore and migration
# 33222222222211111111110000000000
# 10987654321098765432109876543210
cpuid = ['1:edx=xxxxxxxxxxxxxxxxxxx0xxxxxxxxxxxx']
disk = ['file:/OVS/Repositories/eb37ee72b0e84062b2abc40cd67534eb/VirtualDisks/214ed0feb0a64e8ea7bd1bf96f8b4fac.img,xvda,w','file:/OVS/Repositories/eb37ee72b0e84062b2abc40cd67534eb/VirtualDisks/68140ae48d814c9f9afe453637053bbb.img,xvdb,w','file:/OVS/Repositories/eb37ee72b0e84062b2abc40cd67534eb/VirtualDisks/cba34f1848ba44b5b86c2a2a4dfb3ae7.img,xvdc,w','file:/OVS/Repositories/eb37ee72b0e84062b2abc40cd67534eb/VirtualDisks/85e41dddc2164434910b33e2280fc416.img,xvdd,w','file:/OVS/Repositories/eb37ee72b0e84062b2abc40cd67534eb/VirtualDisks/490bab94f8f343929bea4fd6328d708c.img,xvde,w','file:/OVS/Repositories/eb37ee72b0e84062b2abc40cd67534eb/VirtualDisks/392cb72d94aa411da152082c885be76b.img,xvdf,w']
memory = '65536'
maxmem = '65536'
OVM_simple_name = 'Exadata VM'
name = 'vm5.loredata.com.br'
OVM_os_type = 'Oracle Linux 6'
vcpus = 12
maxvcpus = 68
uuid = 'eb37ee72b0e84062b2abc40cd67534eb'
on_crash = 'restart'
on_reboot = 'restart'
serial = 'pty'
keymap = 'en-us'
vif = ['type=netfront,mac=00:16:3e:01:1f:8f,bridge=vmeth3.224','type=netfront,mac=00:16:3e:26:9d:ae,bridge=vmeth0','type=netfront,mac=00:16:3e:20:c4:f6,bridge=vmbondeth0.200']
timer_mode = 2
ib_pfs = ['03:00.0']
ib_pkeys = [{'pf':'03:00.0','port':'1','pkey':['0xffff',]},{'pf':'03:00.0','port':'2','pkey':['0xffff',]},]
Now let’s stop the database running in this VM:
[oracle@vm5 ~]$ srvctl stop database -d orcl
And shutdown the VM:
[root@exadb3 ~]# xm shutdown vm5.loredata.com.br -w Domain vm5.loredata.com.br terminated All domains terminated
Once shutdown we just start it up again with the correct vm.cfg file:
[root@exadb3 ~]# xm create /EXAVMIMAGES/GuestImages/vm5.loredata.com.br/vm.cfg -c Using config file "/EXAVMIMAGES/GuestImages/vm5.loredata.com.br/vm.cfg"
Once the CRS resources are back online start the database which was stopped earlier:
[oracle@vm5 ~]$ srvctl start database -d orcl
Let’s list if the change took place on the restart of the VM:
[root@exadb3 ~]# xm list -l vm5.loredata.com.br |grep cpus (vcpus 68) (cpus (online_vcpus 12)
Increasing the CPU cores online
Now if we want to increase the CPU cores for the VM online we just do:
[root@exadb3 ~]# xm vcpu-set vm5.loredata.com.br 36
Let’s check if it really changed:
[root@exadb3 ~]# xm list -l vm5.loredata.com.br |grep cpus (vcpus 68) (cpus (online_vcpus 36)
Cool! It worked as expected.
Just one more thing… once the CPUs are increased online there is still the need to change the cpu parameter in the vm.cfg file for the next restart. So let’s change it:
[root@exadb3 ~]# vi /EXAVMIMAGES/GuestImages/vm5.loredata.com.br/vm.cfg
acpi = 1
apic = 1
pae = 1
builder = 'hvm'
kernel = '/usr/lib/xen/boot/hvmloader'
device_model = '/usr/lib/xen/bin/qemu-dm'
# To make VMs with more than 12 vCPUs work on exadata server
# 1: Processor Info and Feature Bits
# This returns the CPU's stepping, model, and family information in EAX (also called the signature of a CPU),
# feature flags in EDX and ECX, and additional feature info in EBX.
# The format of the information in EAX is as follows:
# 3:0 - Stepping
# 7:4 - Model
# 11:8 - Family
# 13:12 - Processor Type
# 19:16 - Extended Model
# 27:20 - Extended Family
# Each register has 32 bits with 31st bit in the left end and 0 bit in the right end.
# edx register:
# 12 bit - Memory Type Range Registers. Force to 0 set uncached access mode to memory ranges.
# Each successive character represent a lesser-significant bit:
# '1' -> force the corresponding bit to 1
# '0' -> force to 0
# 'x' -> Get a safe value (pass through and mask with the default policy)
# 'k' -> pass through the host bit value
# 's' -> as 'k' but preserve across save/restore and migration
# 33222222222211111111110000000000
# 10987654321098765432109876543210
cpuid = ['1:edx=xxxxxxxxxxxxxxxxxxx0xxxxxxxxxxxx']
disk = ['file:/OVS/Repositories/eb37ee72b0e84062b2abc40cd67534eb/VirtualDisks/214ed0feb0a64e8ea7bd1bf96f8b4fac.img,xvda,w','file:/OVS/Repositories/eb37ee72b0e84062b2abc40cd67534eb/VirtualDisks/68140ae48d814c9f9afe453637053bbb.img,xvdb,w','file:/OVS/Repositories/eb37ee72b0e84062b2abc40cd67534eb/VirtualDisks/cba34f1848ba44b5b86c2a2a4dfb3ae7.img,xvdc,w','file:/OVS/Repositories/eb37ee72b0e84062b2abc40cd67534eb/VirtualDisks/85e41dddc2164434910b33e2280fc416.img,xvdd,w','file:/OVS/Repositories/eb37ee72b0e84062b2abc40cd67534eb/VirtualDisks/490bab94f8f343929bea4fd6328d708c.img,xvde,w','file:/OVS/Repositories/eb37ee72b0e84062b2abc40cd67534eb/VirtualDisks/392cb72d94aa411da152082c885be76b.img,xvdf,w']
memory = '65536'
maxmem = '65536'
OVM_simple_name = 'Exadata VM'
name = 'vm5.loredata.com.br'
OVM_os_type = 'Oracle Linux 6'
vcpus = 36
maxvcpus = 68
uuid = 'eb37ee72b0e84062b2abc40cd67534eb'
on_crash = 'restart'
on_reboot = 'restart'
serial = 'pty'
keymap = 'en-us'
vif = ['type=netfront,mac=00:16:3e:01:1f:8f,bridge=vmeth3.224','type=netfront,mac=00:16:3e:26:9d:ae,bridge=vmeth0','type=netfront,mac=00:16:3e:20:c4:f6,bridge=vmbondeth0.200']
timer_mode = 2
ib_pfs = ['03:00.0']
ib_pkeys = [{'pf':'03:00.0','port':'1','pkey':['0xffff',]},{'pf':'03:00.0','port':'2','pkey':['0xffff',]},]
That is all for today folks.
Hope you enjoy it. I’ll have more posts about Oracle VM in the next upcoming days.
Regards,
Franky