Friday, July 05, 2013

EM interesting feature - Deploying Oneoff patches to Agents

Enterprise Manager 12c Already allows for deploying one-off patches thru the provisioning and patching module but what  if you need to deploy a lot of agents and don't want to keep patching them after the fact. EM has  a new feature that allows for keeping management agent as well as plugin patches on the OMS for a particular agent version and the patches will automatically be applied when the agent  or plugin is deployed or upgraded. This is particularly a useful feature when you need to deploy or upgrade in bulk and have to apply one off patches on the environment as well.

The documentation for this feature is available in Technet EM Docs. The feature allows for generic patches by  putting them on each OMS
. In case of a Multi OMS this needs to be done on all OMS's Create a directory like below

On deployment or upgrade. The Patches will automatically  be  applied. The patches can be validated using the usual methods . Either by looking at the Manage Cloud Control --> Agents Screens or  Opatch lsinventory on the agent.

This is a very useful feature and will allow for rapid deployment and upgrades for agents without having to worry about applying one-off patches later.

Wednesday, June 05, 2013

EM12c Disk Busy Alert for Oracle Database Appliance V1 & X3-2

Oracle Just Published a Document ID 1558550.1 that talks about an issues that i've had an SR out for 6 months now.
Due to a Linux iostat bug  BUG: 1672511 (unpublished)  - oda - disk device sdau1 & sdau2 are 100% busy due to avgqu-sz value
This forces host level monitoring to report Critical Disk Busy alerts . This Bug will be fixed in an upcoming release of the the Oracle Database Appliance Software. 
This workaround is to disable Disk Activity Busy alert in EM12c. After the issue is resolved the user now has the responsibility to remember to reenable this alert.

  The alert in the document makes me laugh though 

Note:  Once you apply the iostat fix through an upcoming ODA release, make sure that you re-enable this metric by adding the Warning and Critical threshold values and applying the changes.

Tuesday, May 28, 2013

Patching an Exadata Compute Node

An Oracle Exadata Full Rack consists of 8 DB compute nodes. Oracle has shifted the strategy to patching the exadata in onwards to using Yum as the method of patching. the Quarterly Full Stack  does not include the DB Compute nodes patches anymore. So that has to be done separately. 
So where does someone start when they are new to exadata and need to patch to a newer release of the Software.
For the Compute Nodes Start here
Exadata YUM Repository Population, One-Time Setup Configuration and YUM upgrades [ID 1473002.1]

This note walks you thru either setting up a direct connection to ULN and building a repository or using an ISO image that you can down for setting up the rpeository. Best Practice would be to setup a repository external to the Exadata and then add the repo info in the Exadata compute nodes. Once the repository is created and updated or ISO downloaded. you will need to create
name=Oracle Exadata DB server 11.2 Linux $releasever - $basearch - latest
This needs to be added to all Exadata Compute Nodes . then ensure all repositories are disabled to avoid any accidents
sed -i 's/^[\t ]*enabled[\t ]*=[\t ]*1/enabled=0/g' /etc/yum.repos.d/*
Download and stage patch patch 13741363 in a software directory of each node This will have the helper scripts needed . Always make sure to get the updated versions. You will need to disable and stop the crs on the node you are patching as root and then perform a server backup .
$GRID_HOME/bin/crsctl disable crs
$GRID_HOME/bin/crsctl stop  crs -f
This will providecreate a backup and results similar to below will show up.
INFO] Unmount snapshot partition /mnt_snap
[INFO] Remove snapshot partition /dev/VGExaDb/LVDbSys1Snap
  Logical volume "LVDbSys1Snap" successfully removed
[INFO] Save partition table of /dev/sda in /mnt_spare/part_table_backup.txt
[INFO] Save lvm info in /mnt_spare/lvm_info.txt
[INFO] Unmount spare root partition /mnt_spare
[INFO] Backup of root /dev/VGExaDb/LVDbSys1 and boot partitions is done successfully
[INFO] Backup partition is /dev/VGExaDb/LVDbSys2
[INFO] /boot area back up named boot_backup.tbz (tar.bz2 format) is on the /dev/VGExaDb/LVDbSys2 partition.
[INFO] No other partitions were backed up. You may manually prepare back up for other partitions.
Once The backup is complete you can proceed with the update
yum --enablerepo=exadata_dbserver_11.2_x86_64_latest  repolist   // thisis the official channel for all updates 
yum --enablerepo=exadata_dbserver_11.2_x86_64_latest update
This will Download the appropriate rpm's and update the compute and reboot. The process can take between 10-30 mins . Once the node is up the Clusterware will not come up. Validate the image using imageinfo
[root@exa]# imageinfo
Kernel version: 2.6.32-400.21.1.el5uek #1 SMP Wed Feb 20 01:35:01 PST 2013 x86_64
Image version:
Image activated: 2013-05-27 14:41:45 -0500
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys1
This confirms that the compute node has been upgraded to Unlock crs as root
$GRID_HOME/crs/install/ -unlock
su - oracle 
--select oracle database to set home
relink all
make -C $ORACLE_HOME/rdbms/lib -f ipc_rds ioracle
su root
$GRID_HOME/crs/install/  -patch 
$GRID_HOME/bin/crsctl enable crs
This concludes a compute node patch application.  Rinse and repeat for all compute nodes 8 in X2-8
Now if you have read thru all this you will kind of see how many manual steps are involved. Fortunately Oracle Just Released a utility ot automate all these Tasks for you. Rene Kundersma of Oracle Talks about this new utility Call in his Blog Post Here
 Andy Colvin has published on his Blog his take on these Scripts and a demo Here

Tuesday, May 07, 2013

Oracle Database Appliance 2.6 Is now Available

Oracle Database Appliance Software version 2.6 is now available for Download. ODA 2.6 is the first version that contains combined software for ODA V1 and ODA X3-2.
This release also has an offline configuration Tool that will work with Virtualized and Non Virtualized ODA Configurations. This provides a lot of help in upfront planning and configuration of the Database Appliance.
The offline configurator is available on Oracle Technology Network and can be downloaded here.
The Configurator now asks a set of Questions like Environment and Hardware to determine deployment and network structure to use.
From the patching perspective ODA 2.6 Offers a few enhancements
For the Virtualized Platform
1. Remote Template Support
2. Support for Assemblies i.e .ova Support
3. a GUI VM Console can be accessed via oakcli
For Bare Metal (aka Non Virtualized)
1. SAP Application deployment is supported
2. PSU Patch
3. Unified Patch for Both V1 and X3-2

I'm still testing the patch and will  put up something on how to patch a virtualized ODA as well as Baremetal steps shortly
The Information Center and various notes on MOS still have not been updated yet with ODA 2.6 information.
As always please test before deploying to production.

Tuesday, April 09, 2013

Collaborate 13 Presentation - Oracle Database Appliance RAC In a box some strings attached.

I just uploaded my presentation on Oracle Database Appliance  - RAC In a box some strings attached. Thank you for all the people that attended and i hope i was able to convey my points about the Database Appliance in a manner that was objective and factual

Tuesday, March 26, 2013

Collaborate 2013 - Lots Of learning Lots of Fun

Collaborate 2013 is right around the corner and that is the place to be this April in Denver.
 There are a lot of Learning and a lot of social Activities. I'm goign to be down there presenting an Paper on Oracle Database Appliance .
I will be talking about how to deploy the Database Appliance and how it has changed how i do my day to day work. If you are interested in listening to me talk about Oracle Database Appliance , How it works , how easy it is to deploy etc . Please  join me at
There will be a lot of Tracks and Sessions and even time with a lot of influential People.
Below is a sneak peak of what i will be talking about. So if you feel that the Topic interests you. Please Come and join me or just come by to say Hi.

Moving Datafile from Physical Standby to Primary via Rman

Rman is an interesting tool and it seems everyday you learn something new. As part of a production issue we lost a diskgroup . This Diskgroup was only a small subset of the actual data but due to an issue in the underlying disk (External Redundancy). The database crashed. As we were debugging the issue with our vendor . We decided to use Rman and our Physical Standby to bring back the datafiles to help mitigate the outage. Now we could have failed over to our standby but since  our Standby hardware was not sized to handle production workload that was a risk the business was not willing to take.
As Oracle 11.1 you can use file from a standby(physical) database and move them to the primary.
We went thru to the Standby and determined which datafiles were in the particular group and then connected to primary from the standby as an auxiliary and copied the files over the network to the different diskgroup.

connect sys@standby AUXILIARY sys@primary;
Once the Files are copied you can switch the datafiles .

This will allow for a move of the datafile as well as rename on the primary.
you will then  have do a crash recovery on the datafile(s) in question.

recover database 2;
alter database datafile 2 online;
you have just copied a file over the network and plugged it into your primary from standby. This is a pretty convenient if accidents happens.

Tuesday, February 26, 2013

Changing Enterprise Manager 12c Default inactive Timeout

Oracle Enterprise defaults to a timeout of 45 minutes . Depending on the organization or security policies you might want to change that to a lesser amount of time or in my case a longer period .
To check what the timeout is set
em@emap1>./emctl get property -name oracle.sysman.eml.maxInactiveTime
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
SYSMAN password:
Value for property oracle.sysman.eml.maxInactiveTime for oms All Management Servers is null    null = default of 45 minutes
After you have checked the value . you can change it
em@emap1>./emctl set property -name oracle.sysman.eml.maxInactiveTime -value 90
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
SYSMAN password:
Property oracle.sysman.eml.maxInactiveTime has been set to value 90 for all Management Servers
OMS restart is required to reflect the new property value  
This will set the inactive time to 90 minutes . you will need an OMS bounce for this to take effect. Please ensure that whatever value you set meets your requirements and the security requirements of your company

Monday, February 11, 2013

Deploying a Virtualized Oracle Database Appliance ODA 2.5

So I finally got the opportunity  to deploy ODA 2.5 on a development Oracle Database Appliance. The Documentation  is very lacking and needs more refinement for the masses.
Here are the steps to deploy ODA 2.4 by reimaging the ODA . Concepts and procedures for bare maetaling the box remain the same.
    1. Use the ILOM to connect to the box via remote control and mount the VM Image  ( Need to do this on both hosts individually)
    2. After the reboot . The imaging Process will start and should takes between 1-2 hrs (Took about 2 hrs 30 minutes for me)
    3. Once you get the Oracle VM Server 3.1.1 screen. Your box has been imaged with a dom0 image.
    4. If you are using the ILOM you can ALT-F2 to get a login prompt
    5. Login as root
    6. Ensure that both boxes have been reimaged before starting the next step
    7. Run /opt/oracle/oak/bin/oakcli configure firstnet  ( on the first node)
    8. You have 2 options  (Local or Global)
    9. Global should be selectedi f both nodes are ready ot be ip'ed
    10. Select the network net1 , net2, net3, net4)
    11. Please note it is a little confusing but Here is a break down
        a.  priv1=bond0 (Interconnect)
        b.  Net1=bond1
        c. Net2=bond2
        d. Net3=bond3
        d. Net4=xbond0
    12. Oracle Failed to mention this but on startup does provide the MAC Addresses as well as ethernet names and bond info so be careful and ensure that  you understand your network topology prior to installing.
    13. You do want to make sure you have a DNS entry and a new IP Address for the DOM0 for each Server Node (2 x Dom0)
    14. Needless to say the network should be same on both nodes e.g public should be cabled on net1 on both nodes for consistency
    15. The network config will configure the public on both nodes  for Dom0 only
    16. After the config scp patch 16186172 into /OVS on the Dom0 box 0
    17. Unzip the patch 1 & 2 files and cat them together  cat a b >templates.
    18. Deploy the oda_base
 [root@odadb1-dom0 bin]# ./oakcli deploy oda_base
 Enter the template location: /OVS/templateBuild-2013-01-15-08-53.tar.gz
 Core Licensing Options:
         1. 2 CPU Cores
         2. 4 CPU Cores
         3. 6 CPU Cores
         4. 8 CPU Cores
         5. 10 CPU Cores
         6. 12 CPU Cores
         Selection[1 : 6]        : 5
         ODA base domain memory in GB(min 8, max 88)[default 80] :
 INFO: Using default memory size i.e. 80 GB
 INFO: Node 0
 INFO: Deployment in non local mode
 INFO: Running the command to copy the template /OVS/templateBuild-2013-01-15-08-                                                            53.tar.gz to remote node 1
 templateBuild-2013-01-15-08-53.tar.gz         100% 4620MB  47.6MB/s   01:37
 INFO: Node 0
 INFO: Spawned the process 26679 in the deployment node 0
 INFO: Trying to setup on deployment node 0
 INFO: Spawned the process 26680 in the node 1
 INFO: Trying to setup on node 1
 Using config file "/OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1/vm.cfg"                                                            .
 Started domain oakDom1 (id=1)
 INFO: Deployment in local mode
 INFO: Node 1
 INFO: Extracted the image files on node 1
 INFO: Node 1
 INFO: The VM Configuration data is written to /OVS/Repositories/odabaseRepo/Virt                                                            ualMachines/oakDom1/vm.cfg file
 INFO: Running /sbin/losetup /dev/loop0 /OVS/Repositories/odabaseRepo/VirtualMach                                                            ines/oakDom1/System.img command to mount the image file
 INFO: Mount is successfully completed on /dev/loop0
 INFO: Making change to the /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1                                                            /tmpmnt/boot/grub/grub.conf file
 INFO: Node 1
 INFO: Node 1
 INFO: Assigning IP to the second node...
 INFO: Node 1
 INFO: Created oda base pool
 INFO: Starting ODA Base...
 Using config file "/OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1/vm.cfg"                                                            .
 Started domain oakDom1 (id=1)
 INFO: Deployment in local mode
 INFO: Node 0
 INFO: Extracted the image files on node 0
 INFO: Node 0
 INFO: The VM Configuration data is written to /OVS/Repositories/odabaseRepo/Virt                                                            ualMachines/oakDom1/vm.cfg file
 INFO: Running /sbin/losetup /dev/loop0 /OVS/Repositories/odabaseRepo/VirtualMach                                                            ines/oakDom1/System.img command to mount the image file
 INFO: Mount is successfully completed on /dev/loop0
 INFO: Making change to the /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1                                                            /tmpmnt/boot/grub/grub.conf file
 INFO: Node 0
 INFO: Node 0
 INFO: Assigning IP to the first node...
 INFO: Node 0
 INFO: Created oda base poo1
 INFO: Starting ODA Base…
19. Once oda_base is deployed
 [root@podadb31-dom0 bin]# ./oakcli show oda_base
 ODA base domain
 ODA base CPU cores      :10
 ODA base domain memory  :80
 ODA base template       :/OVS/templateBuild-2013-01-15-08-53.tar.g
20.Once the oda_base is installed you will have to vnc in using the dom0 port 5900 to get access to the database server ( Due to a bug you will need to vnc in on both servers first and press the press any key to continue).
21. Once logged in you will need to IP the oda_base
22.You can either use the /opt/oracle/oak/bin/oakcli configure firstnet ( Please note it detects VM Environment and gives eth1,eth2,eth3 and eth4 as options )
23. Better to use  ./oakcli deploy
24. Oakcli now has option to change the proxy port for ASR as wella s configure external ASR server
25. External ASR server needs a server name as well as port (no definition of what the port is supposed to be )
26.  Also due to a bug if vm manager is bounced   you will have to vnc in and hit "press any key to continue". you can see that below
 The deployment process has not changed and will follow the same deployment steps.
This is the first release of ODA on a virtualized platform and glitches are to be expected. but it does seems to have been rushed out .
Please feel free to comment or ask questions. I have only deployed a DOm0 and the ODA_base here but i will deploy an app shortly and post my experience

Update: Edited steps and changed the network names

Friday, January 25, 2013

Oracle Database Appliance 2.5 And Virtualization (Updated)

So the cats out of the bag and Oracle has finally announced that Jan 31st ODA 2.5 will be available. The ODA 2.5 is a pretty significant update to the ODA Platform. it will have 1. Virtualization as an option with OVM 3.1.1 2. Official External ASR management. So The virtualization is huge since it allows for using the capacity of the ODA for things other than the Database. The concept is a DOM0 and an ODA_BASE which is a privileged domain that ensures that The Database Appliance performs optimally. Intially the DOMU or user Domain will not have access to ASM Disks or ACFS mounts but that is something that will show up in a future release . The User Domain will not be patched as part of the Single Patching technique which mean if you go with virtualization the user domain needs to be patched independently. Also initially only 250GB extra slice is the only space available for the DOM0 and DOMU areas. This can bring in a little more complexity but from a DR perspective it helps in building idle capacity with a smaller footprint . The question that is important is should ODA deployments start with a Virtual deployment even though there might be no requirement for it initially. Oracle will ship the physical deployment and customers can bare metal it to be virtualized. Oracle Claims to be able to provide virtualization with ease of use but only time will tell. I am pretty excited about this since i can build virtualized boxes and do DR at a cheaper cost but the same horse power that i need. I do intend to test this and see how the ease of use to determine if that is my goign in stance for all new ODA deployments . Update: The Database Appliance has 500GB of Local Disk space which means DOM0 and DOMU will have access to 250GB and ODA_BASE will have 250GB available to it.

Google Search