Since in my organization the Database team own complete control of the Database Appliance. We have to learn Unix/Linux administration. Oracle Database Appliance while a pretty hands off experience still need some work if you need to manage and maintain password synchronization etc.
In that category I wrote a script that ssh'ed in the box moved the /etc/passwd and then scp'ed a new set of files.
As you can see the statement above is the problem itself. if you move the /etc/passwd you cannot authenticate which means you have in essence created a box that no one can login to via any means.
This is really bad situation since you now have a node with no access since no one can authenticate.
Fortunately for me i had a very good Sysadmin who rescued me from this pickle.
He download OEL ( Oracle Enterprise Linux) and was able to rescue the system.
I'm putting below the steps we took
----- Procedure to rescue start ---
Log into ILOM.
Select Remote Control - Redirection Tab.
Click on Launch Remote Console
Once connected the remote console:
Select Devices Menu Item and start redirection to a CDROM or CDROM image.
Select Keyboard Menu Item and cleck Control Alt Delete … to reboot the system.
Go back to the ILOM console.
Select Remote Control – Host Control Tab.
Change Next Boot Device to CDROM..
Go back to the Remote console and watch for the system to boot.
Do NOT hit any keys once its boots off of the CDROM or you may start the interactive install.
At the booted CDROM prompt type.
The will step through the rescue boot mode.
Select the appropriate regional responses when prompted.
Select ok to find and mount root filesystems as read/write.
These will be mounted under /mnt.
Once you get a shell prompt run df –h to confirm.
From here you will be able restore/replace corrupt or missing files.
--- Procedure to rescue Stop----
The Moral of the story is Be very careful with what you are trying to do . The intention might be right but the wrong execution can cause you a lot of grief.