Oracle 11gR2 GI和DB安装目录权限属主被修改后的恢复方法

某位仁兄新装一套Oracle 11gR2 RAC的过程中，在GI的安装配置阶段遇到了安装目录无法写入的报错，于是他便将$GRID_HOME下所有目录和文件属主改成了grid:oinstall，将$GRID_HOME下所有目录和文件权限改成了757，将$ORACLE_HOME下所有的目录和文件权限改成了757，侥幸过了安装这一关，紧接着麻烦就找上门了：使用srvctl无法启动数据库，症状如下：$ /oracle/app/oracle/product/11.2.0/db_1/bin/srvctl start database -d shpboss -o open PRCR-1079 : Failed to start resource ora.shpboss.dbCRS-2503: Resource "ora.shpboss.db" is in UNKNOWN state and must be stopped firstCRS-2680: Clean of "ora.shpboss.db" on "qzp750707b" failedCRS-5802: Unable to start the agent process <--- 关键在于oracle agent无法启动---GI的alert.log：$GRID_HOME/log/qzp750707b/alertqzp750707b.log里显示2016-03-02 12:33:01.991:[crsd（9109618）]CRS-5828:Could not start agent "/oracle/app/grid/product/11.2.0/grid_1/bin/oraagent_oracle". Details at （:CRSAGF00130:） {1:54418:222} in /oracle/app/grid/product/11.2.0/grid_1/log/qzp750707b/crsd/crsd.log.---crsd.log里的日志就有点眼花缭乱了2016-03-02 12:33:01.992: [ CRSD][10539]{1:54418:222} {1:54418:222} Created alert : （:CRSAGF00130:） : Failed to start the agent /oracle/app/grid/product/11.2.0/grid_1/bin/oraagent_oracle2016-03-02 12:33:01.992: [ AGFW][10539]{1:54418:222} Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.shpboss.db 1 1] ID 4098:7972016-03-02 12:33:01.992: [ AGFW][10539]{1:54418:222} Can not stop the agent: /oracle/app/grid/product/11.2.0/grid_1/bin/oraagent_oracle because pid is not initialized2016-03-02 12:33:01.992: [ CRSPE][11824]{1:54418:222} Received reply to action [Start] message ID: 7972016-03-02 12:33:01.992: [ CRSPE][11824]{1:54418:222} RI [ora.shpboss.db 1 1] new internal state: [STABLE] old value: [STARTING]2016-03-02 12:33:01.993: [ CRSPE][11824]{1:54418:222} Fatal Error from AGFW Proxy: Unable to start the agent process2016-03-02 12:33:01.993: [ CRSPE][11824]{1:54418:222} Set LAST_SERVER to qzp750707b for [ora.shpboss.db 1 1]2016-03-02 12:33:01.993: [ CRSPE][11824]{1:54418:222} CRS-2674: Start of "ora.shpboss.db" on "qzp750707b" failed2016-03-02 12:33:01.994: [ CRSPE][11824]{1:54418:222} RI [ora.shpboss.db 1 1] new internal state: [CLEANING] old value: [STABLE]2016-03-02 12:33:01.994: [ CRSPE][11824]{1:54418:222} Sending message to agfw: id = 8982016-03-02 12:33:01.994: [ CRSPE][11824]{1:54418:222} CRS-2679: Attempting to clean "ora.shpboss.db" on "qzp750707b"2016-03-02 12:33:01.995: [UiServer][12081]{1:54418:222} Container [ Name: ORDER MESSAGE: TextMessage[CRS-2674: Start of "ora.shpboss.db" on "qzp750707b" failed] MSGTYPE: TextMessage[1] OBJID: TextMessage[ora.shpboss.db 1 1] WAIT: TextMessage[0]]2016-03-02 12:33:01.995: [ COMMCRS][12081]clscsendx: （1117e3430） Connection not active2016-03-02 12:33:01.995: [UiServer][12081]{1:54418:222} CS（1117e39b0）Error sending msg over socket.62016-03-02 12:33:01.996: [ AGFW][10539]{1:54418:222} Agfw Proxy Server received the message: RESOURCE_CLEAN[ora.shpboss.db 1 1] ID 4100:8982016-03-02 12:33:01.996: [ AGFW][10539]{1:54418:222} Starting the agent: /oracle/app/grid/product/11.2.0/grid_1/bin/oraagent with user id: oracle and incarnation:32016-03-02 12:33:02.021: [ AGFW][10539]{1:54418:222} Starting the HB [Interval = 30000, misscount = 6kill allowed=1] for agent: /oracle/app/grid/product/11.2.0/grid_1/bin/oraagent_oracle2016-03-02 12:33:02.022: [ AGFW][10539]{1:54418:222} Could not forward message [RESOURCE_CLEAN[ora.shpboss.db 1 1] ID 4100:898] to agent. /oracle/app/grid/product/11.2.0/grid_1/bin/oraagent_oracle is not running2016-03-02 12:33:02.022: [ AGFW][10539]{1:54418:222} Starting of the agent: /oracle/app/grid/product/11.2.0/grid_1/bin/oraagent with user id oracle is already in progress.2016-03-02 12:33:02.040: [UiServer][12081]{1:54418:222} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.（File: clsMessageStream.cpp, line: 2752016-03-02 12:33:02.040: [UiServer][12081]{1:54418:222} Container [ Name: ORDER MESSAGE: TextMessage[CRS-2679: Attempting to clean "ora.shpboss.db" on "qzp750707b"] MSGTYPE: TextMessage[3] OBJID: TextMessage[ora.shpboss.db 1 1] WAIT: TextMessage[0]]2016-03-02 12:33:02.040: [UiServer][12081]{1:54418:222} CS（1117e39b0）No connection to client.62016-03-02 12:33:02.041: [UiServer][12081]{1:54418:222} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.（File: clsMessageStream.cpp, line: 275重要的信息已用红色字体标出，大致的意思是，oracle用户下的agent进程无法启动导致ora.shpboss.db 这个DB资源无法启动，我们知道正常情况下srvctl 启动数据库时会同时在oracle用户启动一个名为oraagent_bin的agent进程，就像下面那样root@qzp750707b:/home/grid>ps -ef|grep oraagent grid 15663174 1 0 14:28:38 - 0:01 /oracle/app/grid/product/11.2.0/grid_1/bin/oraagent.bin oracle 10944808 1 0 14:28:44 - 0:03 /oracle/app/grid/product/11.2.0/grid_1/bin/oraagent.bingrid 20578696 1 0 14:27:32 - 0:00 /oracle/app/grid/product/11.2.0/grid_1/bin/oraagent.bin但事发当时却找不到oracle用户下的这个agent进程，只有grid用户下的两个。$GRID_HOME下的安装目录及文件权限比DB要复杂的多，常见的有两种属主：root:oinstall和grid:oinstall，找了另一个11.2.0.4的RAC环境作为参照，$GRID_HOME目录下应该有以下一些目录的属主是root:oinstall的：grid@qc740701a:/oracle/app/grid/product/11.2.0/grid_1>ls -lrt | awk "$3~/root/"drwxr-xr-x 17 root oinstall 4096 Jul 31 2014 crsdrwxr-x--- 3 root oinstall 256 Jul 31 2014 gnsddrwxr-xr-x 3 root oinstall 256 Jul 31 2014 osysmonddrwxr-xr-x 3 root oinstall 256 Jul 31 2014 ologgerddrwxr-xr-x 3 root oinstall 256 Jul 31 2014 ctssdrwxr-x--- 4 root oinstall 256 Jul 31 2014 crfdrwxrwxrwt 6 root oinstall 256 Jul 31 2014 authdrwxr-xr-x 3 root oinstall 12288 Jul 31 2014 libdrwxr-xr-x 2 root oinstall 16384 Jul 31 2014 bindrwxr-xr-x 4 root system 256 Jul 31 2014 tfa而如今这些目录都清一色刷成了grid.oinstall，以下命令自然就没有输出了root@qzp750707b:/oracle/app/grid/product/11.2.0/grid_1>ls -lrt | awk "$3=root"srvctl 命令启动db时无法吊起的agent进程对应的可执行文件oraagent.bin正是位于$GRID_HOME/bin目录下，这样的诡异报错不免驱动我先去解决权限，先把错误的权限改对了再说。如果仅是手动改这些目录的属主还能接受，也就十来个，关键是目录下还有子目录和文件，这些子目录和文件的属主也root.oinstall的，手工逐个修改肯定不现实，难道要重装GI？多番查找终于找到一个较为省力且可靠的方法：在PSU >=11.2.0.3.6的版本下可以通过root用户执行<GRID_HOME>/crs/install/rootcrs.pl -init来恢复GRID_HOME目录下部分权限被篡改的文件或者目录的权限（为什么仅是部分，不是全部后面再解释），这条命令执行用时很快，不到5秒钟就返回命令行提示符了root@qzp750707b:/home/grid>/oracle/app/grid/product/11.2.0/grid_1/crs/install/rootcrs.pl -init Using configuration parameter file: /oracle/app/grid/product/11.2.0/grid_1/crs/install/crsconfig_params <---这个配置文件是在安装GI时生成的，用来帮助rootcrs.pl寻找GRID_HOME路径root@qzp750707b:/home/grid>查看一下效果，的确都改过来了root@qzp750707b:/oracle/app/grid/product/11.2.0/grid_1>ls -lrt | awk "$3~/root/"drwxr-xr-x 17 root oinstall 4096 Feb 23 19:29 crsdrwxr-xr-x 3 root oinstall 256 Feb 23 19:29 osysmonddrwxr-xr-x 3 root oinstall 256 Feb 23 19:29 ologgerddrwxr-x--- 3 root oinstall 256 Feb 23 19:29 gnsddrwxr-xr-x 3 root oinstall 256 Feb 23 19:29 ctssdrwxr-x--- 4 root oinstall 256 Feb 23 19:29 crfdrwxr-xr-x 3 root oinstall 12288 Feb 23 19:29 libdrwxr-xr-x 2 root oinstall 16384 Feb 23 19:31 bin在另一个节点如法炮制，终于srvctl 能够正常启动了db了，这还不放心，在两个节点上都执行了一边crsctl stop crs->crsctl start crs，对GI做一次完整的停启操作，该启的资源都能起来，GI和DB的alert.log里都没有报错，悬着的心这才放下。之前我们曾提到rootcrs.pl -init运行耗时5秒钟就返回了，如果是对于GRID_HOME下所有的文件都检查并修正一边权限和属主5秒钟是远远不够的，这一点相信使用过chmod和chown的童鞋都有体会，经过我的进一步测试发现rootcrs.pl脚本修改的只是$GRID_HOME目录里这些子目录及其下的文件drwxr-xr-x 17 root oinstall 4096 Feb 23 19:29 crs drwxrwxr-x 5 grid oinstall 256 Feb 23 19:29 log drwxrwxr-x 9 grid oinstall 256 Feb 23 19:29 cv drwxr-xr-x 3 root oinstall 256 Feb 23 19:29 osysmonddrwxr-xr-x 3 root oinstall 256 Feb 23 19:29 ologgerddrwxr-x--- 3 grid oinstall 256 Feb 23 19:29 ohasd drwxr-x--- 3 grid oinstall 256 Feb 23 19:29 mdns drwxr-x--- 3 root oinstall 256 Feb 23 19:29 gnsd drwxr-x--- 3 grid oinstall 256 Feb 23 19:29 gipc drwxr-xr-x 3 root oinstall 256 Feb 23 19:29 ctssdrwxr-x--- 4 root oinstall 256 Feb 23 19:29 crfdrwxr-xr-x 3 root oinstall 12288 Feb 23 19:29 libdrwxr-xr-x 2 root oinstall 16384 Feb 23 19:31 bindrwxrwxr-x 5 grid oinstall 256 Feb 23 19:31 cdatadrwxr-x--- 6 grid oinstall 256 Feb 23 19:36 gpnpdrwxrwxr-x 5 grid oinstall 4096 Feb 25 13:07 cfgtoollogsdrwx--x--x 6 grid oinstall 256 Feb 23 19:02 cssdrwxr-x--- 7 grid oinstall 256 Feb 23 19:02 evm从名称上可以看出这些都是GI后台核心进程的工作目录，猜测rootcrs.pl -init的功能只是修复各GI组件密切相关目录与文件的权限，保证GI能够正常启动与停止，但是对于其它目录则不管不问，于是我们可以看到$GRID_HOME目录下仍有大部分目录的权限还处在757root@qzp750707b:/oracle/app/grid/product/11.2.0/grid_1>ls -lrttotal 176-rwxr-xrwx 1 grid oinstall 59 Feb 22 16:05 oraInst.locdrwxr-xrwx 3 grid oinstall 256 Feb 23 18:57 demodrwxr-xrwx 3 grid oinstall 256 Feb 23 18:57 csmigdrwxr-xrwx 6 grid oinstall 256 Feb 23 18:57 assistantsdrwxr-xrwx 6 grid oinstall 256 Feb 23 18:57 nlsdrwxr-xrwx 5 grid oinstall 256 Feb 23 18:57 mddrwxr-xrwx 7 grid oinstall 256 Feb 23 18:57 javavmdrwxr-xrwx 3 grid oinstall 256 Feb 23 18:57 hsdrwxr-xrwx 4 grid oinstall 256 Feb 23 18:57 hasdrwxr-xrwx 3 grid oinstall 256 Feb 23 18:57 diagnosticsdrwxr-xrwx 4 grid oinstall 256 Feb 23 18:57 owmdrwxr-xrwx 7 grid oinstall 256 Feb 23 18:57 orddrwxr-xrwx 4 grid oinstall 256 Feb 23 18:57 oracoredrwxr-xrwx 3 grid oinstall 256 Feb 23 18:57 wwgdrwxr-xrwx 5 grid oinstall 256 Feb 23 18:57 usm。。。。。还有，此处省略了为避免留下后遗症，我们需要将rootcrs.pl弃之不管的目录与文件的权限、属主也修复一下，怎么修复？MOS 1515018.1提供了现成的perl脚本，这个脚本使用方法很简单：从一台权限正常的服务器上抓取GRID_HOME、ORACLE_HOME下的所有文件与目录权限，生成shell脚本，然后在权限错误的主机上执行这个脚本，简单演示一下：<1> 先把permission.pl下载下来复制到一台权限正常的服务器上，并赋予执行权限，这台主机上必须要有perl的执行环境chmod u+x permission.pl<2> 抓取$ORACLE_HOME下所有目录与文件的属主、权限，可以使用oracle用户或者root用户执行./permission.pl /oracle/app/oracle/product/11.2.0/db_1Following log files are generatedlogfile : permission-Thu-Mar-10-14-25-31-2016Command file : restore-perm-Thu-Mar-10-14-25-31-2016.cmdLinecount : 38734生成了两个文件，ls -lrt-rw-r----- 1 oracle oinstall 7890011 Mar 10 14:25 restore-perm-Thu-Mar-10-14-25-31-2016.cmd-rw-r----- 1 oracle oinstall 4061205 Mar 10 14:25 permission-Thu-Mar-10-14-25-31-2016 其中permission*开头的是/oracle/app/oracle/product/11.2.0/db_1目录及其下的所有子目录与文件列表，例如：755 oracle oinstall /oracle/app/oracle/product/11.2.0/db_1640 oracle oinstall /oracle/app/oracle/product/11.2.0/db_1/oraInst.loc750 oracle oinstall /oracle/app/oracle/product/11.2.0/db_1/root.sh755 oracle oinstall /oracle/app/oracle/product/11.2.0/db_1/EMStage775 oracle oinstall /oracle/app/oracle/product/11.2.0/db_1/EMStage/PAF。。。省略部分内容restore*开头的包含了执行修改权限修复所需的脚本，例如：chown oracle:oinstall /oracle/app/oracle/product/11.2.0/db_1chmod 755 /oracle/app/oracle/product/11.2.0/db_1chown oracle:oinstall /oracle/app/oracle/product/11.2.0/db_1/oraInst.locchmod 640 /oracle/app/oracle/product/11.2.0/db_1/oraInst.locchown oracle:oinstall /oracle/app/oracle/product/11.2.0/db_1/root.shchmod 750 /oracle/app/oracle/product/11.2.0/db_1/root.shchown oracle:oinstall /oracle/app/oracle/product/11.2.0/db_1/EMStagechmod 755 /oracle/app/oracle/product/11.2.0/db_1/EMStagechown oracle:oinstall /oracle/app/oracle/product/11.2.0/db_1/EMStage/PAFchmod 775 /oracle/app/oracle/product/11.2.0/db_1/EMStage/PAF。。。省略部分内容<3> 抓取$GRID_HOME下所有目录与文件的属主、权限，必须使用root用户执行./permission.pl /oracle/app/grid/product/11.2.0/grid_1Following log files are generatedlogfile : permission-Thu-Mar-10-14-21-28-2016Command file : restore-perm-Thu-Mar-10-14-21-28-2016.cmdLinecount : 60115结果也生成了两个文件ls -lrt-rw-r----- 1 root system 13372037 Mar 10 14:24 restore-perm-Thu-Mar-10-14-24-03-2016.cmd-rw-r----- 1 root system 6805988 Mar 10 14:24 permission-Thu-Mar-10-14-24-03-2016<4> 在目标主机上执行restore*开头的两个脚本，root用户执行将restore*脚本复制到目标主机后，执行chmod u+x restore-perm-Thu-Mar-10-14-25-31-2016.cmd restore-perm-Thu-Mar-10-14-24-03-2016.cmd./restore-perm-Thu-Mar-10-14-25-31-2016.cmd./restore-perm-Thu-Mar-10-14-24-03-2016.cmd总结：在安装有GI的环境下，权限、属主是严格被设定的，任何对于它们的错误修改容易引发一系列的问题，而且这些问题往往都很诡异很难按照常规的思路去诊断。万一权限或属主被修改了可以通过rootcrs.pl -init及permission.pl进行修复，rootcrs.pl –init仅修复GI的核心目录，所以其修复速度较快，如果遇到GI无法启动的问题，建议首选这种方法以使GI能够快速启动，但其缺点在于无法全量的进行修复，GI虽然正常了，并不能保证之后的运行过程中不出现这样那样的问题，这时就需要permission.pl出场了，permission.pl的运行模式决定了源库（权限正确的库）与目标库（权限错误的库）间的软件版本尽可能的一致，所以源库一定要选好，否则问题会更糟，另外如果源、目标两个库的安装目录不一样还需要对permission*脚本作调整后再执行。补充说明一点rootcrs.pl –init是在PSU>11.2.0.3.6下执行的，如果PSU<11.2.0.3.6可以执行如下两条命令来实现同样的效果<GRID_HOME>/crs/install/rootcrs.pl -unlock<GRID_HOME>/crs/install/rootcrs.pl -patch更多Oracle相关信息见Oracle 专题页面 http://www.linuxidc.com/topicnews.aspx？tid=12本文永久更新链接地址