实例宕机引发的ORA-00240错误

1.环境描述OS：AIX6.1Oracle ：11.2.0.3.0 RAC2.事故发生数据库NODE 2所在小型机发生宕机事故，本应正常切换至NODE1，但切换失败，重启系统得以解决。3.事故分析事故第二天进行了数据库的alert log分析，从日志中可以看到数据库在实例NODE2发生宕机后，RAC已经做出了实例切换步骤，但在切换的过程中遭遇了ORA-00240、ORA-29770错误，导致当时数据库没有切换成功。下面是日志的详细分析。数据库在当时经历了大致以下几个重要步骤：
1. Beginning instance recovery of 1 threads数据库开始在本地恢复宕机的实例2. Started redo application atThread 2: logseq 4556, block 368380 数据库开始从在线redo日志序号为4556的日志恢复3. Completed instance recovery atThread 2: logseq 4556, block 376983, scn 3502123313 数据库redo日志4556已经恢复成功。4. Redo thread 2 internally disabled at seq 4557 （SMON）当数据库准备恢复4557的日志时，遭遇失败。5 ORA-00240: control file enqueue held for more than 120 seconds日志中开始出现ora-00240错误，提示控制文件被持有超过120秒。6 ORA-29770: global enqueue process DIA0 （OSID 12517556） is hung for more than 300 secondsIncident details in:紧接着出现ora29770错误，数据库进程DIA0 hung住超过5分钟。在当天实例宕机的20分钟左右时间内，当时系统的监控等脚本均无法执行，分析Oracle awr报告得知在当天宕机的时间内，未宕机的NODE1系统CPU资源几乎耗尽。经过分析与推测，DIA0进程的主要作用是处理数据库死锁、hung住的一个进程，日志中的现象应该是这个进程发现控制文件被持有超过120秒，属于错误行为，进程开始处理这一问题，但当时系统cpu资源耗尽，进程DIA0解决故障超过5分钟，导致出现了ora-29770错误。所以认定这次实例切换失败的主要原因在于ORA-00240错误。根据日志中给出的trace文件，查看trace文件中的具体报错原因。从trace日志中分析得知，当时控制文件被持有超过120秒的主要原因是KSV master wait等待，KSV master wait耗费了2分03秒。4.事故分析结论根据以上现象及日志体现，从oracle官方metalink中查找资料得知这是一个bug [BUG ID 1308282.1]下面是官方metalink文档中关于这个bug的解释：
High "ksv master wait" And "ASM File Metadata Operation" Waits In Non-Exadata 11g SymptomsHigh waits for "ksv master wait" while doing an ASM file metadata operation were reported when a data migration utility was running. This wait was also seen for a drop of a tablespace.The AWR showed the top events were CPU （> 100%）, with "ASM file metadata operation" （7%）. CauseEvent "KSV master wait" indicates the process on the RDBMS side is waiting for a reply from a process on the ASM side. In 11g, the parameter cell_offload_processing is set to TRUE. Although that is a parameter is not applicable for non-Exadata databases, it caused ASM to try to deliver smart-scan results. The issue was reported in Bug 11800170 - ASM IN KSV WAIT AFTER APPLICATION OF 11.2.0.2 GRID PSU.After applying the workaround for this issue （see Solution below）, a drop of a tablespace that used to take 13 minutes took 4 seconds. SolutionThe following solutions are available for non-Exadata databases:For the quickest solution, use the workaround. The workaround does not negatively impact non-Exadata databases. This parameter is to be set on the database instance.alter system set cell_offload_processing = false;Upgrade to 12.1, when available. ORApply the 11.2.0.3 patch set ORApply one-off Patch 11800170, if available for your RDBMS and Grid HomesNote: At the time this note was written （March 2011）, neither 12.1 nor 11.2.0.3 were available.官方文档中给出的最快解决方式是修改oracle中的一个参数cell_offload_processing，修改为false。更多Oracle相关信息见Oracle 专题页面 http://www.linuxidc.com/topicnews.aspx？tid=12Oracle 11.2.0.1 RAC使用exp遭遇ORA-01455错误解决办法AIX上安装Oracle 10201相关资讯 ORA-00240 本文评论查看全部评论（0）