RHEL5.8文件系统只读造成ORA-00354: corrupt redo log block header

一次RHEL5.8的文件系统只读造成的ORA-00354: corrupt redo log block header。Wed Jun 03 03:31:24 CST 2015
Thread 1 advanced to log sequence 396 （LGWR switch）
Current log# 3 seq# 396 mem# 0: /u01/app/Oracle/oradata/orcl/redo03.log
Wed Jun 03 19:05:48 CST 2015
Thread 1 advanced to log sequence 397 （LGWR switch） ----------------->从这个地方可以看出来：sequence 397为current redo log
Current log# 1 seq# 397 mem# 0: /u01/app/oracle/oradata/orcl/redo01.log
Wed Jun 03 20:44:19 CST 2015
Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_j000_30071.trc:
ORA-07445: exception encountered: core dump [kglats（）+106] [SIGSEGV] [Address not mapped to object] [0x000000000] [] []
ORA-01403: no data found
Wed Jun 03 22:00:26 CST 2015
Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_j001_30285.trc:
ORA-07445: exception encountered: core dump [kglhdiv_callback（）+79] [SIGSEGV] [Address not mapped to object] [0x0000000D4] [] []
Thu Jun 04 20:28:17 CST 2015
Starting ORACLE instance （normal）后来数据库实例重启（人为重启）：Completed: ALTER DATABASE MOUNT
Thu Jun 04 20:28:24 CST 2015
ALTER DATABASE OPEN
Thu Jun 04 20:28:24 CST 2015
Beginning crash recovery of 1 threads
parallel recovery started with 3 processes
Thu Jun 04 20:28:24 CST 2015
Started redo scan
Thu Jun 04 20:28:24 CST 2015
Completed redo scan
1210 redo blocks read, 287 data blocks need recovery
Thu Jun 04 20:28:25 CST 2015
Started redo application at
Thread 1: logseq 397, block 660827
Thu Jun 04 20:28:25 CST 2015
Recovery of Online Redo Log: Thread 1 Group 1 Seq 397 Reading mem 0
Mem# 0: /u01/app/oracle/oradata/orcl/redo01.log
Thu Jun 04 20:28:25 CST 2015
Completed redo application
Thu Jun 04 20:28:29 CST 2015
Completed crash recovery at
Thread 1: logseq 397, block 662037, scn 1359645
287 data blocks read, 287 data blocks written, 1210 redo blocks read
Thu Jun 04 20:28:30 CST 2015
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=19, OS id=3609
Thu Jun 04 20:28:30 CST 2015
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=20, OS id=3611
Thu Jun 04 20:28:30 CST 2015
Thread 1 advanced to log sequence 398 （thread open） --->已经使用sequence 398，此时，sequence 398为current redo log
Thread 1 opened at log sequence 398
Current log# 2 seq# 398 mem# 0: /u01/app/oracle/oradata/orcl/redo02.log
Successful open of redo thread 1
Thu Jun 04 20:28:30 CST 2015
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu Jun 04 20:28:30 CST 2015
ARC0: Becoming the "no FAL" ARCH
ARC0: Becoming the "no SRL" ARCH
Thu Jun 04 20:28:30 CST 2015
SMON: enabling cache recovery
Thu Jun 04 20:28:30 CST 2015
ARC1: Becoming the heartbeat ARCH
Thu Jun 04 20:28:35 CST 2015
ARC0: Log corruption near block 385521 change 1314765 time ？ --->发现redo log 有损坏
Thu Jun 04 20:28:35 CST 2015
Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc:
ORA-00354: corrupt redo log block header --->发现redo log header有损坏
ORA-00353: log corruption near block 385521 change 1314765 time 06/03/2015 19:05:47
ORA-00312: online log 1 thread 1: "/u01/app/oracle/oradata/orcl/redo01.log"
ARC0: All Archive destinations made inactive due to error 354
Thu Jun 04 20:28:35 CST 2015
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_1: "/home/oracle/1_397_880950378.dbf" （error 354）
（orcl）
Committing creation of archivelog "/home/oracle/1_397_880950378.dbf" （error 354）
Thu Jun 04 20:28:43 CST 2015
Successfully onlined Undo Tablespace 1.
Thu Jun 04 20:28:43 CST 2015
SMON: enabling tx recovery
Thu Jun 04 20:28:45 CST 2015
ARCH: Archival stopped, error occurred. Will continue retrying
Thu Jun 04 20:28:45 CST 2015
ORACLE Instance orcl - Archival Error
Thu Jun 04 20:28:45 CST 2015
Database Characterset is ZHS16GBK
Thu Jun 04 20:28:45 CST 2015
ORA-16038: log 1 sequence# 397 cannot be archived
ORA-00354: corrupt redo log block header
ORA-00312: online log 1 thread 1: "/u01/app/oracle/oradata/orcl/redo01.log"
Thu Jun 04 20:28:45 CST 2015
Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc:
ORA-16038: log 1 sequence# 397 cannot be archived
ORA-00354: corrupt redo log block header
ORA-00312: online log 1 thread 1: "/u01/app/oracle/oradata/orcl/redo01.log"
Opening with internal Resource Manager plan
replication_dependency_tracking turned off （no async multimaster replication found）
Starting background process QMNC
QMNC started with pid=21, OS id=3637
Thu Jun 04 20:28:49 CST 2015
Completed: ALTER DATABASE OPEN ------->db open了。open归open，无法归档的话，业务早晚会hang。
--->此时/u01/app/oracle/oradata/orcl/redo02.log，是sequence 398，是current redo log。 --->然后，sqlplus中发出了日志切换的语句：alter system switch logfile;
Thread 1 cannot allocate new log, sequence 400---->已经不能进行日志切换，因为sequence 397未被归档，不允许被覆盖，此时db hang住。
All online logs needed archiving
Current log# 3 seq# 399 mem# 0: /u01/app/oracle/oradata/orcl/redo03.log
Thu Jun 04 21:28:31 CST 2015
ARC0: Log corruption near block 385521 change 1314765 time ？
Thu Jun 04 21:28:31 CST 2015
Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 385521 change 1314765 time 06/03/2015 19:05:47
ORA-00312: online log 1 thread 1: "/u01/app/oracle/oradata/orcl/redo01.log"
ARC0: All Archive destinations made inactive due to error 354
Thu Jun 04 21:28:31 CST 2015
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_1: "/home/oracle/1_397_880950378.dbf" （error 354）
（orcl）
ARCH: Archival stopped, error occurred. Will continue retrying
Thu Jun 04 21:28:34 CST 2015
ORACLE Instance orcl - Archival Error
Thu Jun 04 21:28:34 CST 2015
ORA-16038: log 1 sequence# 397 cannot be archived
ORA-00354: corrupt redo log block header
ORA-00312: online log 1 thread 1: "/u01/app/oracle/oradata/orcl/redo01.log"
Thu Jun 04 21:28:34 CST 2015
Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc:
ORA-16038: log 1 sequence# 397 cannot be archived
ORA-00354: corrupt redo log block header
ORA-00312: online log 1 thread 1: "/u01/app/oracle/oradata/orcl/redo01.log"
ARCH: Archival stopped, error occurred. Will continue retrying
Thu Jun 04 21:28:34 CST 2015
ORACLE Instance orcl - Archival Error
Thu Jun 04 21:28:34 CST 2015
ORA-16014: log 1 sequence# 397 not archived, no available destinations
ORA-00312: online log 1 thread 1: "/u01/app/oracle/oradata/orcl/redo01.log"
Thu Jun 04 21:28:34 CST 2015
Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc:
ORA-16014: log 1 sequence# 397 not archived, no available destinations
ORA-00312: online log 1 thread 1: "/u01/app/oracle/oradata/orcl/redo01.log" 启动db到mount状态，进行处理[oracle@lei1 ~]$ sqlplus / as sysdbaSQL*Plus: Release 10.2.0.5.0 - Production on Fri Jun 12 23:14:24 2015Copyright （c） 1982, 2010, Oracle. All Rights Reserved.Connected to an idle instance.SQL> startup mount;
ORACLE instance started.Total System Global Area 2147483648 bytes
Fixed Size 2097696 bytes
Variable Size 536874464 bytes
Database Buffers 1593835520 bytes
Redo Buffers 14675968 bytes
Database mounted.SQL> set line 200
SQL> select * from v$log; GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM
---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- ---------
1 1 397 536870912 1 NO INACTIVE 1294462 03-JUN-15
3 1 399 536870912 1 NO CURRENT 1361682 04-JUN-15
2 1 398 536870912 1 YES INACTIVE 1359647 04-JUN-15SQL> alter database clear logfile group 1;
alter database clear logfile group 1
*
ERROR at line 1:
ORA-00350: log 1 of instance orcl （thread 1） needs to be archived
ORA-00312: online log 1 thread 1: "/u01/app/oracle/oradata/orcl/redo01.log"
SQL> alter database clear unarchived logfile group 1;Database altered.SQL> select * from v$log; GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM
---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- ---------
1 1 0 536870912 1 YES UNUSED 1294462 03-JUN-15
3 1 399 536870912 1 NO CURRENT 1361682 04-JUN-15
2 1 398 536870912 1 YES INACTIVE 1359647 04-JUN-15SQL> archive log list;
Database log mode Archive Mode
Automatic archival Enabled
Archive destination /home/oracle
Oldest online log sequence 0
Next log sequence to archive 399
Current log sequence 399
SQL>此时alert日志的输出如下：Fri Jun 12 23:20:27 CST 2015
alter database clear logfile group 1
Fri Jun 12 23:20:27 CST 2015
ORA-350 signalled during: alter database clear logfile group 1...
Fri Jun 12 23:22:30 CST 2015
alter database clear unarchived logfile group 1
Fri Jun 12 23:22:31 CST 2015
WARNING！ CLEARING REDO LOG WHICH HAS NOT BEEN ARCHIVED. BACKUPS TAKEN
BEFORE 06/04/2015 20:28:30 （CHANGE 1359647） CANNOT BE USED FOR RECOVERY.
Clearing online log 1 of thread 1 sequence number 397
Fri Jun 12 23:22:43 CST 2015
Completed: alter database clear unarchived logfile group 1
 更多Oracle相关信息见Oracle 专题页面 http://www.linuxidc.com/topicnews.aspx？tid=12本文永久更新链接地址