OraFAQ Forum: Server Administration » ORA-27044, ORA-16038, ORA-19504, ORA-00312

Home » RDBMS Server » Server Administration » ORA-27044, ORA-16038, ORA-19504, ORA-00312 (Oracle 10g, Solaris 10)

Show: Today's Messages :: Polls :: Message Navigator
E-mail to friend

ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346697]

Tue, 09 September 2008 07:39

deepmachine
Messages: 80
Registered: August 2008
Location: United States

Member

Anacedent, Michel...
I am having issues. Oracle database went down due to archive_log destination got filled up 100%. I freed up space and started the database.

Now I am getting following errors.

ORA-27044: unable to write the header block of file,Errors in file /u01/app/oracle/admin/PROD/bdump/prod1_arc1_1459.trc
ORA-16038: log 6 sequence# 138703 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 6 thread 1

I also, tried:

alter system switch log;

but no help...

Here is the my trace file:

$ tail -100 /u01/app/oracle/admin/PROD/bdump/prod1_arc1_1459.trc

ARC1: All standby destinations failed; successful archival assumed
*** 2008-09-09 05:59:02.486 16790 kcrr.c
ORA-16038: log 1 sequence# 138704 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 1 thread 1: '+PROD_DATA1/prod/onlinelog/group_1.261.1'
Exception 19502 encountered when closing file
ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 1 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
Additional information: 7680
Additional information: 1048576
ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 1 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
Additional information: 7680
Additional information: 1048576
*** 2008-09-09 05:59:03.113 53058 kcrr.c
ARC1: Error 19502 Closing archive log file '/u02/archives/1_138704_548018336.dbf'
*** 2008-09-09 05:59:03.113 51534 kcrr.c
kcrrfail: dest:2 err:19502 force:0 blast:1
*** 2008-09-09 05:59:03.113 53058 kcrr.c
ARC1: All standby destinations failed; successful archival assumed
*** 2008-09-09 05:59:03.123 16790 kcrr.c
ORA-16038: log 1 sequence# 138704 cannot be archived
ORA-19502: write error on file "", blockno  (blocksize=)
ORA-00312: online log 1 thread 1: '+PROD_DATA1/prod/onlinelog/group_1.261.1'
Exception 19502 encountered when closing file
ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 16385 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
Additional information: 228864
Additional information: 1048576
ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 16385 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
Additional information: 228864
Additional information: 1048576
*** 2008-09-09 05:59:04.391 53058 kcrr.c
ARC1: Error 19502 Closing archive log file '/u02/archives/1_138704_548018336.dbf'
*** 2008-09-09 05:59:04.391 51534 kcrr.c
kcrrfail: dest:2 err:19502 force:0 blast:1
*** 2008-09-09 05:59:04.391 53058 kcrr.c
ARC1: All standby destinations failed; successful archival assumed
*** 2008-09-09 05:59:04.399 16790 kcrr.c
ORA-16038: log 1 sequence# 138704 cannot be archived
ORA-19502: write error on file "", blockno  (blocksize=)
ORA-00312: online log 1 thread 1: '+PROD_DATA1/prod/onlinelog/group_1.261.1'
Exception 19502 encountered when closing file
ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 18433 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
Additional information: 564736
Additional information: 1046528
ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 18433 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
Additional information: 564736
Additional information: 1046528
*** 2008-09-09 05:59:05.601 53058 kcrr.c
ARC1: Error 19502 Closing archive log file '/u02/archives/1_138704_548018336.dbf'
*** 2008-09-09 05:59:05.601 51534 kcrr.c
kcrrfail: dest:2 err:19502 force:0 blast:1
*** 2008-09-09 05:59:05.601 53058 kcrr.c
ARC1: All standby destinations failed; successful archival assumed
*** 2008-09-09 05:59:05.611 16790 kcrr.c
ORA-16038: log 1 sequence# 138704 cannot be archived
ORA-19502: write error on file "", blockno  (blocksize=)
ORA-00312: online log 1 thread 1: '+PROD_DATA1/prod/onlinelog/group_1.261.1'

I would greatly appreciate if you could take some time out and provide me the solution.

Thanks very much.

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346713 is a reply to message #346697]

Tue, 09 September 2008 08:41

deepmachine
Messages: 80
Registered: August 2008
Location: United States

Member

Would following command fix the issue?

alter system archive log current;

Thanks..

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346730 is a reply to message #346697]

Tue, 09 September 2008 09:35

Mahesh Rajendran
Messages: 10707
Registered: March 2002
Location: oracleDocoVille

Senior Member
Account Moderator

>> I freed up space and started the database
Seems you nuked the log destination area including the current archive log files.
Next time, apply proper methods.
Shutdown.
Startup.
Issue
alter system switch logfile; -- do it a couple of times.

[Updated on: Tue, 09 September 2008 09:39]

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346742 is a reply to message #346730]

Tue, 09 September 2008 10:18

deepmachine
Messages: 80
Registered: August 2008
Location: United States

Member

I actually moved some old files to a different drive freeing up some 16G of space.

I first did a

crs_stat -t

and saw that the TARGET of two services was ONLINE but STATE offline. So I fired a

crs_start -all

command to bring both nodes up.

After this, I tried the

alter system switch log

command 4 times. But I am still getting the errors.

I also tried

alter system archive log current

, but no help so far.

Its live production. What should be the steps to fix this issue?

I am in a terrible situation, awaiting your reply.

Thanks

[Updated on: Tue, 09 September 2008 10:24]

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346744 is a reply to message #346697]

Tue, 09 September 2008 10:24

BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal

Senior Member

This comes with no guarentee, but it won't hurt if you rename /u02/archives/1_138704_548018336.dbf to a different name.
It might allow the archiver process to try again with a fresh copy & when there is free space to complete the copy.

Good Luck!

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346754 is a reply to message #346742]

Tue, 09 September 2008 11:16

Mahesh Rajendran
Messages: 10707
Registered: March 2002
Location: oracleDocoVille

Senior Member
Account Moderator

You have not mentioned that you are using RAC in any of your above posts.
>>ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 1 (blocksize=512)
Is /u02 properly mounted across both nodes?
Are any other log destinations specified?
Repost the relevant portions of alert log in both nodes.

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346779 is a reply to message #346744]

Tue, 09 September 2008 12:41

deepmachine
Messages: 80
Registered: August 2008
Location: United States

Member

It didn't help. I moved the existing file to a different folder, but I am still receiving the same error.

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346782 is a reply to message #346697]

Tue, 09 September 2008 12:50

BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal

Senior Member

If the ASM volumes reside on NetApp, then this could be "known bug". Search Metalink for details.

You should consider opening a Service Request with oracle.

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346787 is a reply to message #346697]

Tue, 09 September 2008 13:02

BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal

Senior Member

If you are desperate, you could create a new folder on the ASM volume & the change Archive destination in spfile to point to new directory.
Again, it can't hurt & may get around current issue involving /u02/archives.

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346791 is a reply to message #346754]

Tue, 09 September 2008 13:07

deepmachine
Messages: 80
Registered: August 2008
Location: United States

Member

Yes its 2-node RAC on 10g 10.1.0.5 (node1/node2 | Active/Passive) configuration.

Quote:

Is /u02 properly mounted across both nodes?

Yes the /u02/archive folder is properly mounted on both nodes. That is the reason it was working fine until this morning when Oracle went down, and after that everything started giving errors.

Quote:

Are any other log destinations specified?

No, but /u02/archives is log_archive_dest_2 in parameter_file and log_archive_dest_1 is empty. Does this matter?

I am more interested in digging into thing rather THINGS_ARE_OK. So please guide me more.

The alert log is all about the information I posted earlier this morning. If you could give me an idea of what to look for specifically, that would be great.

Thanks for your help...

P.S. Please look at the attached file... /forum/fa/4985/0/

Here is the Alert_log's last 100 lines...

ARC0: Evaluating archive thread 1 sequence 138761
Tue Sep  9 13:47:44 2008
ARC0: Beginning to archive thread 1 sequence 138761 (1053121623-1053125943) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:47:44 2008
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138761_548018336.dbf' (thread 1 sequence 138761)
 (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:47:45 2008
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138761_548018336.dbf'
 (PROD1)
ARCH: Connecting to console port...
Committing creation of archivelog '/u02/archives/1_138761_548018336.dbf'
Tue Sep  9 13:47:46 2008
ARC0: Completed archiving thread 1 sequence 138761 (1053121623-1053125943) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:53:46 2008
Private_strands 0 at log switch
Thread 1 advanced to log sequence 138763
  Current log# 6 seq# 138763 mem# 0: +PROD_DATA1/prod/onlinelog/group_6.269.1
Tue Sep  9 13:53:46 2008
ARC0: Evaluating archive thread 1 sequence 138762
Tue Sep  9 13:53:46 2008
ARC0: Beginning to archive thread 1 sequence 138762 (1053125943-1053130125) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:53:46 2008
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138762_548018336.dbf' (thread 1 sequence 138762)
 (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:53:46 2008
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138762_548018336.dbf'
 (PROD1)
ARCH: Connecting to console port...
Committing creation of archivelog '/u02/archives/1_138762_548018336.dbf'
Tue Sep  9 13:53:47 2008
ARC0: Completed archiving thread 1 sequence 138762 (1053125943-1053130125) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:56:46 2008
Private_strands 0 at log switch
Thread 1 advanced to log sequence 138764
  Current log# 1 seq# 138764 mem# 0: +PROD_DATA1/prod/onlinelog/group_1.261.1
Tue Sep  9 13:56:46 2008
ARC0: Evaluating archive thread 1 sequence 138763
Tue Sep  9 13:56:46 2008
ARC0: Beginning to archive thread 1 sequence 138763 (1053130125-1053131953) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:56:46 2008
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138763_548018336.dbf' (thread 1 sequence 138763)
 (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:56:46 2008
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138763_548018336.dbf'
 (PROD1)
ARCH: Connecting to console port...
Committing creation of archivelog '/u02/archives/1_138763_548018336.dbf'
Tue Sep  9 13:56:47 2008
ARC0: Completed archiving thread 1 sequence 138763 (1053130125-1053131953) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:58:10 2008
Private_strands 0 at log switch
Thread 1 advanced to log sequence 138765
  Current log# 5 seq# 138765 mem# 0: +PROD_DATA1/prod/onlinelog/group_5.268.1
Tue Sep  9 13:58:10 2008
ARC0: Evaluating archive thread 1 sequence 138764
Tue Sep  9 13:58:10 2008
ARC0: Beginning to archive thread 1 sequence 138764 (1053131953-1053133576) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:58:10 2008
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138764_548018336.dbf' (thread 1 sequence 138764)
 (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:58:10 2008
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138764_548018336.dbf'
 (PROD1)
ARCH: Connecting to console port...
Committing creation of archivelog '/u02/archives/1_138764_548018336.dbf'
Tue Sep  9 13:58:11 2008
ARC0: Completed archiving thread 1 sequence 138764 (1053131953-1053133576) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 14:00:09 2008
Private_strands 0 at log switch
Thread 1 advanced to log sequence 138766
  Current log# 2 seq# 138766 mem# 0: +PROD_DATA1/prod/onlinelog/group_2.262.1
Tue Sep  9 14:00:09 2008
ARC0: Evaluating archive thread 1 sequence 138765
Tue Sep  9 14:00:09 2008
ARC0: Beginning to archive thread 1 sequence 138765 (1053133576-1053135372) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 14:00:09 2008
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138765_548018336.dbf' (thread 1 sequence 138765)
 (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 14:00:09 2008
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138765_548018336.dbf'
 (PROD1)
ARCH: Connecting to console port...
Committing creation of archivelog '/u02/archives/1_138765_548018336.dbf'
Tue Sep  9 14:00:10 2008
ARC0: Completed archiving thread 1 sequence 138765 (1053133576-1053135372) (PROD1)
ARCH: Connecting to console port...

Attachment: ARCH_Hung.JPG
(Size: 127.92KB, Downloaded 12092 times)

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346793 is a reply to message #346697]

Tue, 09 September 2008 13:11

BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal

Senior Member

My assessment is that the logjam has broken free & all is back to normal (for now).

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346805 is a reply to message #346793]

Tue, 09 September 2008 13:49

deepmachine
Messages: 80
Registered: August 2008
Location: United States

Member

I guess, I would agree with you. The culprit is FogLight. For some reason its continuously sending me the same error/incident that took place before 6:00AM this morning.

Whereas, when I go and look at that alert_log and the *.trc file, it shows me nothing new.

Thanks for being with me. Your support is always greatly appreciated.

Thanks again..

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346806 is a reply to message #346697]

Tue, 09 September 2008 13:52

BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal

Senior Member

> The culprit is FogLight.
Those who live by the GUI, die by the GUI.

the alert_SID.log file should show you when the logjam broke free.
I still suspect things started getting better just after renaming the broken archived redo log file out of the way.

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346809 is a reply to message #346806]

Tue, 09 September 2008 14:03

Mahesh Rajendran
Messages: 10707
Registered: March 2002
Location: oracleDocoVille

Senior Member
Account Moderator

Which is why we point to read the forum rules first. Smile

Posting the relevant contents from alert log could have
saved you from some agony.

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #359886 is a reply to message #346809]

Tue, 18 November 2008 08:04

khushal.k
Messages: 1
Registered: July 2008
Location: Thane

Junior Member

It clearly stated that your filesystem is full. there should be no space left on disk.

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #480102 is a reply to message #346697]

Thu, 21 October 2010 04:03

ashuj20
Messages: 12
Registered: October 2005
Location: new delhi

Junior Member

Hi,

All the suggestion I see in this post are all crap!!

This problem is a know bug in Orcale (Bug 3615505). It is resolved in Release 10.2 or higher.

Thanks

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #480120 is a reply to message #480102]

Thu, 21 October 2010 05:15

Michel Cadot
Messages: 68658
Registered: March 2007
Location: Nanterre, France, http://...

Senior Member
Account Moderator

Thanks for the information, why were not you there in 2008?
Insults are not necessary.

Regards
Michel

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #480135 is a reply to message #480102]

Thu, 21 October 2010 06:49

Mahesh Rajendran
Messages: 10707
Registered: March 2002
Location: oracleDocoVille

Senior Member
Account Moderator

>>All the suggestion I see in this post are all crap!!
Would that include your drivel too?
The said bug is windows OS.
The bug is talking about entries in Event viewer.
This thread is about entries in oracle alert log in Solaris.

Report message to a moderator

Previous Topic:	Clone Database can not be connected after system reboot.
Next Topic:	Contention on data blocks

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Wed Jun 05 13:10:31 CDT 2024