Wednesday, November 7, 2012

FYI: OPMNCTL Failed to Start Managed Process After Maximum Retry Limit on Solaris 10, OBIEE 11g

1:22 PM

Share it Please
If you've happened to glance at Oracle Technical Network, it appears that alot of practitioners are encountering an issue when upgrading their 11g platform to the newest 11.1.1.6.4 version if they're using a Solaris or Linux distribution.

After upgrading to 11.1.1.6.4 as outlined in my previous blog post some users encounter the following error(s) when trying to start their BI services using opmnctl (Oracle Process Manager and Notification Server) with the 'opmnctl startall' command:

================================================================================
Response: 1 of 5 processes started.

ias-instance id=instance1
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++



ias-component/process-type/process-set:
coreapplication_obis1/OracleBIServerComponent/coreapplication_obis1/

Error
--> Process (index=1,uid=1445883997,pid=18522)
failed to start a managed process after the maximum retry limit
Log:
/u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBIServerComponent/coreapplication_obis1/console~coreapplication_obis1~1.log

ias-component/process-type/process-set:
coreapplication_obips1/OracleBIPresentationServicesComponent/coreapplication_obips1/

Error
--> Process (index=1,uid=1445883998,pid=0)
Executable file does not have execute permission
/u01/app/oracle/obiee11g/Oracle_BI1/bifoundation/web/bin/sawserver
failed to start a managed process after the maximum retry limit
Executable file does not have execute permission
/u01/app/oracle/obiee11g/Oracle_BI1/bifoundation/web/bin/sawserver
Log:
none

ias-component/process-type/process-set:
coreapplication_obisch1/OracleBISchedulerComponent/coreapplication_obisch1/

Error
--> Process (index=1,uid=1445884000,pid=18523)
failed to start a managed process after the maximum retry limit
Log:
/u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBISchedulerComponent/coreapplication_obisch1/console~coreapplication_obisch1~1.log

ias-component/process-type/process-set:
coreapplication_obiccs1/OracleBIClusterControllerComponent/BIClusterController/

Error
--> Process (index=1,uid=1445884001,pid=0)
Executable file does not have execute permission
/u01/app/oracle/obiee11g/Oracle_BI1/bifoundation/server/bin/nqsclustercontroller
failed to start a managed process after the maximum retry limit


 The important things to note are:
  • You may encounter all, some, or just 1 of your bi_services which fail to start. In the above example, the following services failed to start:
    • BI Server
    • BI Publisher
    • BI Cluster Controller
    • BI Scheduler
  • There are actually two issues:
    • Managed Processes failed to start after the maximum retry limit
    • Certain executable files do not have the execute permission enabled
Issue 1: Fix Maximum Retry Limit

This can be easily resolved by setting the ulimit at the user level to : ulimit -d unlimited , as outlined in Oracle's note ID: 1292887.1 . Anyone who has worked in OBIEE 10g should be familiar with this as it is required in 10g's installation process as well.

Issue 2: Enable read access on executable files

If you tried re-running opmnctl startall after resolving issue 1, you'll notice that the BI services still fail to run despite increasing the maximum retry limit. Upon examining the log files as outlined in the above error:

  • u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBIServerComponent/coreapplication_obis1/console~coreapplication_obis1~1.log
  • u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBISchedulerComponent/coreapplication_obisch1/console~coreapplication_obisch1~1.log
You'll notice that OPMNCTL still fails because it can't execute: nqsserver, nqscheduler and nqsclustercontroller. Below is the log from the above bullet point #1 :

/export/obiee/11g/Oracle_BI1/bifoundation/server/bin/biserver.sh: nqsserver: cannot execute--------
12/11/06 15:48:06 Start process
--------
/export/obiee/11g/Oracle_BI1/bifoundation/server/bin/biserver.sh: nqsserver: cannot execute


 During the 11.1.1.6.4 upgrade on your Solaris or Linux box, these files became read only!

How to resolve this read only issue:

Navigate to your $ORACLE_HOME/bifoundation/server/bin folder and make the the following files executable by using the chmod +x command:  nqscheduler, nqsclustercontroller, nqsserver. These files should have read/write privileges as outlined in the image below:



















Now when you run opmnctl startall, all  services should execute successfully:

Important:

After the upgrade, it is critical that you follow Oracle's recommended start up & shutdown sequence for weblogic & OBIEE 11g

Start up:
  • $DOMAIN_HOME/bin/startWebLogic.sh (Weblogic)
  • $WL_HOME/server/bin/startNodeManager.sh (Node Manager)
  • $DOMAIN_HOME/bin/startManagedWebLogic.sh bi_server1 http://server.domain:7001 (Managed Server)
  • $ORACLE_INSTANCE/bin/opmnctl startall (OPMN)
Shutdown:
  • $ORACLE_INSTANCE/bin/opmnctl stopall (OPMN)
  • $DOMAIN_HOME/bin/stopManagedWebLogic.sh bi_server1 t3://host.domain:7001 username password (Managed Server)
  • $DOMAIN_HOME/bin/stopWebLogic.sh username password t3://host.domain:7001 (Admin Server)

In my upcoming blog post i'll discuss how to automate this entire process.



keywords: opmn , opmn start failed, obiee 11g, opmn obiee weblogic, bi_server

10 comments:

  1. Thank you very much for your good information.
    As I noted in the ... given information is very usefull to every student who ever want to learn about
    best obiee Online Training
    .

    ReplyDelete
  2. Appreciation for nice Updates, I found something new and folks can get useful info about BEST obiee ONLINE TRAINING

    ReplyDelete
  3. When I originally commented I seem to have clicked the -Notify me when new comments are added- checkbox and now whenever a comment is added I receive 4 emails with the same comment. Is there a means you are able to remove me from that service? Thanks!
    Hadoop Online Training

    ReplyDelete
  4. I get a lot of great information from this blog. Thank you for your sharing this informative blog.
    sap scm training

    ReplyDelete
  5. Vey very nice blog!!
    Hi there,I enjoy reading through your article post, I wanted to write a little comment to support you and wish you a good continuationAll the best for all your blogging efforts.
    Appreciate the recommendation! Let me try it out.
    Keep working ,great job!
    sap scm training
    sap fico training

    ReplyDelete


  6. Thank you For tutorials on Testing and the also the other tutorials on Blog are Awosme
    SAP Success Factors Training In Hyderabad

    ReplyDelete
  7. FYI: OPMNCTL Failed to Start Managed Process After Maximum Retry Limit on Solaris 10, OBIEE 11g nice posts..
    SAP HANA training in hyderabad

    ReplyDelete


  8. Hadoop training in hyderabad.All the basic and get the full knowledge of hadoop.
    hadoop training in hyderabad


    ReplyDelete
  9. Big data and data warehousing related information is always updated to me at hadoop online training
    in hyderabad. Nice insight on the topic refer the details at
    hadoop online training

    ReplyDelete
  10. Nice explanation with snapshots, keep it up for more Sharepoint online training

    ReplyDelete