Concurrent Program

Concurrent Manager:
Internal Concurrent Manager - The master manager
Standard Manager - The SM functions to run any reports and batch jobs
Conflict Resolution Manager - check concurrent program definitions for incompatibility rules

FNDLIBR
    Marketing Data Mining Manager
    Transportation Manager
    Session History Cleanup
    UWQ Worklist Items Release for Crashed session
    Collections Manager
    OAM Metrics Collection Manager
    Contracts Core Concurrent Manager
    Standard Manager
    WMS Task Archiving Manager
    Oracle Provisioning Manager

INVLIBR
    Inventory Manager

MRCLIB
    MRP Manager

PALIBR
    PA Streamline Manager

FNDSM
    NDSM is executable and core component in GSM (Generic Service Management Framework)

Parallel Concurrent Processing (PCP)

The Internal Concurrent Manager (ICM) will only restart a request if the following conditions are met

1. The ICM got the manager's database lock for the manager that was running the request
2. The phase of the request is "running" (phase_code = 'R')
3. The program for this request is set to "restart on failure"

All of the above requirements have been met AND at least one of the following:
a.  The ICM is just starting up, (ie. it has just spawned on a given node and going through initial code before the main loop)
b.  The node of the concurrent manager for which we got the lock is down
c.  The database instance (TWO_TASK) defined for the node of that concurrent  manager is down (this is not applicable if one is using some "balance" @ TWO_TASK on that node)

PHASE_CODE and Status Code
Pending (waiting to be run)
    Normal(Request is waiting for the next available manager)
    Standby (Program to run request is incompatible with other program(s) currently running)
    Scheduled (Request is scheduled to start at a future time or date)
    Waiting (A child request is waiting for its Parent request to mark it ready to run)
Running
    Normal (Request is running normally. )
    Paused (Parent request pauses for all its child requests to finish running)
    Resuming (All requests submitted by the same parent request have completed running)
    Terminating (Request is terminated by choosing the Cancel Request button in Requests window)
Completed
    Normal (Request completed successfully)
    Error (Request failed to complete successfully)
    Warning (Request completed with warnings)
    Cancelled (Pending or Inactive request is cancelled by choosing the Cancel Request button in the Requests window)
    Terminated (Request is terminated by choosing the Cancel Request button in the Requests window)
Inactive (unable to run)
    Disabled (Program to run request is not enabled)
    On Hold (Pending request is placed on hold by choosing the Hold Request button in the Requests window)
    No Manager (No manager is defined to run the request)

This post covers overview of How to troubleshoot long running concurrent request in  Oracle Apps 11i/R12(Edit later)

Step 1 : Check Concurrent Request ID of long running concurrent request from front end
Step 2 : Find SID, SERIAL# and SPID by running SQL (given below)
Step 3 : Enable event 10046 trace with level 12 using oradebug ( for 15-20 minute)
Step 4 : Disable trace (once you are happy with trace size)
Step 5 : Convert raw trace to TKPROF using various sort options like fchela, prsela, execpu
Step 6 : Check TKPROF out file to find root cause of slow concurrent request


Step 1 : Check Request ID from Find Concurrent request screen (In my case Request ID is 1145)

Step 2 : Run below command to find SPID, provide concurrent request ID (1145 in my case) when prompted

SELECT a.request_id, d.sid, d.serial# ,d.osuser,d.process , c.SPID
FROM apps.fnd_concurrent_requests a,
apps.fnd_concurrent_processes b,
v$process c,
v$session d
WHERE a.controlling_manager = b.concurrent_process_id
AND c.pid = b.oracle_process_id
AND b.session_id=d.audsid
AND a.request_id = &Request_ID;

REQUEST_ID        SID    SERIAL# OSUSER           PROCESS    SPID
—————----------------------------------------------------------------------------
   1145                  514            28         applmgr             17794        2987.
.

Step 3.1 : Check and confirm SPID on Database Node
$   ps-ef | grep 2987
  proddb   2987     1  0 13:30:43 ?        0:03 oracledbr12 (LOCAL=NO)

Step 3.2 : Set OSPID (2987 in my case) for ORADEBUG
SQL> oradebug setospid 2987

Step 3.3 : Enable trace for 10046 event with level 12
SQL> oradebug event 10046 trace name context forever, level 12

LEVEL 12 – Both Binds and Waits
LEVEL 8 – Only WAITS
LEVEL 4 – Only BIND Variables

Step 3.4 : Locate Trace file as
SQL>oradebug tracefile_name

/oracle/apps/proddb/10.2.0/admin/prod_CSDoracle/udump/ prod _ora_2987.trc

Wait for 15-20 minutes

Step 4 : Disable trace
SQL> oradebug event 10046 trace name context off

Step 5: Create tkprof file like
$ tkprof prod _ora_2987.trc prod _ora_2987.txt explain=apps/<****>  sort=(exeela,fchela) sys=no

Step 6 : Check TKPROF file to find root cause of slow concurrent requet