Data Role Creation Request Stays In New Request Submitted Status

Today i was creating new data role which was inherited from "Human Capital Management Application administrator" role. Role creation request was stuck with "New Request Submitted Status" status. To troubleshoot issue i logged on fusion database and queried PER_LDAP_REQUESTS table to check my request status, error code and error description but was not getting any data.

select REQUEST_ID,REQUEST_STATUS,ERROR_CODE , ERROR_DESCRIPTION from PER_LDAP_REQUESTS

Then i reviewed soa_server1-diagnostic.log (HCMDomain) log files and found below errors

=======================================================================

[2012-08-30T16:14:27.273-07:00] [soa_server1] [ERROR] [] [oracle.integration.platform.blocks.scheduler.FabricJobStoreCMT] [tid: QuartzScheduler_FabricScheduler-pxfuscrmsec01.capgeminihosting.com1344532959144_ClusterManager] [ecid: 0000J^CQ5OGADSM_mLs1yZ1G3YVC0004N3,1:31442] [APP: soa-infra] ClusterManager: Error managing cluster: Failed to obtain DB connection from data source 'soaNonManagedDS': java.sql.SQLException: Could not retrieve datasource via JNDI url 'jdbc/SOALocalTxDataSource' java.sql.SQLException: weblogic.common.ResourceException: [[

========================================================================

Above both steps confirmed that soa is unable to communicate with Fusion database. 

I researched above error in metalink and found Note [ID 1477685.1]. As per Note, bounce fusion application and database to fix issue.

After complete bounce, above issue got fixed and but now my request was staying in "Request pending" status.

PER_LDAP_REQUESTS table was showing request_status as "IN PROGRESS" but no error and also there were no errors in soa_diagnostic.log so that means my request went successfully from fusion application and it might be stuck on OIM side. 

Next i checked oim_server1-diagnostic.log log file and found below error,

================================================================

[2012-09-20T15:20:46.283-07:00] [oim_server1] [ERROR] [IAM-2030153] [oracle.iam.callbacks.common] [tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: oiminternal] [ecid: 004mYa6SQLXADSM_mLw0yZ0006m_000tCB,0] [APP: oim#11.1.1.3.0]  [CALLBACKMSG] Exception encountered while triggering callback for correlation id 158. Updating in db for orchestration 618, event 2,002. Message is HTTP transport error: java.net.UnknownHostException: hcm-internal.oracleoutsourcing.com.

=================================================================

As per log file, idm server was not able to communicate with 

hcm-internal.oracleoutsourcing.com server. ping command also confirmed same issue.

To fix issue i have modified /etc/hosts and added server entry and that fixed role creation issue.

oracle.wsm.policymanager.PolicyManagerException: WSM-02081 : Failed to login to perform requested action

When trying to perform "Manage Business Unit" setup tasks, I was getting "A remote portlet error was received for the task: Manage Business Unit. Review the portlet producer error and report it to the owning product Financials Common Module"

We were getting same error for any setup tasks.

Solution:

First i checked "FunctionalSetupServer_1-diagnostic.log" and log file showed authentication error and it also pointed me out Credential key.


FunctionalSetupServer_1-diagnostic-1.log:"[2012-07-18T06:59:48.074-04:00] [FunctionalSetupServer_1] [ERROR] [WSM-06225] [oracle.wsm.resources.policyaccess] [tid: [ACTIVE].ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'] [ecid: 0000JYPjaJYADSM_mLp2iZ1G1BTq001ZIC,0] [APP: TopologyManagerServices] Unable to retrieve credentials for the specified CSF key FUSION_APPS_WSM_APPID-KEY."

Login to http://commoninternal.mycompany.com:7001/em

Go to Weblogic_Domain => Right click on CommonDomain => Click Security => Credential

Search for "FUSION_APPS_WSM_APPID-KEY" and click on edit on FUSION_APPS_WSM_APPID-KEY and get username.

As AppId users are created during installation by provisioning tools so not aware about current password and i also didn't want to reset with new password because not sure what all other places application stored it.

To retrieve current password, I used  listCred command. (Please find below steps to retrieve current password of Appid).


[oracle@famachine logs]$ cd /u01/oracle/fa/products/fusionapps/oracle_common/common/bin/
[oracle@famachine bin]$ ./wlst.sh
wls:/offline> connect('weblogic_fa','Welcome1','t3://commoninternal.mycompany.com:7001')
Connecting to t3://commoninternal.mycompany.com:7001 with userid weblogic_fa ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'CommonDomain'.
wls:/CommonDomain/serverConfig> listCred(map="oracle.wsm.security",key=" FUSION_APPS_WSM_APPID-KEY ")
Already in Domain Runtime Tree

PASSWORD:tzj|mgqv5cPi2w

After i got password, i performed ldapsearch command using id/password and got password expiration error.

[oracle@idmmachine logs]$  ldapsearch -h idmmachine -p 389 -D "cn=FUSION_APPS_WSM_APPID,cn=AppIDUsers,cn=Users,dc=mycompany,dc=com" -w "tzj|mgqv5cPi2w" -b "dc=mycompany,dc=com" -s sub "objectclass=inetorgperson" dn

ldap_bind: Invalid credentials

ldap_bind: additional info: Password Policy Error :9000: GSL_PWDEXPIRED_EXCP :Your Password has expired. Please contact the Administrator to change your password.

[oracle@idmmachine logs]$

so to fix issue, i have reset password for "FUSION_APPS_WSM_APPID".

Login on ODSM Console ( http://idmserver.domainname.com:7006/odsm) as cn=orcladmin.

click on Data Browser

click on dc=com,dc=mycompany,cn=users,cn=appIdUsers,cn=FUSION_APPS_WSM_APPID

Click on attributes

reset password to your existing password.

To avoid similar issue in future, create new password policy and assign to all the service accounts.

1. Log in to ODSM

2. Click Security (tab) -> Password Policy

3. Create a new password policy for service accounts.
Under the Effective Subtree tab of the new password policy, enter the service accounts:

Click Apply.

Targets found: Failed to find any product family.

Last week, i have installed Oracle Enterprise Manager 12c(12.1.0.1) and when trying to add Fusion Application Instance, i have received below error:

"Targets found: Failed to find any product family. Check host, port, credentials, protocol and be sure the admin server is running. Reason: Failed to discover any product family. Only Weblogic related targets are found. For discovering Fusion Middleware use Fusion Middleware Discovery"

Solution:

I have followed   Oracle® Enterprise Manager Cloud Control Basic Installation Guide  for agent configuration but was not able to discoverer product family. Then i talked to Oracle and got missing step details which fixed my issue. 

If you are stuck in same step, Provide below database grants to fusion_runtime user which fixed  issue and able to discoverer all the targets...

1. grant execute on DBMS_SYSTEM to  FUSION_RUNTIME; 
2. grant select on  v_$parameter to  FUSION_RUNTIME;
3. grant select on gv_$instance to  FUSION_RUNTIME;
4. grant select on V_$INSTANCE to  FUSION_RUNTIME;
5. grant select on gv_$diag_info to  FUSION_RUNTIME;
6. grant select on gv_$parameter to  FUSION_RUNTIME;
7. grant select on gv_$instance to  FUSION_RUNTIME; 

Happy Monitoring..