How to create APPSORA.env file in Fusion Application Environment

Most of fusion application patching or maintenance documents ask to set fusion application environment using APPSORA.env file but when i used to go that location i couldn't find that file.

After doing some research, i found out how to create APPSORA.env file..

Go to /u01/app/oracle/fa/products/fusionapps/applications/lcm/ad/bin location:

Run adsetenv.sh file

You can run adsetenv.sh file with View option or Noview (Default option).

If you run adsetenv.sh file with view option, APPSORA.env file will set environment using setenv which could use for csh shell.

Noview (default) option will create file which would set environment with export.


adsetenv.sh file DESCRIPTION
#   Shell script to create env file for Oracle Fusion Application.
#   By default script generate the env file with export command
#   For view, pass extra argument 'VIEW', to get env file compatible to view

This small manual step should include as one of the post installation step.

[Security:090479]Certificate chain received from scmhost1.domain.com failed date validity checks

Recently i faced certification expiration issue during starting Fusion Application using fastartstop.sh script. In fusion Application, to connect node manager for start/stop Admin/Managed servers, it will use fusion_trust.jks keystore. If certification is already expired then you will receive below error..

Error:
WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : [Security:090479]Certificate chain received from scmhost1.domain.com - <IP Address> failed date validity checks.
Use dumpStack() to view the full stacktrace


Solution:
1. First go to KeyStore directory
/u01/app/oracle/fa/products/fusionapps/wlserver_10.3/server/lib
2. if you do ls -ltr *.jks, you will find below two keystores

-rw-r--r-- 1 oracle oinstall 82535 Oct 16 14:49 fusion_trust.jks
-rw-r--r-- 1 oracle oinstall  1416 Oct 16 14:50 scmhost1.domain.com_fusion_identity.jks

3. Let's check validity of certificate
keytool -list -v -keystore  scmhost1.domain.com_fusion_identity.jks -storepass Welcome1
=====================================================================
Output:
Alias name: scmhost1.domain.com_fusion
Creation date: Mar 14, 2012
Entry type: trustedCertEntry

Owner: CN=scmhost1.domain.com, OU=defaultOrganizationUnit, O=defaultOrganization, C=US
Issuer: CN=scmhost1.domain.com, OU=defaultOrganizationUnit, O=defaultOrganization, C=US
Serial number: 4f60c436
Valid from: Wed Mar 14 12:15:50 EDT 2012 until: Mon Sep 10 12:15:50 EDT 2012
Certificate fingerprints:
         MD5:  67:D5:E7:26:17:D9:A9:1B:AF:E0:1B:E1:DA:D2:8B:9A
         SHA1: 80:1D:AF:B5:0F:01:B3:B9:54:3E:E3:56:BB:54:99:4A:55:64:1F:CF
         Signature algorithm name: SHA1withRSA
         Version: 3
=========================================================================
As per above output, my certification was expired on Sep 10 2012. If you run above list command for fusion_trust.jks keystore, you will find scmhost1.domain.com_fusion alias certificate has expired with same date.

To fix issue, Perform below steps:

1. Backup existing keystore
2. create /tmp/certificate directory (will create new keystore in this directory)
3. Generate New keystore

keytool -genkeypair -keypass Welcome1 -storepass Welcome1 -keyalg RSA -alias "scmhost1.domain.com_fusion" -keystore scmhost1.domain.com_fusion_identity.jks -validity 3650 -dname "CN=scmhost1.domain.com, OU=defaultOrganizationUnit, O=defaultOrganization, C=US"

4. Export Certificate (from New Key Store)

keytool -export -alias  scmhost1.domain.com_fusion -rfc -keypass Welcome1 -storepass Welcome1  -keystore scmhost1.domain.com_fusion_identity.jks -file scmhost1.domain.com_fusion.cer

5. Now delete scmhost1.domain.com_fusion alias from fusion_trust.jks Keystore
cd /u01/app/oracle/fa/products/fusionapps/wlserver_10.3/server/lib

keytool -delete -alias  scmhost1.domain.com_fusion -keypass Welcome1 -storepass Welcome1 -keystore fusion_trust.jks

Note: Make sure your fusion application environment is down before performing above step.

6. Now import scmhost1.domain.com_fusion.cer certificate to fusion_trust.jks keystore

cp /tmp/certificate/scmhost1.domain.com_fusion.cer .

keytool -import -alias  scmhost1.domain.com_fusion -noprompt -keystore fusion_trust.jks -keypass Welcome1 -storepass Welcome1 -file scmhost1.domain.com_fusion.cer

7. Now validate certificate validity for fusion_trust.jks keystore

keytool -list -v -keystore  fusion_trust.jks -storepass Welcome1
========================================================================
Alias name: scmhost1.domain.com_fusion
Creation date: Oct 16, 2012
Entry type: trustedCertEntry

Owner: CN=scmhost1.domain.com, OU=defaultOrganizationUnit, O=defaultOrganization, C=US
Issuer: CN=scmhost1.domain.com, OU=defaultOrganizationUnit, O=defaultOrganization, C=US
Serial number: 507da94d
Valid from: Tue Oct 16 14:37:01 EDT 2012 until: Fri Oct 14 14:37:01 EDT 2022
Certificate fingerprints:
         MD5:  EF:50:0E:C8:C1:A2:6D:7B:3D:9D:3B:8C:06:44:17:6A
         SHA1: DB:57:08:FC:BB:1D:BC:52:86:C1:EF:14:D7:D1:28:58:0D:8C:B0:3F
         Signature algorithm name: SHA1withRSA
         Version: 3
========================================================================
8. Start Node manager, Admin and Managed servers

Data Role Creation Request Stays In New Request Submitted Status

Today i was creating new data role which was inherited from "Human Capital Management Application administrator" role. Role creation request was stuck with "New Request Submitted Status" status. To troubleshoot issue i logged on fusion database and queried PER_LDAP_REQUESTS table to check my request status, error code and error description but was not getting any data.

select REQUEST_ID,REQUEST_STATUS,ERROR_CODE , ERROR_DESCRIPTION from PER_LDAP_REQUESTS

Then i reviewed soa_server1-diagnostic.log (HCMDomain) log files and found below errors

=======================================================================

[2012-08-30T16:14:27.273-07:00] [soa_server1] [ERROR] [] [oracle.integration.platform.blocks.scheduler.FabricJobStoreCMT] [tid: QuartzScheduler_FabricScheduler-pxfuscrmsec01.capgeminihosting.com1344532959144_ClusterManager] [ecid: 0000J^CQ5OGADSM_mLs1yZ1G3YVC0004N3,1:31442] [APP: soa-infra] ClusterManager: Error managing cluster: Failed to obtain DB connection from data source 'soaNonManagedDS': java.sql.SQLException: Could not retrieve datasource via JNDI url 'jdbc/SOALocalTxDataSource' java.sql.SQLException: weblogic.common.ResourceException: [[

========================================================================

Above both steps confirmed that soa is unable to communicate with Fusion database. 

I researched above error in metalink and found Note [ID 1477685.1]. As per Note, bounce fusion application and database to fix issue.

After complete bounce, above issue got fixed and but now my request was staying in "Request pending" status.

PER_LDAP_REQUESTS table was showing request_status as "IN PROGRESS" but no error and also there were no errors in soa_diagnostic.log so that means my request went successfully from fusion application and it might be stuck on OIM side. 

Next i checked oim_server1-diagnostic.log log file and found below error,

================================================================

[2012-09-20T15:20:46.283-07:00] [oim_server1] [ERROR] [IAM-2030153] [oracle.iam.callbacks.common] [tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: oiminternal] [ecid: 004mYa6SQLXADSM_mLw0yZ0006m_000tCB,0] [APP: oim#11.1.1.3.0]  [CALLBACKMSG] Exception encountered while triggering callback for correlation id 158. Updating in db for orchestration 618, event 2,002. Message is HTTP transport error: java.net.UnknownHostException: hcm-internal.oracleoutsourcing.com.

=================================================================

As per log file, idm server was not able to communicate with 

hcm-internal.oracleoutsourcing.com server. ping command also confirmed same issue.

To fix issue i have modified /etc/hosts and added server entry and that fixed role creation issue.

oracle.wsm.policymanager.PolicyManagerException: WSM-02081 : Failed to login to perform requested action

When trying to perform "Manage Business Unit" setup tasks, I was getting "A remote portlet error was received for the task: Manage Business Unit. Review the portlet producer error and report it to the owning product Financials Common Module"

We were getting same error for any setup tasks.

Solution:

First i checked "FunctionalSetupServer_1-diagnostic.log" and log file showed authentication error and it also pointed me out Credential key.


FunctionalSetupServer_1-diagnostic-1.log:"[2012-07-18T06:59:48.074-04:00] [FunctionalSetupServer_1] [ERROR] [WSM-06225] [oracle.wsm.resources.policyaccess] [tid: [ACTIVE].ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'] [ecid: 0000JYPjaJYADSM_mLp2iZ1G1BTq001ZIC,0] [APP: TopologyManagerServices] Unable to retrieve credentials for the specified CSF key FUSION_APPS_WSM_APPID-KEY."

Login to http://commoninternal.mycompany.com:7001/em

Go to Weblogic_Domain => Right click on CommonDomain => Click Security => Credential

Search for "FUSION_APPS_WSM_APPID-KEY" and click on edit on FUSION_APPS_WSM_APPID-KEY and get username.

As AppId users are created during installation by provisioning tools so not aware about current password and i also didn't want to reset with new password because not sure what all other places application stored it.

To retrieve current password, I used  listCred command. (Please find below steps to retrieve current password of Appid).


[oracle@famachine logs]$ cd /u01/oracle/fa/products/fusionapps/oracle_common/common/bin/
[oracle@famachine bin]$ ./wlst.sh
wls:/offline> connect('weblogic_fa','Welcome1','t3://commoninternal.mycompany.com:7001')
Connecting to t3://commoninternal.mycompany.com:7001 with userid weblogic_fa ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'CommonDomain'.
wls:/CommonDomain/serverConfig> listCred(map="oracle.wsm.security",key=" FUSION_APPS_WSM_APPID-KEY ")
Already in Domain Runtime Tree

PASSWORD:tzj|mgqv5cPi2w

After i got password, i performed ldapsearch command using id/password and got password expiration error.

[oracle@idmmachine logs]$  ldapsearch -h idmmachine -p 389 -D "cn=FUSION_APPS_WSM_APPID,cn=AppIDUsers,cn=Users,dc=mycompany,dc=com" -w "tzj|mgqv5cPi2w" -b "dc=mycompany,dc=com" -s sub "objectclass=inetorgperson" dn

ldap_bind: Invalid credentials

ldap_bind: additional info: Password Policy Error :9000: GSL_PWDEXPIRED_EXCP :Your Password has expired. Please contact the Administrator to change your password.

[oracle@idmmachine logs]$

so to fix issue, i have reset password for "FUSION_APPS_WSM_APPID".

Login on ODSM Console ( http://idmserver.domainname.com:7006/odsm) as cn=orcladmin.

click on Data Browser

click on dc=com,dc=mycompany,cn=users,cn=appIdUsers,cn=FUSION_APPS_WSM_APPID

Click on attributes

reset password to your existing password.

To avoid similar issue in future, create new password policy and assign to all the service accounts.

1. Log in to ODSM

2. Click Security (tab) -> Password Policy

3. Create a new password policy for service accounts.
Under the Effective Subtree tab of the new password policy, enter the service accounts:

Click Apply.

Targets found: Failed to find any product family.

Last week, i have installed Oracle Enterprise Manager 12c(12.1.0.1) and when trying to add Fusion Application Instance, i have received below error:

"Targets found: Failed to find any product family. Check host, port, credentials, protocol and be sure the admin server is running. Reason: Failed to discover any product family. Only Weblogic related targets are found. For discovering Fusion Middleware use Fusion Middleware Discovery"

Solution:

I have followed   Oracle® Enterprise Manager Cloud Control Basic Installation Guide  for agent configuration but was not able to discoverer product family. Then i talked to Oracle and got missing step details which fixed my issue. 

If you are stuck in same step, Provide below database grants to fusion_runtime user which fixed  issue and able to discoverer all the targets...

1. grant execute on DBMS_SYSTEM to  FUSION_RUNTIME; 
2. grant select on  v_$parameter to  FUSION_RUNTIME;
3. grant select on gv_$instance to  FUSION_RUNTIME;
4. grant select on V_$INSTANCE to  FUSION_RUNTIME;
5. grant select on gv_$diag_info to  FUSION_RUNTIME;
6. grant select on gv_$parameter to  FUSION_RUNTIME;
7. grant select on gv_$instance to  FUSION_RUNTIME; 

Happy Monitoring..