Oracle EBS weblogic 禁用筛选器T3导致weblogic服务不能重启问题

背景:
2020年1月15号安全网站出了一个 CVE-2020-2546,于是按照里面的“指示”,做了T3禁用,导致weblogic服务启动失败,EBS应用不能正常登陆,启动失败日常如下:

AdminServer logs are located at /u01/DEV/app/fs1/FMW_Home/user_projects/domains/EBS_domain_DEV/servers/AdminServer/logs

01/17/20-09:13:58 :: adadminsrvctl.sh: exiting with status 1

================================================================================

01/17/20-09:24:07 :: adadminsrvctl.sh version 120.10.12020000.10
Validated the passed arguments for the option ebs-get-serverstatus
AdminServer is currently not running.

Validated the passed arguments for the option ebs-nmstart-adminsrv
Checking if the Admin Server is already up.
The Admin Server is not already up.
FMW Version is 11.1.1.9
Checking if the Node Manager is already up..
Connecting to Node Manager …
Successfully Connected to Node Manager.

The Node Manager is already up.

Starting server AdminServer …
Error Starting server AdminServer: weblogic.nodemanager.NMException: Exception while starting server ‘AdminServer’

ERROR: Unable to connect the AdminServer.

StackTrace:
java.io.IOException
at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:209)
at weblogic.management.remote.common.ClientProviderBase.newJMXConnector(ClientProviderBase.java:97)
at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnectorFactory.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
at oracle.apps.ad.util.WLUtil.initMBeanServerConnection(WLUtil.java:131)
at oracle.apps.ad.tools.configuration.EBSProvisioner.ebs_nmstart_adminsrv(EBSProvisioner.java:3682)
at oracle.apps.ad.tools.configuration.EBSProvisioner.ebs_nmstart_adminsrv(EBSProvisioner.java:3915)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at oracle.apps.ad.tools.configuration.EBSProvisioner.main(EBSProvisioner.java:8880)
Caused by: javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3://devfin.guobaojinrong.com:7001: Destination unreachable; nested exception is:
java.net.ConnectException: 拒绝连接; No available router to destination]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:40)
at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:366)
at weblogic.jndi.Environment.getContext(Environment.java:315)
at weblogic.jndi.Environment.getContext(Environment.java:285)
at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
at javax.naming.InitialContext.init(InitialContext.java:242)
at javax.naming.InitialContext.<init>(InitialContext.java:216)
at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:193)
… 11 more
Caused by: java.net.ConnectException: t3://devfin.guobaojinrong.com:7001: Destination unreachable; nested exception is:
java.net.ConnectException: 拒绝连接; No available router to destination
at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:216)
at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:170)
at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:165)
at weblogic.jndi.WLInitialContextFactoryDelegate$1.run(WLInitialContextFactoryDelegate.java:345)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:340)
… 19 more
Caused by: java.rmi.ConnectException: Destination unreachable; nested exception is:
java.net.ConnectException: 拒绝连接; No available router to destination
at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:470)
at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:321)
at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:262)
at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:199)
at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:238)
at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:200)
… 25 more

AdminServer logs are located at /u01/DEV/app/fs1/FMW_Home/user_projects/domains/EBS_domain_DEV/servers/AdminServer/logs

01/17/20-09:24:24 :: adadminsrvctl.sh: exiting with status 1

================================================================================

应用界面登陆也显示weblogic连接失败。

 

解决过程探索:

一般遇到这类问题,第一映像就是先恢复之前的设置,于是去找相关的表或者配置文件,从表中很难找到指向表,于是从另外一个正常的环境进入相应的weblogic主页面,并进入之前配置的页面查看帮助,于是找到了一个比较有指向性的文件(不知道具体文件在哪个目录下,通过find命令搜索),config.xml。

以下文件是来自帮助文件里面的内容:

Changes take effect after you redeploy the module or restart the server.

If this attribute configures a module that you deploy (such as an application or a JDBC data source that is part of an application) or a system resource whose configuration is saved in a descriptor file instead of in the domain’s config.xml file (such as a JDBC data source that is scoped at the system level), the module or resource cannot process the change until you redeploy it or restart its host server. If the module is a component in an application, Oracle recommends that you redeploy the entire application to avoid complications due to intra-application dependencies.

If this attribute configures some other part of the domain (such as a server, a cluster, or an EJB container), the system cannot process the change until you restart the server or cluster.

进入相应目录后,发现这个config.xml文件有很多序列号的文件名,如config39.xml,config38.xml,看这些文件的更新时间,推测应该是每次更新都会做一次备份,于是备份当前的config.xml文件,恢复最近的一个config.xml文件,然后重启weblogic服务,重启(正常重启,登陆页面也正常出现,并可登陆)如下文本:

[appldev@devfin scripts]$ sh adadminsrvctl.sh start

You are running adadminsrvctl.sh version 120.10.12020000.10

Enter the WebLogic Admin password:
Enter the APPS Schema password:
Starting WLS Admin Server…
Refer /u01/DEV/app/fs1/inst/apps/DEV_devfin/logs/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at /u01/DEV/app/fs1/FMW_Home/user_projects/domains/EBS_domain_DEV/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile /u01/DEV/app/fs1/inst/apps/DEV_devfin/logs/appl/admin/log/adadminsrvctl.txt for more information …

说明:ebs weblogic尽量少变动,在改动前也一定要做好备份。

相关参考文件:
https://www.oracle.com/security-alerts/cpujan2020.html

https://docs.oracle.com/cd/E23943_01/web.1111/e13707/ssl.htm#SECMG389

http://www.ijiandao.com/2b/baijia/345629.html

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=508979212187743&id=1280374.1&_afrWindowMode=0&_adf.ctrl-state=e2y1sye3v_1027

Oracle EBS R12.1.3登陆环境问题(JSP问题)

错误描述:

Unable to generate forwarding URL. Exception: oracle.apps.fnd.common.AppsException: java.lang.NullPointerException: null Connection

方案参考:
EBS Homepage Errors Unable To Generate Forwarding URL Exception oracle.apps.fnd.common.AppsException java.lang.NullPointerException null Connection (文档 ID 2055421.1)

说明:查看文档,发现官方是认为系统缓存或者某些jsp文件crash造成,因此主要思路是将_pages相关jsp文件进行整体重编译,然后删除相关cache文件来解决。

执行过程(关闭应用及备份删除相关操作、重启应用不在此记录):

[appldev@erpdevap bin]$ ./ojspCompile.pl –compile –flush -p 2
logfile set: /app/DEV/inst/apps/DEV_erpdevap/logs/appl/rgf/ojsp/ojspc_error.log
starting…(compiling all)
using 10i internal ojsp ver: 10
synchronizing dependency file:
enumerating jsps…8095
parsing jsp…8095
writing deplist…8095
initializing compilation:
eliminating children…6024 (-2071)
translating and compiling:
translating jsps…6024/6024 in 1m33s
compiling jsps…6024/6024 in 6m20s
Finished!

跟踪错误日志:

[BEGIN] 2020/1/13 17:38:11
[appldev@erpdevap _pages]$ tailf /app/DEV/inst/apps/DEV_erpdevap/logs/appl/rgf/ojsp/ojspc_error.log
[19659] compiling: 7s elapsed, 50 successful 0 failed
[19658] compiling: 7s elapsed, 50 successful 0 failed
[19659] compiling: 6s elapsed, 50 successful 0 failed
[19659] compiling: 4s elapsed, 50 successful 0 failed
[19658] compiling: 7s elapsed, 50 successful 0 failed
[19659] compiling: 2s elapsed, 50 successful 0 failed
[19658] compiling: 3s elapsed, 50 successful 0 failed
[19659] compiling: 2s elapsed, 24 successful 0 failed
COMPILED: 6024 [failed: 0] in 6m20s
18890 FINISHING Mon Jan 13 17:36:33 2020

[appldev@erpdevap _pages]$ exit
logout

[END] 2020/1/13 17:40:35

Oracle ebs冲销凭证报错APP-SQLGL-08087

错误如图:

原因:

此种现象是因为之前对凭证做过冲销,但是在未过账的时候,将生成的冲销凭证删除,但是冲销的记录在凭证头表字段accrual_rev_period_name留下记录,记录的内容是报错界面默认的冲销期间(2019-10)。

解决方案:

直接更新gl_je_headers表中字段accrual_rev_period_name为空即可正常冲销动作(更新前一定要备份,更新后重新期间可选)。

Oracle 在线重定义(普通表变更为分区表)

–背景:cux_gl_interfacebak数据量过大(cux_gl_interfacebak有主键),需按accouting_date按年分区,以下命令直接在command窗口执行即可。
–1、检查需要在线冲定义的表是否
begin
dbms_redefinition.can_redef_table(‘apps’,’cux_gl_interfacebak’,dbms_redefinition.cons_use_pk);
end;
/

或验证是否可以通过rowid方式定义

begin
–dbms_redefinition.can_redef_table(‘scott’,’tb_cablecheck_equipment_bak’,2);
dbms_redefinition.can_redef_table(‘apps’,’cux_gl_interfacebak’,dbms_redefinition.cons_use_rowid);
end;
/

–2、创建中间表
create table CUX_GL_INTERFACEBAK_1
(
source_batch_id VARCHAR2(50) not null,
source_line_id NUMBER(10),
je_group_id VARCHAR2(50) not null,
ledger_id VARCHAR2(50) not null,
accounting_date DATE not null,
process_date DATE not null,
je_category_name VARCHAR2(25) not null,
je_source_name VARCHAR2(25) not null,
currency_code VARCHAR2(15) not null,
currency_conversion_date DATE,
currency_conversion_rate NUMBER(38,2),
currency_conversion_type VARCHAR2(30),
entered_dr NUMBER(38,2),
entered_cr NUMBER(38,2),
accounted_dr NUMBER(38,2),
accounted_cr NUMBER(38,2),
actual_flag VARCHAR2(25) not null,
import_flag VARCHAR2(1) not null,
import_date VARCHAR2(25),
gl_request_id NUMBER(30),
error_message VARCHAR2(255),
doc_seq_num VARCHAR2(100),
segment1 VARCHAR2(25),
segment2 VARCHAR2(25),
segment3 VARCHAR2(25),
segment4 VARCHAR2(25),
segment5 VARCHAR2(25),
segment6 VARCHAR2(25),
segment7 VARCHAR2(25),
segment8 VARCHAR2(25),
segment9 VARCHAR2(25),
segment10 VARCHAR2(25),
segment11 VARCHAR2(25),
segment12 VARCHAR2(25),
segment13 VARCHAR2(25),
line_description VARCHAR2(240),
attribute1 VARCHAR2(25),
attribute2 VARCHAR2(150),
attribute3 VARCHAR2(150),
attribute4 VARCHAR2(150),
attribute5 VARCHAR2(150),
attribute6 VARCHAR2(150),
attribute7 VARCHAR2(150),
attribute8 VARCHAR2(150),
attribute9 VARCHAR2(150),
attribute10 VARCHAR2(150),
attribute11 VARCHAR2(150),
attribute12 VARCHAR2(150),
attribute13 VARCHAR2(150),
attribute14 VARCHAR2(150),
attribute15 VARCHAR2(150),
source_key_id NUMBER(10) not null
)

partition by range(accounting_date)(
PARTITION tb_cablecheck_equipment_p1 VALUES LESS THAN (TO_DATE(‘2017-01-01′,’YYYY-MM-DD’)),
PARTITION tb_cablecheck_equipment_p2 VALUES LESS THAN(TO_DATE(‘2018-01-01’, ‘YYYY-MM-DD’)),
PARTITION tb_cablecheck_equipment_p3 VALUES LESS THAN(TO_DATE(‘2019-01-01’, ‘YYYY-MM-DD’)),
PARTITION tb_cablecheck_equipment_p4 VALUES LESS THAN(TO_DATE(‘2020-01-01’, ‘YYYY-MM-DD’)),
PARTITION tb_cablecheck_equipment_p5 VALUES LESS THAN(TO_DATE(‘2021-01-01’, ‘YYYY-MM-DD’)),
PARTITION tb_cablecheck_equipment_p6 VALUES LESS THAN(MAXVALUE)
);

–3、进行冲定义命令
begin
dbms_redefinition.start_redef_table(‘apps’,’CUX_GL_INTERFACEBAK’,’CUX_GL_INTERFACEBAK_1′,null,2);
end;
/

–4、复制依赖对象
declare
num_errors pls_integer;
begin
dbms_redefinition.copy_table_dependents(‘apps’, ‘CUX_GL_INTERFACEBAK’,’CUX_GL_INTERFACEBAK_1′,
dbms_redefinition.cons_orig_params, true, true, true, true, num_errors);
end;
/

–5、同步中间表,保证数据的一致性
begin
dbms_redefinition.sync_interim_table(‘apps’,’CUX_GL_INTERFACEBAK’,’CUX_GL_INTERFACEBAK_1′);
end;
/

–6、完成重定义命令
begin
dbms_redefinition.finish_redef_table(‘apps’,’CUX_GL_INTERFACEBAK’,’CUX_GL_INTERFACEBAK_1′);
end;
/
–7、验证冲定义是否正常
select * from CUX_GL_INTERFACEBAK partition(tb_cablecheck_equipment_p4);

select *
from cux_gl_interfacebak partition(tb_cablecheck_equipment_p3)
where 1 = 1
and segment3 = ‘6031010101’

–8、删除表
drop table apps.CUX_GL_INTERFACEBAK_1;