Darwin IT

Subscribe to Darwin IT feed
Darwin-IT professionals do ICT-projects based on a broad range of Oracle products and technologies. We write about our experiences and share our thoughts and tips.Martien van den Akkerhttps://plus.google.com/110503432901891966671noreply@blogger.comBlogger392125
Updated: 5 hours 14 min ago

Start, Restart and stop OHS Components

Tue, 2017-03-21 05:08
In Fusion Middleware 12c the Oracle HTTP Server is now configured as SystemComponents within a Weblogic Domain. This can be embedded in a FMW Domain, like SOA, OSB, or WebCenter.  Or it can be in a stand alone domain.

When configured, there are scripts to start and stop the system components in the bin folder of the domain: startComponent.sh and stopComponent.sh.

These scripts first do a readDomain() to read the domain before actually starting or stopping the component. An OHS Standalone Domain is not that large, but does take a few seconds. With an embedded OHS it can actually a quite expensive operation. And is done twice when you need to stop and start a component. And that per component.

So, this can be smarter. Yesterday, I created a script that first looks for a property in the property file, that lists the possible components as a comma seperated list. It only does a readDomain() when that property is not defined.

The property file ohs.properties contains:

#############################################################################
# Set OHS SA Domain properties
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.0, 2017-02-20
#
#############################################################################
#
#Per domain nodemanager...
nmHome=/u01/data/oracle/config/domains/ohs_domain/nodemanager
nmHost=localhost
nmPort=5555
nmType=plain
#
#Domain
wlsDomainName=ohs_domain
wlsDomainsHome=/u01/data/oracle/config/domains
#
#AdminServer
adminUser=weblogic
adminPwd=welcome1
#usrCfgFile=wlsCfgFile
#usrKeyFile=wlsKeyFile
#
# OHS Component List
ohsComponents=ohs1

The wlst script startStopOhs.py is:
#############################################################################
# Start OHS via NodeManager.
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.0, 2017-03-01
#
#############################################################################
# Modify these values as necessary
import sys, traceback
scriptName = sys.argv[0]
#
#
lineSeperator='__________________________________________________________________________________'
#
#
def usage():
print 'Call script as: '
print 'Windows: wlst.cmd '+scriptName+' -loadProperties localhost.properties'
print 'Linux: wlst.sh '+scriptName+' -loadProperties environment.properties'
print 'Property file should contain the following properties: '
print "adminUrl=localhost:7101"
print "adminUser=weblogic"
print "adminPwd=welcome1"
#
#
def connectToNM(nmHost, nmPort, nmHome, wlsDomainName, wlsDomainHome, nmType):
try:
print(lineSeperator)
print('Try to connect to the Node Manager')
nmConnect(username=adminUser, password=adminPwd, host=nmHost, port=nmPort, domainName=wlsDomainName, domainDir=wlsDomainHome, nmType=nmType)
print('Connected to the Node Mananger')
except WLSTException:
message='Apparently NodeManager not Started!'
print (message)
raise Exception(message)
#
#
def getSystemComponents(wlsDomainHome):
try:
print(lineSeperator)
print('Get SystemComponents')
try:
sysComps=ohsComponents.split(',')
except NameError, e:
print 'Property ohsComponents not set, read from Domain'
print('Read domain from : '+wlsDomainHome)
readDomain(wlsDomainHome)
cd('/SystemComponent')
sysComps=ls(returnMap='true')
return sysComps
except WLSTException:
message='Exception getting SystemComponents'
print (message)
raise Exception(message)
#
# Start a System Component
def startSystemComponent(sysComp):
try:
print(lineSeperator)
print('StartSystemComponent '+sysComp)
nmStart(serverName=sysComp, serverType='OHS')
except WLSTException:
message='Exception starting SystemComponent: '+sysComp
print (message)
raise Exception(message)
#
# Stop a System Component
def stopSystemComponent(sysComp):
try:
print(lineSeperator)
print('StartSystemComponent '+sysComp)
nmKill(serverName=sysComp, serverType='OHS')
except WLSTException:
message='Exception stopping SystemComponent: '+sysComp
print (message)
raise Exception(message)
#
# Start, restart or stop System Components, based on startStopOption.
def startStopSystemComponents(startStopOption, sysComps):
try:
print(lineSeperator)
print('StartSystemComponents')
if startStopOption=="stop" or startStopOption=="restart":
for sysComp in sysComps:
stopSystemComponent(sysComp)
if startStopOption=="start" or startStopOption=="restart":
for sysComp in sysComps:
startSystemComponent(sysComp)
except WLSTException:
message='Exception starting or stopping SystemComponents, startStopOption:'+startStopOption
print (message)
raise Exception(message)
#
#
def main():
try:
startStopOption=sys.argv[1]
wlsDomainHome = wlsDomainsHome+'/'+wlsDomainName
print(lineSeperator)
if startStopOption=="start" or startStopOption=="restart":
print('(Re)Start OHS for domain in : '+wlsDomainHome+', with option '+startStopOption)
else:
print('Stop OHS for domain in : '+wlsDomainHome+', with option '+startStopOption)
print(lineSeperator)
print ('Connect to the Node Manager')
connectToNM(nmHost, nmPort, nmHome, wlsDomainName, wlsDomainHome, nmType)
#
systemComponents=getSystemComponents(wlsDomainHome)
#
startStopSystemComponents(startStopOption, systemComponents)

except NameError, e:
print('Apparently properties not set.')
print "Please check the property: ", sys.exc_info()[0], sys.exc_info()[1]
usage()
except:
apply(traceback.print_exception, sys.exc_info())
exit(exitcode=1)
#
main();

It uses a startStopOption that can have the values:

  • start
  • restart
  • stop
Based on that, it determines if the components need to be stopped, and/or, started again.

So, this time I only have one wlst script. I could make my weblogic server start scripts smarter the same way...

To start the ohs you can use startOhs.sh:
#!/bin/bash
. ./ohs12c_env.sh
echo
echo Start OHS
wlst.sh ./startStopOhs.py start -loadProperties ohs.properties

To restart the ohs you can use restartOhs.sh:
#!/bin/bash
. ./ohs12c_env.sh
echo
echo Restart OHS
wlst.sh ./startStopOhs.py restart -loadProperties ohs.properties

And to stop the ohs you can use stopOhs.sh:
#!/bin/bash
. ./ohs12c_env.sh
echo
echo Stop OHS
wlst.sh ./startStopOhs.py stop -loadProperties ohs.properties

With a OHS Standalone domain, you probably have stored the nodemanager credentials once with the storeConfig option. In that case you can remove the credentials in the nmConnnect line:

#nmConnect(username=adminUser, password=adminPwd, host=nmHost, port=nmPort, domainName=wlsDomainName, domainDir=wlsDomainHome, nmType=nmType)
nmConnect(host=nmHost, port=nmPort, domainName=wlsDomainName, domainDir=wlsDomainHome, nmType=nmType)

For a standalone OHS the ohs12c_env.sh script contains:
#!/bin/bash
echo set OHS StandAlone 12cR2 environment
#export JAVA_HOME=/usr/java/jdk1.8.0_101
#export ORACLE_BASE=/app/oracle
export ORACLE_BASE=/u01/app/oracle
export OHS_HOME=$ORACLE_BASE/product/middleware/OHS12212
export WL_HOME=${OHS_HOME}/wlserver
export SHARED_CONFIG_DIR=/u01/data/oracle/config
export OHS_DOMAIN_NAME=ohs_domain
export OHS_DOMAIN_HOME=$SHARED_CONFIG_DIR/domains/$OHS_DOMAIN_NAME
export NODEMGR_HOME=$OHS_DOMAIN_HOME/nodemanager
#
echo Set Weblogic Common Env.
. $OHS_HOME/oracle_common/common/bin/commEnv.sh
#
export PATH=$OHS_HOME/oracle_common/common/bin:$PATH


Install and configure Oracle HTTP Server Standalone

Thu, 2017-02-16 09:30
Last week I had an assignment to install and configure Oracle HTTP Server as a reversed proxy in a DMZ. Many years ago I worked with Apache a little, so I had not have the details at hand.

Although installing and configuring the HTTP Server is not hard, I found that I had to do quite some searching around to get to all the essentials. To help you out, and to have it logged for myself, let's get through it.

Install OHS Software
For installing the OHS Software I created a script as I did for other FMW installations (FMW Infrastructure, OSB, SOA/BPM, etc.). I did my installation on a Oracle Linux 7 VM, that already contained the database and Weblogic. So I did not need to add the necessary packages. See for instance the requirements doc.

The script starts with the following settings:

Variable
Meaning
Suggested Value (12.2.1.2)OHS_ZIP_HOMELocation of the Zip file(s) of the installation. $PWD/../Zipped/OHSOHS_INSTALL_HOMELocation to extract the Zip file(s) of the installation to.$PWD/../Extracted/stage/OHSOHS_INSTALL_BINName of the installer within the Zip file(s)fmw_12.2.1.2.0_ohs_linux64.binOHSSA_INSTALL_RSPName of the response file to use in the installerfmw_12.2.1.2.0_ohs_sa.rspOHSSA_INSTALL_RSP_TPLName of the response file template, to expand to the actual response file.$OHSSA_INSTALL_RSP.tplOHS_INSTALL_ZIPName of the zip file that contains the installer.V789368-01.zip
As in earlier blogs I use a fmw12c_env.sh script to set variables to point to the locations/homes of the software. For this script I need the following variables (so add them if not present):
Variable
Meaning
Suggested Value (12.2.1.2)ORACLE_BASEBase location of the oracle installations/u01/app/oracle or /app/oracleOHSSA_HOMEHome for the installation of a StandAlone OHS.$ORACLE_BASE/product/middleware/OHS12212

The script installOHS_SA.sh is as follows:

#!/bin/bash
. $PWD/fmw12c_env.sh
#
export OHS_ZIP_HOME=$PWD/../Zipped/OHS
export OHS_INSTALL_HOME=$PWD/../Extracted/stage/OHS
export OHS_INSTALL_BIN=fmw_12.2.1.2.0_ohs_linux64.bin
export OHSSA_INSTALL_RSP=fmw_12.2.1.2.0_ohs_sa.rsp
export OHSSA_INSTALL_RSP_TPL=$OHSSA_INSTALL_RSP.tpl
export OHS_INSTALL_ZIP=V789368-01.zip
#
# Oracle Webtier 12c
if [ ! -d "$OHSSA_HOME" ]; then
#
#Unzip OHS Software
if [ ! -f "$OHS_INSTALL_HOME/$OHS_INSTALL_BIN" ]; then
if [ -f "$OHS_ZIP_HOME/$OHS_INSTALL_ZIP" ]; then
if [ ! -d "$OHS_INSTALL_HOME" ]; then
echo Create folder "$OHS_INSTALL_HOME"
mkdir -p "$OHS_INSTALL_HOME"
fi
echo Unzip $OHS_ZIP_HOME/$OHS_INSTALL_ZIP to $OHS_INSTALL_HOME/$OHS_INSTALL_BIN
unzip -o $OHS_ZIP_HOME/$OHS_INSTALL_ZIP -d $OHS_INSTALL_HOME
else
echo $OHS_INSTALL_HOME/$OHS_INSTALL_ZIP does not exist!
fi
else
echo $OHS_INSTALL_BIN already unzipped
fi
if [ -f "$OHS_INSTALL_HOME/$OHS_INSTALL_BIN" ]; then
echo Substitute $OHSSA_INSTALL_RSP_TPL to $OHSSA_INSTALL_RSP
envsubst < $OHSSA_INSTALL_RSP_TPL > $OHSSA_INSTALL_RSP
echo Install Oracle HTTP Server 12cR2 StandAlone
$OHS_INSTALL_HOME/$OHS_INSTALL_BIN -silent -responseFile $PWD/$OHSSA_INSTALL_RSP
else
echo $OHS_INSTALL_BIN not available!
fi
else
echo $OHSSA_HOME available: WebTier installed
fi

The script checks if the OHS Home already exists, if so it concludes that the software apparently is installed already. If not, it checks if the installer is available, and if that is not the case it checks if the zip is available. It will unzip the installer if needed and if one of the checks fail it will stop.

A new thing I found out after my earlier blogs is that I learned about the envsubst command that is available in Oracle Linux. It enables you to expand the environment variables that occur in a file. And for this script this is handy, because it needs a response file for the installer, but that contains for instance an Oracle Home to be set to the target $OHSSA_HOME from the fmw12c_env.sh script. In my earlier scripts, when I needed or wanted to install using other folder structures I needed to change both the script and the response file. Now I used the response file saved from the manual installation and then replaced the folder with the particular environment variable.

The response file template, I called fmw_12.2.1.2.0_ohs_sa.rsp.tpl, looks like:
[ENGINE]

#DO NOT CHANGE THIS.
Response File Version=1.0.0.0.0

[GENERIC]

#Set this to true if you wish to skip software updates
DECLINE_AUTO_UPDATES=true

#My Oracle Support User Name
MOS_USERNAME=

#My Oracle Support Password
MOS_PASSWORD=<SECURE VALUE>

#If the Software updates are already downloaded and available on your local system, then specify the path to the directory where these patches are available and set SPECIFY_DOWNLOAD_LOCATION to true
AUTO_UPDATES_LOCATION=

#Proxy Server Name to connect to My Oracle Support
SOFTWARE_UPDATES_PROXY_SERVER=

#Proxy Server Port
SOFTWARE_UPDATES_PROXY_PORT=

#Proxy Server Username
SOFTWARE_UPDATES_PROXY_USER=

#Proxy Server Password
SOFTWARE_UPDATES_PROXY_PASSWORD=<SECURE VALUE>

#The oracle home location. This can be an existing Oracle Home or a new Oracle Home
ORACLE_HOME=${OHSSA_HOME}

#Set this variable value to the Installation Type selected as either Standalone HTTP Server (Managed independently of WebLogic server) OR Collocated HTTP Server (Managed through WebLogic server)
INSTALL_TYPE=Standalone HTTP Server (Managed independently of WebLogic server)

#Provide the My Oracle Support Username. If you wish to ignore Oracle Configuration Manager configuration provide empty string for user name.
MYORACLESUPPORT_USERNAME=

#Provide the My Oracle Support Password
MYORACLESUPPORT_PASSWORD=<SECURE VALUE>

#Set this to true if you wish to decline the security updates. Setting this to true and providing empty string for My Oracle Support username will ignore the Oracle Configuration Manager configuration
DECLINE_SECURITY_UPDATES=true

#Set this to true if My Oracle Support Password is specified
SECURITY_UPDATES_VIA_MYORACLESUPPORT=false

#Provide the Proxy Host
PROXY_HOST=

#Provide the Proxy Port
PROXY_PORT=

#Provide the Proxy Username
PROXY_USER=

#Provide the Proxy Password
PROXY_PWD=<SECURE VALUE>

#Type String (URL format) Indicates the OCM Repeater URL which should be of the format [scheme[Http/Https]]://[repeater host]:[repeater port]
COLLECTOR_SUPPORTHUB_URL=


Create a folder structure, place the scripts fmw12c_env.sh, installOHS_SA.sh and fmw_12.2.1.2.0_ohs_sa.rsp.tpl  in a folder named scripts. Then create a folder named Zipped/OHS next to it. Download the OHS installer from edelivery.oracle.com and save it in that folder.
Also create an empty folder named Extracted/stage/OHS next to it. Then run the installOHS_SA.sh.


Configure the DomainUnfortunately it is apparently not possible to create the domain  with a wlst script. Main reason for this (I think) is that the wlst installation in the OHS FMW_home lacks an implementation of the encrypt() method to encrypt a password against the SA-domain folder. Therefor you can't set the (encrypted) domain password and nodemanager password. And thus you can't start the OHS components in the regular way using the nodemanager.

So configuring the domain is done manually using the configurator.
Start the config.sh vanuit $OHS_HOME/oracle_common/common/bin:
[oracle@darlin-vce-db scripts]$ cd /u01/app/oracle/product/middleware/OHS12212/oracle_common/common/bin

[oracle@darlin-vce-db bin]$ ls

cam_clonedunpack.sh cam_wlst.sh config.sh setHomeDirs.sh

cam_config.sh clonedunpack.sh configWallet.sh unpack.sh

cam_pack.sh commBaseEnv.sh getproperty.sh wlst.sh

cam_reconfig.sh commEnv.sh pack.sh

cam_unpack.sh commExtEnv.sh reconfig.sh

[oracle@darlin-vce-db bin]$ ./config.sh


Provide a domain home like: /u01/app/work/domains/ohs_domain.
Check the Oracle HTTP Server (Standalone) option.
The documentation suggest to select the Oracle Hotspot JDK, delivered with the installation.
Here you can add system components to the domain.
Per system component provide the listen address and port, as well as the Admin Host and Port.
Let's keep the defaults.
Keep the ‘Per Domain Default location’ option.
From Weblogic 12c this is the standard.
Keep the username weblogicand provide a password. For demo purposes I use the password welcome1.

Actually this is the step I couldn't manage to do with WLST.
Validate the settings.
The progress of the creation of the domain.
Confirm the configuration of the domain.Further configuration of OHSYou'll need to add routes. For the plain http routes you can add a configuration file, like service_vh.conf (where service is the name of the application or purpose for which you want to use OHS). In my case I was routing an APEX application through ORDS, so I named it ords_vh.conf.
An example of the file is:
<VirtualHost *:7777>
ServerName admin.darlin-vce-db:7777
ServerAdmin martien.van.den.akker@darwin-it.nl
RewriteEngine On
RewriteOptions inherit
# Admin Server and EM
<Location /console>
WLSRequest ON
WebLogicHost darlin-vce-db
WeblogicPort 7001
</Location>
<Location /consolehelp>
WLSRequest ON
WebLogicHost darlin-vce-db
WeblogicPort 7001
</Location>
<Location /em>
WLSRequest ON
WebLogicHost darlin-vce-db
WeblogicPort 7001
</Location>
# APEX in Admin Server
<Location /ords>
WLSRequest ON
WebLogicHost darlin-vce-db
WeblogicPort 7001
</Location>
<Location /ords/apex_admin>
WLSRequest ON
WebLogicHost darlin-vce-db
WeblogicPort 7001
</Location>
<Location /i>
WLSRequest ON
WebLogicHost darlin-vce-db
WeblogicPort 7001
</Location>
</VirtualHost>

Place this file the folder $OHS_DOMAIN_HOME/config/fmwconfig/components/OHS/ohs1/moduleconf. All files with extension ‘.conf’ in this folder are picked up automatically by the OHS component.
Start OHSAdapt Nodemanager
Change the nodemanager.properties in $OHS_DOMAIN/nodemanager/nodemanager.properties:
#Node manager properties
#Tue Jan 24 14:41:26 CET 2017
DomainsFile=/u01/app/work/domains/ohs_domain/nodemanager/nodemanager.domains
LogLimit=0
PropertiesVersion=12.2.1
AuthenticationEnabled=true
NodeManagerHome=/u01/app/work/domains/ohs_domain/nodemanager
JavaHome=/usr/java/jdk1.8.0_74
LogLevel=INFO
DomainsFileEnabled=true
ListenAddress=localhost
NativeVersionEnabled=true
#ListenPort=5556
ListenPort=7555
LogToStderr=true
weblogic.StartScriptName=startWebLogic.sh
#SecureListener=true
SecureListener=false
LogCount=1
QuitEnabled=false
LogAppend=true
weblogic.StopScriptEnabled=false
StateCheckInterval=500
CrashRecoveryEnabled=false
weblogic.StartScriptEnabled=true
LogFile=/u01/app/work/domains/ohs_domain/nodemanager/nodemanager.log
LogFormatter=weblogic.nodemanager.server.LogFormatter
ListenBacklog=50


In this file change the following properties:
  • SecureListener=false
  • ListenPort=5555

If you already have a nodemanager running, possibly for another domain (not likely when using this in a DMZ) you can use another port. For instance 7555.
As a good practice I tend to duplicate the lines, comment out the original ones and change the duplicates.
Because we changed the nodemanager config, we need to update the domain. This can be done by a wlst file like the following setNodeManagerListenPort.py:
readDomain('/u01/app/work/domains/ohs_domain')
cd ('Machine/localmachine/NodeManager/localmachine')
set('ListenPort',7555)
set('NMType','plain')
updateDomain()

That can be run as:
[oracle@darlin-vce-db OHSSaDomain]$ . ./ohs12c_env.sh
set OHS StandAlone 12cR2 environment
Set Weblogic Common Env.
[oracle@darlin-vce-db OHSSaDomain]$ wlst.sh setNodeManagerListenPort.py


Here the content of the environment script ohs12c_env.sh is as follows:
#!/bin/bash
echo set OHS StandAlone 12cR2 environment
#export JAVA_HOME=/usr/java/jdk1.8.0_101
#export ORACLE_BASE=/app/oracle
export ORACLE_BASE=/u01/app/oracle
export OHS_HOME=$ORACLE_BASE/product/middleware/OHS12212
export WL_HOME=${OHS_HOME}/wlserver
export SHARED_CONFIG_DIR=/u01/app/work
export OHS_DOMAIN_NAME=ohs_domain
export OHS_DOMAIN_HOME=$SHARED_CONFIG_DIR/domains/$OHS_DOMAIN_NAME
export NODEMGR_HOME=$OHS_DOMAIN_HOME/nodemanager
#
echo Set Weblogic Common Env.
. $OHS_HOME/oracle_common/common/bin/commEnv.sh
#
export PATH=$OHS_HOME/oracle_common/common/bin:$PATH

Start NodemanagerYou can start the nodemanager using $DOMAIN_HOME/bin/startNodemanager.sh. It can also using a custom script like:

#!/bin/bash
. ohs12c_env.sh
echo Start NodeManager
nohup $OHS_DOMAIN_HOME/bin/startNodeManager.sh > $OHS_DOMAIN_HOME/nodemanager/nm.out&

This starts the nodemanager in the background.
Start OHS
When you start OHS initially, you can do it like below, to store the userconfig:
[oracle@darlin-vce-db bin]$ ./startComponent.sh ohs1 storeUserConfig
Starting system Component ohs1 ...

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Reading domain from /u01/app/work/domains/ohs_domain


Please enter Node Manager password:
Creating the key file can reduce the security of your system if it is not kept in a secured location after it is created. Creating new key...
The username and password that were used for this WebLogic NodeManager connection are stored in /home/oracle/.wlst/nm-cfg-ohs_domain.props and /home/oracle/.wlst/nm-key-ohs_domain.props.
Connecting to Node Manager ...
Successfully Connected to Node Manager.
Starting server ohs1 ...
Successfully started server ohs1 ...
Successfully disconnected from Node Manager.


Exiting WebLogic Scripting Tool.

Done
[oracle@darlin-vce-db bin]$

This starts the OHS and stores the password encrypted. Every following time, you can start it like:
#!/bin/bash
. ./ohs12c_env.sh
echo Stop Ohs1
$OHS_DOMAIN_HOME/bin/stopComponent.sh ohs1
echo Start Ohs1
$OHS_DOMAIN_HOME/bin/startComponent.sh ohs1


FinallyIn my case I had to configure SSL as well. This included creating a wallet with a certificate etc.
I'll save that for another article, because this one is already quite extensive.

Property expansion under windows.

Thu, 2017-01-12 08:00
A little over a year ago I wrote an article about automatic scripted installation of the SOA and BPM QuickStarts. One thing I wanted to improve is to be able to dynamically expand properties in the response file. I already found out how to do that under Linux but most QuickStart installations are done under Windows. So how to 'how to replace properties in textfile in Windows'?

I found this StackOverflow question, and favored the PowerShell option, so I modified that answer to be able to expand the ORACLE_HOME property in the response file.

I modified the response file that I got from the manual installation wizard of the BPM QuickStart as follows:
[ENGINE]

#DO NOT CHANGE THIS.
Response File Version=1.0.0.0.0

[GENERIC]

#Set this to true if you wish to skip software updates
DECLINE_AUTO_UPDATES=true

#My Oracle Support User Name
MOS_USERNAME=

#My Oracle Support Password
MOS_PASSWORD=<SECURE VALUE>

#If the Software updates are already downloaded and available on your local system, then specify the path to the directory where these patches are available and set SPECIFY_DOWNLOAD_LOCATION to true
AUTO_UPDATES_LOCATION=

#Proxy Server Name to connect to My Oracle Support
SOFTWARE_UPDATES_PROXY_SERVER=

#Proxy Server Port
SOFTWARE_UPDATES_PROXY_PORT=

#Proxy Server Username
SOFTWARE_UPDATES_PROXY_USER=

#Proxy Server Password
SOFTWARE_UPDATES_PROXY_PASSWORD=<SECURE VALUE>

#The oracle home location. This can be an existing Oracle Home or a new Oracle Home
ORACLE_HOME=${ORACLE_HOME}



I saved this as bpmqs1221_silentInstall.rsp.tpl. Then I created a simple command file called expandProperties.bat with the following content:
set ORACLE_HOME=c:\oracle\jdeveloper\12212_bpmqs
set QS_RSP=bpmqs1221_silentInstall.rsp
set QS_RSP_TPL=%QS_RSP%.tpl
powershell -Command "(Get-Content %QS_RSP_TPL%) -replace '\$\{ORACLE_HOME\}', '%ORACLE_HOME%' | Out-File -encoding ASCII %QS_RSP%"

This does the following:
  1. Set the ORACLE_HOME environment variable. This is what I also do in the QuickStart install script. The content of this variable should replace the '${ORACLE_HOME}' property in the response file template.
  2. Set QS_RSP to the Response File name
  3. Set QS_RSP_TPL to the Response File Template name
  4.  Call Powershell with a command line command using the '-Command' argument
    1. Read the template file denoted with %QS_RSP_TPL%' using the Get-Content argument.
    2. Replace the occurrences of the string  '${ORACLE_HOME}' with the value of the corresponsing environment variable '%ORACLE_HOME%'. But since the input is considered as a regular expression. Thus the special characters $, { and } need to be prefixed with a backslash.
    3. 'Pipe' the output to the output file denoted with %QS_RSP%. It's important to add -encoding ASCII for the encoding. Otherwise its apparently encoded in UTF in a way the Oracle Installer does not comprehend.
Run this as:
Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

c:\Data\Zarchief\Stage\FMW\bpm12cR2QS>expandProperties.bat

c:\Data\Zarchief\Stage\FMW\bpm12cR2QS>set ORACLE_HOME=c:\oracle\jdeveloper\12212_bpmqs

c:\Data\Zarchief\Stage\FMW\bpm12cR2QS>set QS_RSP=bpmqs1221_silentInstall.rsp

c:\Data\Zarchief\Stage\FMW\bpm12cR2QS>set QS_RSP_TPL=bpmqs1221_silentInstall.rsp.tpl

c:\Data\Zarchief\Stage\FMW\bpm12cR2QS>rem powershell -Command "(gc bpmqs1221_silentInstall.rsp.tpl) -replace '\$\{ORACLE_HOME\}', 'c:\oracle\jdeveloper\12212_bpmqs' | Out-File -encoding ASCII bpmqs1221_silentInstall.rsp"

c:\Data\Zarchief\Stage\FMW\bpm12cR2QS>powershell -Command "(Get-Content bpmqs1221_silentInstall.rsp.tpl) -replace '\$\{ORACLE_HOME\}', 'c:\oracle\jdeveloper\12212_bpmqs' | Out-File bpmqs1221_silentInstall.rsp"

c:\Data\Zarchief\Stage\FMW\bpm12cR2QS>

This creates a new file named bpmqs1221_silentInstall.rsp, where the content of the last line is changed to:
...

#The oracle home location. This can be an existing Oracle Home or a new Oracle Home
ORACLE_HOME=c:\oracle\jdeveloper\12212_bpmqs

Just what I need...

Recursion in XSLT

Thu, 2017-01-12 02:35
Yesterday I got involved in a question to handle a list of input documents where there is a start value and for every other element this start value has to be increased until an end value is reached.

In more advanced XSLT cases you may end up in a situations that aren't solvable using a for-each. Why not? Well, in the for-each you can't iteratively re-calculate a variable. Like a program language as Scala, XSLT is in fact a so called 'functional' language. Maybe not in the strict sense of the definition (I haven't checked), but one of the definitions is that in a functional language, or at least in Scala, variables are immutable. In XSLT once assigned a value, a variable cannot be changed. So a variable in XSLT is only variable at first assignment.

How to solve this? Well, with recursion. This morning I googled on XSLT and recursion and found this nice explanation.

Recursion is a paradigm in which a certain object or functionality is repeated by itself.

In computer programming this is done by defining a function with a specific implementation of functionality and where before of after that implementation the function calls itself with an altered set of input values until a certain end-condition is reached. It's one of the first computer algorithms I learned in a language as Pascal. And a typical example is calculating the factorial of a natural number.

I use recursion in XSLT regularly, for instance in a template that replaces a string in an another string until all occurrences have been replaced. I wrote about that in 2010 already.

In this blog post I'll omit providing an example, since I already provided a few in the articles linked above. I just want to draw your attention to this concept in XSLT. But if you like to read more on XSLT and recursion, you might be interested in this blog article.


Note to myself: when handling large payloads

Thu, 2016-12-01 07:36
Today I stumbled on a question in the communities about handling large payloads in BPEL/XSLT. Although I know that SOASuite from 11g onwards can do paging of XML to disk, I never had the need. However, you could need it from time to time. And it's good to know how to do it.

It's noted on My Oracle Support with Doc ID 1327970.1. Which refers to the 11g documentation on Managing Large Documents and Large Numbers of Instances.

Learning all the time....


XMind 8 is published

Wed, 2016-11-02 06:16
Already several years I´m a user of XMind. It is a very rich free MindMapping tool. I find the free version very usefull.

Today I found out that the new XMind 8 is published. You can get it on xmind.net.

Enjoy.

OSB Thread handling recommendations

Mon, 2016-10-31 02:29
I have got questions on performance of OSB quite a few times already, during the years. A few years ago on a project I got eyes on a set of recommendations on workmanagers for OSB. Many developers know that for instance Service Call outs are blocking activities. And that you should use workmanagers to solve performance problems resulting from the use of those blocking activities.

If you do nothing on dispatch policies in OSB proxy or business services, all is done in the Default Workmanager. But since some constructions, not only service call-outs, need other threads to finish the job, you can get stuck threads because the workmanager's threadpool gets empty having all or near to all threads waiting, leaving no threads to pickup work to free the others.

More on this in the terrific blog of Anthony Reynolds on the subject: Following the thread.

By the way,  I've seen that some people by default use service call-outs for almost everything. But the default use should be the routing node with a route activity. Even in some services you need to gather information from several sources, while you can only have one route node, pick or choose a 'driving-service' to use in the Routing node. Just like creating a query on several tables where you have to choose a 'driving-table'. Then  use service call outs only to do the extra enrichment.

From that earlier project I got the following recommendations, based on the blog of Anthony Reynolds. Since I refer back to it regularly, I think it would be good to share it.

For OSB to work optimally and prevent floading WebLogic’s threadpool with hogged/stuck threads you should create 3 FairShareRequest classes in ratio of 33/33/33, to distinguish different “kinds of threadpools”.

Then create 4 workmanagers:

  • FTPPollingWorkManager: file based inbound OSB proxy services. Polling a filesystem (or FTP). Uses FairShareReqClass-1, and ignores stuck threads.
  • InboudWorkManager: inbound OSB proxy services, not polling file based. Also uses FairShareReqClass-1, not ignoring stuck threads.
  • CallOutWorkManager: Service Call Out operations in a OSB proxy. Uses FairShareReqClass-2.
  • DeliveryWorkManager: outbound business services in OSB. Uses FairShareReqClass-3.
Use these as a dispatch policy in the particular Proxy/Business Services. In OSB 11g this is:
I don't have screendumps of 12c at hand. But the idea would be the same there. I haven't learned that the thread model in 12c is architecturally different.

Get the hostname of the executing server in BPEL

Wed, 2016-10-19 04:48
This week I got involved in a question on the Oracle Forums on getting the hostname of the server executing the bpel process. In itself this is not possible in BPEL. Also if you have a long running async process, the process gets dehydrated at several points (at a receive, wait, etc.). After an incoming signal, another server could process it further. You can't be sure that one server will process it to the end.

However, using Java, you can get the hostname of an executing server, quite easily. @AnatoliAtanasov suggested this question on stackOverflow. I thought that it would be fun to try this out.

Although you can opt for creating an embedded java activity, I used my earlier article on SOA and Spring Contexts to have it in a separate bean. By the way, in contrast to my suggestions in the article, you don't have to create a separate spring context for every bean you use.

My java bean looks like:
package nl.darwinit.soasuite;
import java.net.InetAddress;
import java.net.UnknownHostException;


public class ServerHostBeanImpl implements IServerHostBean {
public ServerHostBeanImpl() {
super();
}

public String getHostName(String hostNameDefault){
String hostName;
try
{
InetAddress addr;
addr = InetAddress.getLocalHost();
hostName = addr.getHostName();
}
catch (UnknownHostException ex)
{
System.out.println("Hostname can not be resolved");
hostName = hostNameDefault;
}
return hostName;
}

}

The interface class I generated is:
package nl.darwinit.soasuite;

public interface IServerHostBean {
String getHostName(String hostNameDefault);
}

Then I defined a Spring Context, getHostNameContext, with the following content
<?xml version="1.0" encoding="UTF-8" ?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:util="http://www.springframework.org/schema/util"
xmlns:jee="http://www.springframework.org/schema/jee" xmlns:lang="http://www.springframework.org/schema/lang"
xmlns:aop="http://www.springframework.org/schema/aop" xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:sca="http://xmlns.oracle.com/weblogic/weblogic-sca" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/tool http://www.springframework.org/schema/tool/spring-tool.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop.xsd http://www.springframework.org/schema/cache http://www.springframework.org/schema/cache/spring-cache.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task.xsd http://www.springframework.org/schema/jee http://www.springframework.org/schema/jee/spring-jee.xsd http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-jdbc.xsd http://www.springframework.org/schema/jms http://www.springframework.org/schema/jms/spring-jms.xsd http://www.springframework.org/schema/oxm http://www.springframework.org/schema/oxm/spring-oxm.xsd http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc.xsd http://xmlns.oracle.com/weblogic/weblogic-sca META-INF/weblogic-sca.xsd">
<!--Spring Bean definitions go here-->
<sca:service name="GetHostService" target="ServerHostBeanImpl" type="nl.darwinit.soasuite.IServerHostBean"/>
<bean id="ServerHostBeanImpl" class="nl.darwinit.soasuite.ServerHostBeanImpl"/>
</beans>

After wiring the context to my BPEL the composite looks like:


Then, deploying and running it, gives the following output:


Nice, isn't it?

Weblogic 11g to 12c: strictness in listen address

Tue, 2016-10-18 07:50
Let's say you have a virtual machine with two network adapters, both set on 'HostOnly'.
I used to do that and set the first one of those to a fixed IP address, say 10.0.0.1. To this one I coupled the hostname, for instance darlin-vce-db, using the /etc/hosts file. That way I had a fixed, always existing network address for the database.

Together with the database, you install WebLogic, for instance to serve SOASuite, or OSB, or whatever custom application you want to serve. Now, wouldn't it be nice to be able to use WebLogic from a browser out of the virtual machine? Of course, because this is what you do nowadays: almost everywhere I come, servers are hosted on virtualized computing environments, like VMWare VSE or Oracle VM. So that's where the second adapter comes in, dynamically coupled to an address in the form of 192.168.56.101, for instance. Externally, using the etc/host file on your host OS (in my case Windows), you couple that address also to darlin-vce-db.

So you have two /etc/hosts settings for the hostname, darlin-vce-db:
In internally in the VM:
10.0.0.1        darlin-vce-db     darlin-vce-db.darwin-it.local
And externally on your host OS:
192.168.56.101  darlin-vce-db     darlin-vce-db.darwin-it.local

Nothing special, right? Well, WebLogic 11g, apparently just listens to the hostname 'darlin-vce-db', if that is entered as a listen-address. It seems not to care if the request for 'darlin-vce-db' comes from 192.168.56.101 in stead of the 10.0.0.1 to where the hostname actually is bound.

However, in this particular case WebLogic 12c seems to behave differently. If you provice 'darlin-vce-db' as listen address, that is bound to a network adapter that has 10.0.0.1 it expects that requests also come in via that network adapter. It seems to ignore requests that come in via other adapters (in my case 192.168.56.101).

You can solve this partially using Channels: in the Weblogic Console, navigate to the particular managed server, click Protocols, Channels.
Create a new channel:

Give it a name like 'Extern-Intern' or something else properly denoting the purpose of it and choose 'http' as a protocol:

Then Provide the internal address, for instance 'darlin-vce-db', and the external listen address:

Leave the ports to the default listen-port, in this case. Then finish the wizard.
Although this helps in connecting to the WebLogic console, EM, or with the same method on the SOA Server, to the SOA Composer (soaserver:port/soa/composer), BPM Workspace (soaserver:port/bpm/composer) etc., this will not work for JDeveloper.

When trying to deploy a soa composite from JDeveloper, you define/choose a ApplicationServer with connection to the AdminServer. But in case of deploying a composite, the AdminServer figures out which running SOA Servers there are, and let JDeveloper provide the composite to those servers. But then the soaserver(s) refuse(s) the connections from JDeveloper. Testing the ApplicationServer connection will show success for the Http connection to the AdminServer, but fails for all the other components.

The solution is then to denote a particular Network Adapter/ip address and make sure that internally and externally the particular hostname is coupled to that same particular ip-address.

OTN Appreciation Day: - the day after - BPEL, SOASuite and SCA (in that order...)

Wed, 2016-10-12 07:21
Unfortunately I noticed this nice initiative only yesterday: OTN Appreciation Day. I did not had a change to cook something up, but I do like to add some mustard after the meal, as we say in Dutch.

In the titles of the 'OTN Appreciation Day'-blogs I miss BPEL. In 2004,when Oracle acquired Collaxa, I worked at Oracle Consulting in the Netherlands. I worked with Oracle Workflow and Interconnect. Oracle wasn't yet into SOA really. But with BPEL PM they acquired the tool that stood at the base of SOA Suite, together with Webservices Manager and what we now know as the Mediator. And it changed my professional life, really. Its extremely powerful, especially with the added  JCA Adapters, xslt-mapper, and since 11g the SCA architecture wich enables you to assemble composite applications with Adapters BusinessRules, Human Workflow, Mediator, Spring Components and of course ... BPEL 2.0.

Sorry, Tim, for being late.

SQL Developer: Live and Let Live my db-connection

Wed, 2016-10-12 03:59
At my current customer I have SQLDeveloper open the whole day, and regularly I come back to it to query my throttle-table to see if my requests have been picked up. But regularly my database connection have been broken because of being idle. Probably because of a nasty firewall between my remote development desktop and the database.

Googling on it I found an article of That Jeff Smith on busy connections. That blog is really one to follow. Feed it to your Feedly if your a recent SQL Developer user.

But in my search I found the 'keepAlive-4' extension for SQL Developer 4. Download the KeepAlive zip, go to Help->Check for Updates in SQLDeveloper and install it via the 'Install From Local File' option. Then look for the Keep Alive icon in the toolbar:
Click on it and enter a check frequency, of at least 60 seconds. I try 180 seconds.

You can disable it by clicking it again. Another click and it asks for a new interval again.

Keep your Service Directory valid

Wed, 2016-10-05 07:49
To day I ran into a trap that trapped me before...

I tried to edit a file adapter service in SOA/BPM QuickStart 12.2.1. And got the error 'Service Directory is not valid' in a pop-up dialog, that prevents me from editing.

The problem is described in this forum thread.

The problem is a space in the path to the JDeveloper project. Now spaces are a drag in filenames and paths (I hate it that Microsoft Windows introducced "Program Files" as the default folder for installing applications).

The solution is in Windows quite simple actually: create a substitute drive like:
echo define drive S: referring to current folder
subst s: /d
subst s: "d:\Data\svn\Applicatie Integration"

Put this in a .bat file and run it before starting JDeveloper and load the application workspaces from the S: drive.

I had this already in place because of my first run into this trap. Why I tripped again? It turned out I accidentally had opened the application from the d: drive again...

Use WLST to test all your datasources

Tue, 2016-10-04 07:10
To day I had some problems with deploying my composite to the server of my current customer. Apparently the Server had some problems with datasources. But since there are many I did not feel much for checking them one by one with the console. Using Google I got the following examples

Thus I had to combine those two, sauce it with my own way of wlst-scripting (see my other blog entries). Also I want a tabular form, that got me into troubles with printing the result of the testPool() method. But I came up with the following script, testDS.py:
#############################################################################
# Test Datasources on a domain
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 2.1, 2016-10-04
#
#############################################################################
# Modify these values as necessary
import sys, traceback
scriptName = sys.argv[0]
#
#
lineSeperator='__________________________________________________________________________________'
#
#
def usage():
print 'Call script as: '
print 'Windows: wlst.cmd '+scriptName+' -loadProperties localhost.properties'
print 'Linux: wlst.sh '+scriptName+' -loadProperties environment.properties'
print 'Property file should contain the following properties: '
print "adminUrl=hhs-sbm3015:7001"
print "adminUser=weblogic"
print "adminPwd=welcome1"
#
#
def connectToadminServer(adminUrl, adminServerName):
print(lineSeperator)
print('Try to connect to the AdminServer')
try:
connect(userConfigFile=usrCfgFile, userKeyFile=usrKeyFile, url=adminUrl)
except NameError, e:
print('Apparently user config properties usrCfgFile and usrKeyFile not set.')
print('Try to connect to the AdminServer adminUser and adminPwd properties')
connect(adminUser, adminPwd, adminUrl)
#
#
def main():
try:
pad=' '
print(lineSeperator)
print('Check datasources for domain')
print(lineSeperator)
print ('Connect to the AdminServer: '+adminServerName)
connectToadminServer(adminUrl, adminServerName)
print(lineSeperator)
allServers=domainRuntimeService.getServerRuntimes();
if (len(allServers) > 0):
for tempServer in allServers:
jdbcServiceRT = tempServer.getJDBCServiceRuntime();
dataSources = jdbcServiceRT.getJDBCDataSourceRuntimeMBeans();
print('\nServer '+tempServer.getName())
if (len(dataSources) > 0):
print('Datasource '[:30]+'\tState\tTest')
for dataSource in dataSources:
testPool = dataSource.testPool()
dataSourceName = dataSource.getName()+pad
dataSourceNamePad=dataSourceName[:30]
if (testPool == None):
print dataSourceNamePad+'\t'+dataSource.getState()+'\tOK'
else:
print dataSourceNamePad+'\t'+dataSource.getState()+'\tFailure: '
print testPool
print(lineSeperator)
print('Done...')
print(lineSeperator)
except NameError, e:
print('Apparently properties not set.')
print "Please check the property: ", sys.exc_info()[0], sys.exc_info()[1]
usage()
except:
apply(traceback.print_exception, sys.exc_info())
exit(exitcode=1)
#
main();

Of course it's easy to extent the table with properties from the monitor script in WebLogic DataSource Monitoring Using WLST.

Which runs pretty neat. Run it with a shell script like the following testDS.sh script:
#!/bin/bash
#############################################################################
# Test DataSources using wlst.
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 2.1, 2016-06-27
#
#############################################################################
#
. fmw12c_env.sh
echo
echo Test Datasources
wlst.sh ./testDS.py -loadProperties fmw.properties

For the fmw12c_env.sh and fmw.properties files look here.

Replacement of environment variables or properties in Bash

Sun, 2016-09-25 10:47
Earlier I wrote about the automatic installation of Fusion Middleware components using response files. A thing that lacked in my scripts was that although I had a FMW_HOME variable set in my enviroment shell script, the response files had the location hard coded in them. At the time I hadn't had the chance to figure out how to do property/variable replacement in shell. I do know how to do it with ANT. But I figured that installing ANT for only this was a bit too much, since with the installation of FMW you already get ANT as a module.

For an upgrade of my scripts to FMW 12.2.1.1, I did a Google-search on it and found: http://stackoverflow.com/questions/415677/how-to-replace-placeholders-in-a-text-file. The top 2 suggestions were:

  1. sed -e "s/\${i}/1/" -e "s/\${word}/dog/" template.txt
  2. i=32 word=foo envsubst < template.txt
Although the first option was favoured by many and considered the answer on the querstion, I personally favour the second. It turns out that sed does not accept references to the environment variables as a replacement. And that makes the replacements hardcoded again. The second does accept environment variable references. Actually, if the variable-reference in the template file  is already present in the environment, no actual replacement assignment have to be provided.

So let's say my response file template looks like:

[ENGINE]

#DO NOT CHANGE THIS.
Response File Version=1.0.0.0.0

[GENERIC]

#Set this to true if you wish to skip software updates
DECLINE_AUTO_UPDATES=true

#
MOS_USERNAME=

#
MOS_PASSWORD=<SECURE VALUE>

#If the Software updates are already downloaded and available on your local system, then specify the path to the directory where these patches are available and set SPECIFY_DOWNLOAD_LOCATION to true
AUTO_UPDATES_LOCATION=

#
SOFTWARE_UPDATES_PROXY_SERVER=

#
SOFTWARE_UPDATES_PROXY_PORT=

#
SOFTWARE_UPDATES_PROXY_USER=

#
SOFTWARE_UPDATES_PROXY_PASSWORD=<SECURE VALUE>

#The oracle home location. This can be an existing Oracle Home or a new Oracle Home
ORACLE_HOME=${FMW_HOME}

#Set this variable value to the Installation Type selected. e.g. Fusion Middleware Infrastructure, Fusion Middleware Infrastructure With Examples.
INSTALL_TYPE=Fusion Middleware Infrastructure

#Provide the My Oracle Support Username. If you wish to ignore Oracle Configuration Manager configuration provide empty string for user name.
MYORACLESUPPORT_USERNAME=

#Provide the My Oracle Support Password
MYORACLESUPPORT_PASSWORD=<SECURE VALUE>

#Set this to true if you wish to decline the security updates. Setting this to true and providing empty string for My Oracle Support username will ignore the Oracle Configuration Manager configuration
DECLINE_SECURITY_UPDATES=true

#Set this to true if My Oracle Support Password is specified
SECURITY_UPDATES_VIA_MYORACLESUPPORT=false

#Provide the Proxy Host
PROXY_HOST=

#Provide the Proxy Port
PROXY_PORT=

#Provide the Proxy Username
PROXY_USER=

#Provide the Proxy Password
PROXY_PWD=<SECURE VALUE>

#Type String (URL format) Indicates the OCM Repeater URL which should be of the format [scheme[Http/Https]]://[repeater host]:[repeater port]
COLLECTOR_SUPPORTHUB_URL=



Saved as 'fmw_12.2.1.1.0_infrastructure.rsp.tpl'; note the reference ORACLE_HOME=${FMW_HOME}. And I have set FMW_HOME with an fmw12c_env.sh script, as described in former posts. Then I only have to do:
envsubst < fmw_12.2.1.1.0_infrastructure.rsp.tpl >>fmw_12.2.1.1.0_infrastructure.rsp
To have the file copied to fmw_12.2.1.1.0_infrastructure.rsp with a replaced FMW_HOME variable:

...
#
SOFTWARE_UPDATES_PROXY_PASSWORD=<SECURE VALUE>

#The oracle home location. This can be an existing Oracle Home or a new Oracle Home
ORACLE_HOME=/u01/app/oracle/FMW12211

#Set this variable value to the Installation Type selected. e.g. Fusion Middleware Infrastructure, Fusion Middleware Infrastructure With Examples.
INSTALL_TYPE=Fusion Middleware Infrastructure
...

Couldn't be more simple, I'd say. Nice thing is that this enables me to do more directives. So, learned something again, from a question dated 7,5 years ago...

How to solve JSchException Algorithm negotiation fail. And get logging out of JSch in SoapUI.

Wed, 2016-09-07 05:43
I was so glad with my SoapUI solution to SFTP files to a server. But so dissapointed I couldn't have it working at my customer.

After I changed the log entries to log with e.message, I got the line:
Wed Sep 07 11:17:43 CEST 2016:INFO:JSchException Algorithm negotiation fail

Now I needed more information than that. But the hint is at least that there is a mismatch in the available cyphers client side verses server side.

So first I wanted to get more logging out of Jsch. It turns out that it has it's own Logger framework, but the bright side of that is that you can easily wrap your own logging mechanism. In the case of SoapUI it is log4j. So create a java project with the libraries jsch-0.1.54.jar and from the SoapUI libs: log4j-1.2.14.jar. Then I created the following class file from an example from the answer in this stackoverflow question.

My version is:
package nl.darwin.jsch.log;


import com.jcraft.jsch.Logger;

/**
* Class to route log messages generated by JSch to Apache Log4J Logging.
*
* @author mrabbitt
* @see com.jcraft.jsch.Logger
*/
public class JSchLog4JLogger implements Logger {
private org.apache.log4j.Logger logger;

/**
* Constructor with custom category name
*
* @param logger the logger from Apache Log4J.
*/
public JSchLog4JLogger(org.apache.log4j.Logger logger) {
this.logger = logger;
}

/**
* Default constructor
*/
public JSchLog4JLogger() {
this(org.apache.log4j.Logger.getLogger(Logger.class.getName()));
}

/* (non-Javadoc)
* @see com.jcraft.jsch.Logger#isEnabled(int)
*/
public boolean isEnabled(int level) {
switch (level) {
case DEBUG:
return logger.isDebugEnabled();
case INFO:
return logger.isInfoEnabled();
case WARN:
return logger.isInfoEnabled();
case ERROR:
return logger.isInfoEnabled();
case FATAL:
return logger.isInfoEnabled();
}
return false;
}

/* (non-Javadoc)
* @see com.jcraft.jsch.Logger#log(int, java.lang.String)
*/
public void log(int level, String message) {
switch (level) {
case DEBUG:
logger.debug(message);
break;
case INFO:
logger.info(message);
break;
case WARN:
logger.warn(message);
break;
case ERROR:
logger.error(message);
break;
case FATAL:
logger.fatal(message);
break;
}
}
}

Then Jar it and add it to the bin/ext older of SoapUI (like in the previous blog post).
Now a simple extra line is needed and an import in your groovy script :
import nl.darwin.jsch.log.JSchLog4JLogger
...
JSch.setLogger(new JSchLog4JLogger(log))
JSch ssh = new JSch()

So simply set the logger on the JSch class, before the instantiation. Then the logging of JSch appears in the SoapUI logging, as easy as that.
It turned out that the server required the use of aes256-ctr, while the jre of SoapUI (which is Java 7 in SoapUI 5.2.1) has limited JCE policy. As is suggested here.

You can download the unlimited JCE policies here:
JDK
Unlimited JCE Download
JDK 1.6http://www.oracle.com/technetwork/java/javase/downloads/jce-6-download-429243.htmlJDK 1.7http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.htmlJDK 1.8http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
For SoapUI, download the JDK 1.7 policy. Go to your SoapUI Home folder, and navigate to the security library folder within the JRE. For instance: c:\Program Files\SmartBear\SoapUI-5.2.1\jre\lib\security.

Unzip the JCE to a new folder UnlimitedJCEPolicy within the security folder. Create a another backup folder like LimitedJCEPolicy and copy the jars US_export_policy.jar and local_policy.jar to the LimitedJCEPolicy folder. And copy the corresponding files from UnlimitedJCEPolicy to the security folder, replacing the original ones.

Restart SoapUI and you're good to go.


Use SoapUI to test your SFTP based services

Wed, 2016-09-07 04:13
SoapUI is my favorite tool to do unit tests. I'd try to keep my self to test based development and build up tests along with the development service. For SOAP or REST based services this goes quite intuitively using SoapUI. For database driven it is a little harder, but SoapUI has a nice JDBC activity, that supports DML as well as callable statements as stored procedures.

But for files and especially SFTP its a little more complicated. For a while I'm working on a filebased integration with SAP as source system.

I configured and defined the SOASuite FTP adapter to use my SSH user (oracle) on my VirtualBox VM. Until now I tested it using the SFTP/SCP client from MobaXTerm (enthousiastically recommended: download here). But not so handy for unit tests.

I wanted to automate this using SoapUI. With some searching I found that JCraft Java Secure Channel library was the best and easiest option. I did take a look at Apache Commons Net. But couldn't get it to work so easily. Download the jsch-0.1.54.jar (or newer) file and copy it to the ./bin/ext folder in your SoapUI home:


And restart SoapUI.

Create a new empty SoapUI project, create a TestSuite called something like 'Utilities' and a TestCase called 'TC-FTP'. Add the following properties to the TestCase:

Property
Value
ftpHostdarlin-vce-dbftpPort22ftpUsernameoracleftpPasswordwelcome1localFilePathd:/Projects/2016MySapProject/ExampleFiles/SAP-MESSAGE.XMLremoteFileDir/home/oracle/SapHR/in
In the TestCase add a Groovy Script called FTP add the script below. I took the example from snip2code.com (also found elsewhere) and refactered it to:
//Send Files to SSH Location
//
//Download jsch-0.1.54.jar from http://www.jcraft.com/jsch/ and copy it to SoapUI-Home/bin/ext location
//Example from: https://www.snip2code.com/Snippet/413499/SoapUI-Groovy-Script-compatible-SFTP-fil

//import org.apache.commons.net.ftp.FTPSClient
import com.jcraft.jsch.*
//
// Properties
//
def testCase = testRunner.testCase;


def String ftpHost = testCase.getPropertyValue("ftpHost")
def int ftpPort = testCase.getPropertyValue("ftpPort").toInteger()
def String ftpUsername = testCase.getPropertyValue("ftpUsername")
def String ftpPassword = testCase.getPropertyValue("ftpPassword")
def String localFilePath = testCase.getPropertyValue("localFilePath")
def String remoteFileDir = testCase.getPropertyValue("remoteFileDir")
//
//
Session session = null
Channel channel = null
try {
log.info("Starting sftp upload process")
JSch ssh = new JSch()

session = ssh.getSession(ftpUsername, ftpHost, ftpPort)
session.setConfig("StrictHostKeyChecking", "no"); //auto accept secure host
session.setPassword(ftpPassword)
session.connect()
log.info("Connected to session")

channel = session.openChannel("sftp")
channel.connect()
log.info("Connected to channel")

ChannelSftp sftp = (ChannelSftp) channel;
sftp.put(localFilePath, remoteFileDir);
log.info("File Uploaded " + localFilePath + " TO " + remoteFileDir)

} catch (JSchException e) {
e.printStackTrace()
log.info("JSchException " + e.message)
} catch (SftpException e) {
e.printStackTrace()
log.info("SftpException " + e.message)
} finally {
if (channel != null) {
channel.disconnect()
log.info("Disconnected from channel")
}
if (session != null) {
session.disconnect()
log.info("Disconnected from session")
}
log.info("sftp upload process complete")
}

Changes I did was to define the input values based on the properties from the testcase. And move the session and channel variable declartions out of the try, to get it accessible from the finally branch. And to replace e.printStackTrace from the logging by e.message, to have a propery message (e.printStackTrace returns null) in the logging.

The reason that I suggest to have it in a separate test cases is to enable it to be called from actual testcases with parameters. To do so add to your test case a call-test case activity:

Set the following properties:

Choose 'Run primary TestCase (wait for running to finish, Thread-Safe)' option as Run Mode.

And provide the property values.

This script copies a file from a file location and uploads it. But I want to be able to insert some runtime specific options to refer to in asserts and later JDBC calls. To check on specific running instances. So I want to be able to adapt the content running in my test case. Actually I want to upload a string fetched from a property, maybe with expanded properties.

So I copied the testcase and groovy activity and adapted the script to:
//Send Files to SSH Location
//
//Download jsch-0.1.54.jar from http://www.jcraft.com/jsch/ and copy it to SoapUI-Home/bin/ext location
//Example from: https://www.snip2code.com/Snippet/413499/SoapUI-Groovy-Script-compatible-SFTP-fil

//import org.apache.commons.net.ftp.FTPSClient
import com.jcraft.jsch.*
import java.nio.charset.StandardCharsets
//
// Properties
//
def testCase = testRunner.testCase;
//
def String ftpHost = testCase.getPropertyValue("ftpHost")
def int ftpPort = testCase.getPropertyValue("ftpPort").toInteger()
def String ftpUsername = testCase.getPropertyValue("ftpUsername")
def String ftpPassword = testCase.getPropertyValue("ftpPassword")
def String fileContent = testCase.getPropertyValue("fileContent")
def String remoteFile = testCase.getPropertyValue("remoteFile")
//
Channel channel = null
Session session = null
try {
log.info("Starting sftp upload process")
JSch ssh = new JSch()

session = ssh.getSession(ftpUsername, ftpHost, ftpPort)
session.setConfig("StrictHostKeyChecking", "no"); //auto accept secure host
session.setPassword(ftpPassword)
session.connect()
log.info("Connected to session")

channel = session.openChannel("sftp")
channel.connect()
log.info("Connected to channel")

ChannelSftp sftp = (ChannelSftp) channel;

byte[] fileContentBytes = fileContent.getBytes(StandardCharsets.UTF_8)
InputStream fileInputStream = new ByteArrayInputStream(fileContentBytes);
log.info("Start uploaded to " + remoteFile)
sftp.put(fileInputStream, remoteFile);
log.info("File Content uploaded to " + remoteFile)

} catch (JSchException e) {
e.printStackTrace()
log.info("JSchException " + e.message)
} catch (SftpException e) {
e.printStackTrace()
log.info("SftpException " + e.message)
} catch (Exception e) {
e.printStackTrace()
log.info("Exception " + e.message)
} finally {
if (channel != null) {
channel.disconnect()
log.info("Disconnected from channel")
}
if (session != null) {
session.disconnect()
log.info("Disconnected from session")
}
log.info("sftp upload process complete")
}

here the lines and related properties:
def String localFilePath = testCase.getPropertyValue("localFilePath")
def String remoteFileDir = testCase.getPropertyValue("remoteFileDir")

are changed to:
def String fileContent = testCase.getPropertyValue("fileContent")
def String remoteFile = testCase.getPropertyValue("remoteFile")
Then the lines:
  byte[] fileContentBytes =   fileContent.getBytes(StandardCharsets.UTF_8)
InputStream fileInputStream = new ByteArrayInputStream(fileContentBytes);

convert the fileContent property value to an InputString. And that is given as an input to the statement sftp.put(fileInputStream, remoteFile);. Notice that since we upload file content, we need to provide a remoteFile path, including file name, insead of a remote directory. And that we need an extra import java.nio.charset.StandardCharsets.

It would be nice if the guys from SmartBear add both put and get as a seperate activity.  

.... this is only a psuedo object?

Fri, 2016-09-02 06:00
Yesterday I was working on a BPEL project that I created before the summer holidays. I wanted to implement it further. But on first redeployment I ran into:
[12:18:01 PM] ----  Deployment started.  ----
[12:18:01 PM] Target platform is (Weblogic 12.x).
[12:18:01 PM] Running dependency analysis...
[12:18:01 PM] Building...
[12:18:08 PM] Deploying profile...
[12:18:09 PM] Wrote Archive Module to D:\Projects\2016DWN\SOASuite\HRApplication\DWN_CdmHR\trunk\SOA\DWN_CdmHR\CDMHRDomainService\deploy\sca_CDMHRDomainService.jar
[12:18:18 PM] Deploying sca_CDMHRDomainService.jar to partition "default" on server SoaServer1 [http://darlin-vce-db.darwin-it.local:8001]
[12:18:18 PM] Processing sar=/D:/Projects/2016DWN/SOASuite/HRApplication/DWN_CdmHR/trunk/SOA/DWN_CdmHR/CDMHRDomainService/deploy/sca_CDMHRDomainService.jar
[12:18:18 PM] Adding sar file - D:\Projects\2016DWN\SOASuite\HRApplication\DWN_CdmHR\trunk\SOA\DWN_CdmHR\CDMHRDomainService\deploy\sca_CDMHRDomainService.jar
[12:18:18 PM] Preparing to send HTTP request for deployment
[12:18:18 PM] Creating HTTP connection to host:darlin-vce-db.darwin-it.local, port:8001
[12:18:18 PM] Sending internal deployment descriptor
[12:18:19 PM] Sending archive - sca_CDMHRDomainService.jar
[12:18:19 PM] Received HTTP response from the server, response code=500
[12:18:19 PM] Error deploying archive sca_CDMHRDomainService.jar to partition "default" on server SoaServer1 [http://darlin-vce-db.darwin-it.local:8001]
[12:18:19 PM] HTTP error code returned [500]
[12:18:19 PM] Error message from server:
There was an error deploying the composite on SoaServer1: Deployment Failed: Error occurred during deployment of component: HREmployeeProcess to service engine: implementation.bpel, for composite: CDMHRDomainService: ORABPEL-05215

Error while loading process.
The process domain is encountering the following errors while loading the process "HREmployeeProcess" (composite "default/CDMHRDomainService!1.0*soa_6e4206b5-3297-4f53-9944-734349aed8ab"): this is only a psuedo object.
This error contained an exception thrown by the underlying process loader module.
Check the exception trace in the log (with logging level set to debug mode). If there is a patch installed on the server, verify that the bpelcClasspath domain property includes the patch classes.
.

[12:18:19 PM] Check server log for more details.
[12:18:19 PM] Error deploying archive sca_CDMHRDomainService.jar to partition "default" on server SoaServer1 [http://darlin-vce-db.darwin-it.local:8001]
[12:18:19 PM] Deployment cancelled.
[12:18:19 PM] ---- Deployment incomplete ----.
[12:18:19 PM] Error deploying archive file:/D:/Projects/2016DWN/SOASuite/HRApplication/DWN_CdmHR/trunk/SOA/DWN_CdmHR/CDMHRDomainService/deploy/sca_CDMHRDomainService.jar
(oracle.tip.tools.ide.fabric.deploy.common.SOARemoteDeployer)

So the I was googling around, and found this blog entry. This one suggested a missmatch between the project and referenced wsdl's/xsd's in the MDS.

So I refreshed the MDS, restarted the whole SOA Server, but no luck.

At the doorstep of removing the lot of components and references, I decided to take a last closer look to the composite.xml.

The BPEL process component HREmployeeProcess had a reference to the service HREmployeeProcessSubscriber. The latter was based on a wsdl in the mds:
  <reference name="HREmployeeProcessSubscriber"
ui:wsdlLocation="oramds:/apps/CDM/services/domain/operations/hrm/v2/EmployeeDomainEntityEventService.wsdl">
<interface.wsdl interface="http://hhs.nl/services/domain/operations/hrm/v2/#wsdl.interface(EmployeeDomainEntityEventServicePortType)"/>
<binding.ws port="http://hhs.nl/services/domain/operations/hrm/v2/#wsdl.endpoint(hremployeeprocessa_client/EmployeeDomainEntityEventServicePort)"
location="http://darlin-vce-db:8001/soa-infra/services/default/HRSubscriberA/HREmployeeEventServiceA?WSDL"
soapVersion="1.1"/>
</reference>
But in the reference in the bpel component it refered to the BPEL process on the server:
<reference name="HREmployeeProcessSubscriber"
ui:wsdlLocation="http://darlin-vce-db:8001/soa-infra/services/default/HRSubscriberA/HREmployeeEventServiceA?WSDL">
<interface.wsdl interface="http://hhs.nl/services/domain/operations/hrm/v2/#wsdl.interface(EmployeeDomainEntityEventServicePortType)"/>
</reference>

Since the wsdl defined in the ui:wsdlLocation attribute needs to be available on compiling and loading of the component by the component-engine it's recommended to use a reference to an abstract wsdl in the mds. In this case I replaced the ui:wsdlLocation in the service reference by the mds. But apparently I forgot the BPEL Comnponent. To replace that, you should do this in the partnerlink definition in the BPEL Process. Because the composite.xml is automatically updated. Because the abastract wsdl lacks the partnerlink types, as you might know, JDeveloper suggests to create a wrapper wsdl for you.

Now, because of the synchronizations between bpel and the composite, you might need to hack the composite and the bpel process, to get thinks consistent again (at least I had to). But then, having it resolved, the composite was deployable again... And the BPEL process wasn't so pseudo anymore.

The third one on Creating weblogic user, now for SOA Suite

Thu, 2016-09-01 08:43
A few months ago I figured out how to create specific users with restricted access to Service Bus components. I blogged about it in part 2 of creating WebLogic users for ServiceBus 12c. But the series lacks an explanation on restricted user access on SOASuite.

Today in a question about Roles on Oracle Community Forums, the reference to this elaborate blog entry was given: Restricted View, by Antony Reynolds.

I think that blog explains it well. Unfortunately the link to 7.2 Partition Roles, Anthony mentioned, did not work. What I found (12.1.3) is 7.3 Securing Access to Partitions (12.1.3) and 7.3 Securing Access to SOA Folders (12.2.1). (Apparently from 12.2.1 onwards, partitions are called SOA folders...)




Mount NTFS disk in Linux 7

Fri, 2016-08-26 06:03
Today I wanted to pass an old disk in a usb-case to my son. It was from an old Windows Laptop and even though I'm administrator, I wasn't able to read the documents in an other user's folder.

So I thought, let's do it from an Oracle Linux 7 VM, as root. But it turns out that Oracle linux did not support NTFS by default.

But with the trick in this link I managed to do it.

To sum up, especially for my self:

Add the EPEL-7 repository from Fedora:
# wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

# rpm -ivh epel-release-latest-7.noarch.rpm

This worked for me. Although I had to step over my issue that I add a Fedora repo to Oracle Linux.... An other option could have been (probably if under Redhat Linux or CentOS:
# yum install epel-release
...
# yum clean all
...
# yum update
...

Install the NTFS-3g package:
# yum install ntfs-3g -y

And enable NTFS support for FileManagers:
# yum install ntfsprogs -y

Then I got to the FileManager and I could browse the disk. It was mounted at '/run/media/oracle/803638DB3638D3BE'.

Generate Admin Channels to improve Weblogic Admin Console performance (and of FMW-Control)

Wed, 2016-08-24 11:02
At one of my customers we have quite an impressive domain configuration. It's a FMW domain with SOA, OSB, BAM, WSM, MFT in clusters of 4 nodes. The thing is that when having started all the servers, the console becomes slooooooowwwww. Not to speak of FMW Control (em).

One suggestion is to set the 'Invocation Timeout Seconds' under MyDomain->Configuration->General->Advanced to a value like 2. And 'Management Operation Timeout' under Preferences->Shared Preferences to a value like 5:

This surely makes the console responsive again. But it actually means that the console gives up right away when querying for the (health) state of the servers. So in stead of a health of 'OK', you get a 'server unreachable' message.

When having a lot of servers in the domain, they all share the same Admin Channel, and this seams to get over flooded. AdminServer does not get the responses in time. Sometimes a new request leads to a proper response, but in fact it takes a lot of time.

To reduce the load on the default channel, we created a admin server per managed server. Since it's a lot of servers, and we need to do it on several environments, so I created a wlst-script for it.
The script createAdminChannels.py:
#############################################################################
# Create AdminChannels for WebLogic Servers
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.1, 2016-08-24
#
#############################################################################
# Modify these values as necessary
import sys, traceback
scriptName = sys.argv[0]
#
#
lineSeperator='__________________________________________________________________________________'
#
#
def usage():
print 'Call script as: '
print 'Windows: wlst.cmd '+scriptName+' -loadProperties localhost.properties'
print 'Linux: wlst.sh '+scriptName+' -loadProperties environment.properties'
print 'Property file should contain the following properties: '
print "adminUrl=localhost:7001"
print "adminUser=weblogic"
print "adminPwd=welcome1"
#
#
def connectToadminServer(adminUrl, adminServerName):
try:
print(lineSeperator)
print('Try to connect to the AdminServer')
try:
connect(userConfigFile=usrCfgFile, userKeyFile=usrKeyFile, url=adminUrl)
except NameError, e:
print('Apparently user config properties usrCfgFile and usrKeyFile not set.')
print('Try to connect to the AdminServer adminUser and adminPwd properties')
connect(adminUser, adminPwd, adminUrl)
except WLSTException:
message='Apparently AdminServer not Started!'
print (message)
raise Exception(message)
#
#
def createAdminChannel(serverName, adminListenPort):
print(lineSeperator)
channelName = serverName+'-AdminChannel'
try:
cd('/Servers/'+serverName+'/NetworkAccessPoints/'+channelName)
print('Channel '+channelName +' for '+serverName+' already exists.')
except WLSTException:
try:
print('Create Admin Channel for server: '+serverName+', with port: '+adminListenPort)
cd('/Servers/'+serverName)
cmo.createNetworkAccessPoint(channelName)
cd('/Servers/'+serverName+'/NetworkAccessPoints/'+channelName)
cmo.setProtocol('admin')
cmo.setListenPort(int(adminListenPort))
cmo.setEnabled(true)
cmo.setHttpEnabledForThisProtocol(true)
cmo.setTunnelingEnabled(false)
cmo.setOutboundEnabled(false)
cmo.setTwoWaySSLEnabled(false)
cmo.setClientCertificateEnforced(false)
print ('Succesfully created channel: '+channelName)
except WLSTException:
apply(traceback.print_exception, sys.exc_info())
message='Failed to create channel '+channelName+'!'
print (message)
raise Exception(message)
#
#
def main():
try:
print (lineSeperator)
print ('Start Osb Cluster')
print (lineSeperator)
print('\nConnect to AdminServer ')
connectToadminServer(adminUrl, adminServerName)
print(lineSeperator)
print('Start Edit Session')
edit()
startEdit()
#
#Create Admin Channels
# Administrators
print('\nCreate Admin Channels')
serverNameList=serverNames.split(',')
serverAdminPortList=serverAdminPorts.split(',')
#
idx=0
for serverName in serverNameList:
adminPort=serverAdminPortList[idx]
createAdminChannel(serverName, adminPort)
idx=idx+1
# Activate changes
print(lineSeperator)
print('Activate Changes')
save()
activate(block='true')
#
print('\nExiting...')
exit()
except NameError, e:
print('Apparently properties not set.')
print "Please check the property: ", sys.exc_info()[0], sys.exc_info()[1]
usage()
except:
apply(traceback.print_exception, sys.exc_info())
stopEdit('y')
exit(exitcode=1)
#call main()
main()
exit()


The shell script to call it, createAdminChannels.sh:
#!/bin/bash
#############################################################################
# Create AdminChannels
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 2.1, 2016-08-24
#
#############################################################################
#
. fmw12c_env.sh
export PROPERTY_FILE=$1
echo
echo Create Admin Channels
wlst.sh ./createAdminChannels.py -loadProperties $PROPERTY_FILE

And the example property file, darlin-vce-db.properties:
#############################################################################
# Properties voor Creeëren SOADomain
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.0, 2016-04-15
#
#############################################################################
#
# Properties for AdminServer
adminServerName=Adminserver
adminUrl=darlin-vce-db:7001
# AdminUser
adminUser=weblogic
adminPwd=welcome1
#
serverNames=AdminServer,OsbServer1,SoaServer1
serverAdminPorts=7101,8111,8101
#

Call the script as $> createAdminChannels.sh darlin-vce-db.properties

In the property file you'll need to name every server in the property serverNames. And for each server the particular Admin Listen Port in serverAdminPorts, in the exact same order. Start with the AdminServer.


At the end of the script the changes are activated and then the AdminServer listens over https on the changed port.

Important: the servers need to be down, except for the the AdminServer.

Unfortunately the infrastructure database was apparently down. So I haven't been able to start SOA, BAM, etc. to see if it is performant now. But I have high hopes...

Pages