Weblogic Remote Deploy from Red Hat Enterprise Linux Server: Unknown object in LOCATE_REQUEST

Recently I was attempting to deploy to weblogic from a Jenkins installed on a Red Hat Enterprise Linux Server release 7.3, to a remote Weblogic 12.1.3 cluster. Which was failing with a org.omg.CORBA.OBJECT_NOT_EXIST. Eventually, I ended up trying to do a manual deploy using the ANT task wldeploy, with the following command

 ant -lib /apps/wls12130/wlserver/server/lib deploy -Dweblogic.user=adminuser -Dweblogic.password=adminpassword -Dadminurl=t3://admin-server:admin-port -Dweblogic.cluster=ClusterName

As you can see from the command, we are passing command-line arguments which specify the location and credentials for the remote cluster. On the Linux machine to with the [wldeploy] Caused by: javax.naming.NamingException: Couldn't connect to the specified host [Root exception is org.omg.CORBA.OBJECT_NOT_EXIST: Unknown object in LOCATE_REQUEST vmcid: 0x0 minor code: 0 completed: No] error. Following is the full stacktrace, followed by a cause and resolution to this problem...

 [wldeploy] Exception in thread "main" weblogic.Deployer$DeployerException: weblogic.deploy.api.tools.deployer.DeployerException: Unable to connect to 't3://adminserver:adminport': Unknown object in LOCATE_REQUEST. Ensure the url represents a running admin server and that the credentials are correct. If using http protocol, tunneling must be enabled on the admin server.
 [wldeploy]     at weblogic.Deployer.run(Deployer.java:76)
 [wldeploy]     at weblogic.Deployer.mainWithExceptions(Deployer.java:63)
 [wldeploy]     at weblogic.ant.taskdefs.management.DeployerWrapper.main(DeployerWrapper.java:14)
 [wldeploy] Caused by: weblogic.deploy.api.tools.deployer.DeployerException: Unable to connect to 't3://adminserver:adminport': Unknown object in LOCATE_REQUEST. Ensure the url represents a running admin server and that the credentials are correct. If using http protocol, tunneling must be enabled on the admin server.
 [wldeploy]     at weblogic.deploy.api.tools.deployer.Jsr88Operation.connect(Jsr88Operation.java:317)
 [wldeploy]     at weblogic.deploy.api.tools.deployer.Deployer.perform(Deployer.java:137)
 [wldeploy]     at weblogic.deploy.api.tools.deployer.Deployer.runBody(Deployer.java:88)
 [wldeploy]     at weblogic.utils.compiler.Tool.run(Tool.java:158)
 [wldeploy]     at weblogic.utils.compiler.Tool.run(Tool.java:115)
 [wldeploy]     at weblogic.Deployer.run(Deployer.java:74)
 [wldeploy]     ... 2 more
 [wldeploy] Caused by: javax.enterprise.deploy.spi.exceptions.DeploymentManagerCreationException: [J2EE Deployment SPI:260010]Unable to connect to "t3://adminserver:adminport" as user, "adminuser". Error received: Couldn't connect to the specified host
 [wldeploy]     at weblogic.deploy.api.spi.deploy.WebLogicDeploymentManagerImpl.(WebLogicDeploymentManagerImpl.java:137)
 [wldeploy]     at weblogic.deploy.api.spi.factories.internal.DeploymentFactoryImpl.getDeploymentManager(DeploymentFactoryImpl.java:86)
 [wldeploy]     at weblogic.deploy.api.tools.SessionHelper.getRemoteDeploymentManager(SessionHelper.java:530)
 [wldeploy]     at weblogic.deploy.api.tools.deployer.Jsr88Operation.connect(Jsr88Operation.java:298)
 [wldeploy]     ... 7 more
 [wldeploy] Caused by: weblogic.deploy.api.spi.exceptions.ServerConnectionException: [J2EE Deployment SPI:260010]Unable to connect to "t3://adminserver:adminport" as user, "adminuser". Error received: Couldn't connect to the specified host
 [wldeploy]     at weblogic.deploy.api.spi.deploy.internal.ServerConnectionImpl.getMBeanServer(ServerConnectionImpl.java:258)
 [wldeploy]     at weblogic.deploy.api.spi.deploy.internal.ServerConnectionImpl.getMBeanServerForType(ServerConnectionImpl.java:198)
 [wldeploy]     at weblogic.deploy.api.spi.deploy.internal.ServerConnectionImpl.init(ServerConnectionImpl.java:148)
 [wldeploy]     at weblogic.deploy.api.spi.deploy.WebLogicDeploymentManagerImpl.getNewConnection(WebLogicDeploymentManagerImpl.java:164)
 [wldeploy]     at weblogic.deploy.api.spi.deploy.WebLogicDeploymentManagerImpl.(WebLogicDeploymentManagerImpl.java:134)
 [wldeploy]     ... 10 more
 [wldeploy] Caused by: java.io.IOException: Couldn't connect to the specified host
 [wldeploy]     at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:237)
 [wldeploy]     at weblogic.management.remote.common.ClientProviderBase.newJMXConnector(ClientProviderBase.java:120)
 [wldeploy]     at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnectorFactory.java:371)
 [wldeploy]     at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:269)
 [wldeploy]     at weblogic.deploy.api.spi.deploy.internal.ServerConnectionImpl.getMBeanServer(ServerConnectionImpl.java:247)
 [wldeploy]     ... 14 more
 [wldeploy] Caused by: javax.naming.NamingException: Couldn't connect to the specified host [Root exception is org.omg.CORBA.OBJECT_NOT_EXIST: Unknown object in LOCATE_REQUEST  vmcid: 0x0  minor code: 0  completed: No]
 [wldeploy]     at weblogic.corba.j2ee.naming.Utils.wrapNamingException(Utils.java:83)
 [wldeploy]     at weblogic.corba.j2ee.naming.ORBHelper.getORBReferenceWithRetry(ORBHelper.java:638)
 [wldeploy]     at weblogic.corba.j2ee.naming.ORBHelper.getORBReference(ORBHelper.java:582)
 [wldeploy]     at weblogic.corba.j2ee.naming.InitialContextFactoryImpl.getInitialContext(InitialContextFactoryImpl.java:85)
 [wldeploy]     at weblogic.corba.j2ee.naming.InitialContextFactoryImpl.getInitialContext(InitialContextFactoryImpl.java:31)
 [wldeploy]     at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:46)
 [wldeploy]     at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
 [wldeploy]     at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313)
 [wldeploy]     at javax.naming.InitialContext.init(InitialContext.java:244)
 [wldeploy]     at javax.naming.InitialContext.(InitialContext.java:216)
 [wldeploy]     at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:221)
 [wldeploy]     ... 18 more
 [wldeploy] Caused by: org.omg.CORBA.OBJECT_NOT_EXIST: Unknown object in LOCATE_REQUEST  vmcid: 0x0  minor code: 0  completed: No
 [wldeploy]     at weblogic.iiop.LocateReplyMessage.needsForwarding(LocateReplyMessage.java:171)
 [wldeploy]     at weblogic.iiop.IORManager.locateInitialReference(IORManager.java:251)
 [wldeploy]     at weblogic.corba.orb.ORB.resolve_initial_references(ORB.java:258)
 [wldeploy]     at weblogic.corba.client.naming.ClientORBInfo.(ClientORBInfo.java:34)
 [wldeploy]     at weblogic.corba.client.naming.ClientORBHelper.createORBInfo(ClientORBHelper.java:42)
 [wldeploy]     at weblogic.corba.j2ee.naming.ORBHelper.cacheORB(ORBHelper.java:310)
 [wldeploy]     at weblogic.corba.j2ee.naming.ORBHelper.cacheORBAndCreateURL(ORBHelper.java:319)
 [wldeploy]     at weblogic.corba.j2ee.naming.ORBHelper.getORBReferenceWithRetry(ORBHelper.java:622)
 [wldeploy]     ... 27 more
 [wldeploy] webLogic.Deployer execution failed

Weblogic admin server and the managed servers in the cluster were in RUNNING state without any problems.

Cause for : Unknown object in LOCATE_REQUEST vmcid: 0x0 minor code: 0 completed: No

Although it doesn't mention in the exception stack trace, this error was caused by the missing libraries in the classpath. This was made explicit when we try to use the WLST PING, to ping the server using the following command.

java -cp /apps/wls12130/wlserver/server/lib/ weblogic.Admin -url t3://adminserver:adminport -username adminuser  -password adminpassword    PING

Running this command throws an explicit CLASSPATH error as shown below

Error: Could not find or load main class weblogic.Admin

Clearly, setting the server/lib in the classpath was not enough. This error will obviously happen with other utilities such as weblogic.Deployer or weblogic.Server etc.

Fix for : Unknown object in LOCATE_REQUEST vmcid: 0x0 minor code: 0 completed: No

To fix this issue, we have to set the environment such as PATH and CLASSPATH properly. Luckily, Weblogic offers a setWLSEnv.sh script that sets up the environment as required for WLST commands. Follow these steps to attempt to fix this issue.

1. Set the MW_HOME variable to the Weblogic server home directory. For Weblogic 12.1.3, this was MW_HOME, however if you are using Weblogic 12.2, you may have to set the ORACLE_HOME to the Weblogic home directory (I did not try Weblogic 12.2 though).

   export MW_HOME=/apps/wls12130/

2. Run the setWLSEnv.sh script

   . $MW_HOME/wlserver/server/bin/setWLSEnv.sh

Once run this script, you should be able to run the previous command without setting the classpath explicitly.

java weblogic.Admin -url t3://adminserver:adminport -username adminuser  -password adminpassword    PING

ant deploy -Dweblogic.user=adminuser -Dweblogic.password=adminpassword -Dadminurl=t3://admin-server:admin-port -Dweblogic.cluster=ClusterName