Category: Websphere (Page 1 of 3)

Message Engine doesn’t start after setting up cluster

15 January 2023 / Viet Tran / 0 Comments

This issue hit me a few times and always took me some time to figure out what happened. So I thought it’s a good idea to note it down.

Symptom:

When setting up cluster environment for Maximo, I will need to setup an integration bus with a message engine for each cluster (IF, UI, Cron etc.)

Each message engine will require its own individual schema (and thus individual user if the Oracle DB is used)

After integration bus are setup and Maximo cluster started, we see a lot of errors in the log file, usually in the Cron or MIF cluster due to message engine is not available.

When restarting the cluster, we can see that the message engine for that cluster has a “partial started” status. But a few minutes after the whole cluster is started, the message engine would show an “unavailable” status.

Troubleshoot:

Check ffdc log under [Websphere_Home] AppServerprofilesctgAppSrv01logsffdc, check for [MXServer_Name]_exception.log to see if there are any exception related to integration bus or message engine such as:

com.ibm.ws.sib.msgstore.persistence.DatasourceWrapperStoppedException com.ibm.ws.sib.msgstore.persistence.impl.PersistentMessageStoreImpl.start 1:206:1.47.1.53 D:IBMWebSphereAppServerprofilesctgAppSrv01logsffdcMAXUI-N1-4_58bec328_22.10.24_15.50.24.7956943157914483024680.txt

com.ibm.ws.sib.msgstore.PersistenceException com.ibm.ws.sib.msgstore.impl.MessageStoreImpl.start 755 D:IBMWebSphereAppServerprofilesctgAppSrv01logsffdcMAXUI-N1-4_58bec328_22.10.24_15.50.24.8271423565797649410081.txt

For each of the above exception, open the file it referenced. We might see some detailed error message which could help us to solve the issue.
In this case, I have this vague error message below:

CWSIS1501E: The data source has produced an unexpected exception: com.ibm.ws.sib.msgstore.persistence.DatasourceWrapperStoppedException: New connections cannot be provided because the persistence layer has been stopped

Solution:

For this specific case, it is caused by a failed setup process in previous setup which left a bunch of tables created under different message engine details. Thus, the current message engine doesn’t like to reuse it. To solve this issue, we can simply change the schema name of for the message store so that the next time the cluster starts, it will create a whole new bunch of tables for it own use.
If Oracle database is used, schema is linked with the DB account used for login, and is more difficult to change. Thus, I’ll just drop all the tables used by that schema. One quick way to identify and delete those tables is running this query:

select ‘drop table ‘ || table_name || ‘;’ from user_tables;

Issues with Maximo attachment (DOCLINKS)

22 August 2022 / Viet Tran / 0 Comments

Just a bunch of my personal notes about Maximo attachment (DOCLINKS)

When a file is attached to a record in Maximo, it creates a record in the DOCLINKS and a record in the DOCINFO table to keep the details of the file. The file is copied to a location on disk, usually on a local folder (e.g. D:DOCLINKS) or on a shared network folder. The path for Maximo to read the file is kept in the URLNAME field of the DOCINFO table.
To set up this function, refer to this blog post by Bruno
When a user uploads an attachment, the location where Maximo puts the file will depend on the Folder selected:

In most applications such as Work Order Tracking or Assets, there is a Select Action > Attachment Library/Folders > Manage Folder function, it allows you to specify the location to keep the file for each Folder.

If no path is specified for a folder, Maximo uses the default value of the System Properties: mxe.doclink.doctypes.defpath

This setting controls where to keep the file when uploading. After a migration or upgrade in which you moved your file server location, and if new files still end up in the old location. This is where you should update your settings
When a user clicks on the link to download an attachment, there are two methods for Maximo to serve the file.

If System Property: mxe.doclink.securedAttachment is set to False (default), the HTTP Web Server will read the file from disk, then serve it to the end-users. Maximo application server doesn’t play a role in delivering the file here.

If System Property: mxe.doclink.securedAttachment is set to True, when the end-user downloads an attachment, Maximo application server will grab the file, encrypt it, then serve it to the end-user. Thus, in this case, we don’t really need the web server for it to work. I.e. if you accessing Maximo directly on the application server (via port
908x), it will still work. Thus, in a clustered environment in which a load balancer distributes load directly to Maximo JVMs without a web server in the middle, this is the recommended choice.
The link to download an attachment is kept in DOCINFO.WEBURL non-persistent field. The value of this field is generated by replacing the URLNAME field of the DOCINFO record, with the rule set in the mxe.doclink.pathXX property.
For example, if you have the following rule: mxe.doclink.path01=D:DOCLINKS=https://maximo.company.com/attachments/ For a document with URLNAME=D:DOCLINKSdrawingfile01.pdf, the WEBURL would be https://maximo.company.com/attachments/drawing/file01.pdf. If the first part of URLNAME doesn’t match with the left-side of the PATH01 equation, Maximo will look at mxe.doclink.path02,path03 and so on to replace the value and generate a WEBURL. If there’s no match, it will generate WEBURL with an absolute path: file://[URLNAME]. This means that if the user has access to the file on the shared network path, it will still work without the need for a web server.

In summary, I usually hit with issues with doclinks after an upgrade or migration, I use the following checklist to avoid those issues:

If a new Web server is set up, need to update the DocumentRoot, and the Directory settings in the web server’s httpd.conf file
If the Web URL to access Maximo changes: need to update mxe.doclink.path01
If the file server location was moved:
- Need to copy the file to the new location
- Maximo service account should have permission to access the local/network folder
- Update:
  - mxe.doclink.doctypes.topLevelPaths
  - mxe.doclink.doctypes.defpath
  - mxe.doclink.path01
- Update the DOCTYPES.DEFAULTFILEPATH in
  the database with a REPLACE SQL command
- Update all DOCINFO.URLNAME in the database with a REPLACE SQL command

How to test SMTP with PowerShell for setting up Maximo email notifications?

30 July 2022 / Viet Tran / 0 Comments

In an enterprise IT environment, it is sometimes difficult to set up Maximo to talk with SMTP service due to networking and security restrictions. To troubleshoot SMTP configuration, in the past, we used Telnet from the CMD tool. However, in newer versions of Windows Server, Telnet is often not installed by default. In such cases, we can use PowerShell to test and confirm the SMTP and firewall setting is working before configuring SMTP in Maximo.

The first thing I would do is check whether firewall has been opened so that the port used by SMTP service (e.g. port 25) is reachable from the Maximo server. I’ve provided details on how to do this with PowerShell in this post: Check network firewall status with PowerShell

Once it is confirmed SMTP service is reachable, the next step I would do is to ensure we can send an email with the provided login details. Thus, from PowerShell running on the Maximo server, I would do the following commands:

Note that if SMTP service doesn’t require authentication, we don’t need to run the first command and use the parameter -Credential $cred in the second command.

Replace placeholder values with actual details required by your SMTP service, and for the -To parameter, enter your email address.

The Send-MailMessage command will send an email using your SMTP
service. If it works, you should receive an email.

Once SMTP service is confirmed working, I’ll go ahead putting the details to System Properties in Maximo and that should be it.

To test that Maximo can send emails, the simplest way I would
do is to open any work order, then use the Select Action > Create >
Communication, then entering my own email into the “To” address field. If it
works, after clicking on Send, it should show a successful status, and you’ll
receive an email within seconds. Otherwise, you might want to look at
SystemOut.log file to see if there are any details of the error there.

Websphere messaging engine cannot be started as there is no runtime initialized for it yet

20 May 2022 / Viet Tran / 0 Comments

I sometimes have issues with message engine not running. Usually I’ll just try to restart the whole system and hope that it goes away.

If it doesn’t work, in most cases, it is caused by a corrupted file store used by the message engine and the suggestion from the Internet is to delete these files, which seems to work fine.

Sometimes, with the message engine uses a database store, I had a very similar issue. I find it quite hard to find out the exact root cause. So I chose the easier path by simply deleting the whole message engine, create a new one, giving a new schema name for the data store. This ensures it creates new tables when message engine is initialized the first time.

Creating a new message engine and re-assigning bus destinations usually take less than 5 minutes, and it seems a lot easier than troubleshooting and finding the root cause of the issue.

How to redeploy a single file in Maximo?

10 January 2022 / Viet Tran / 1 Comment

Most Maximo settings or Java code can be deployed by copy/paste the file directly to the installed folder in Websphere without having to rebuild and redeploy the application. However, with web.xml, it doesn’t work that way. Sometimes, we need to update this file to increase the timeout setting or enable/disable LDAP integration

Sure, we can directly modify the file in Websphere without redeployment, but we will also have to update the file in a few temporary folders for which, I find the process quite tedious.

To avoid having to rebuild and redeploy the whole maximo.ear file, which can take a lot of time, we can just redeploy the single web.xml file instead. Below is the process:

Update the web.xml file with new settings

Log in to the Websphere console, open Applications > Application Types > WebSphere Enterprise Applications: select “MAXIMO” application by ticking on the checkbox next to it, click on the Update button
In “Application update options“, select “Replace or add a single file” option
In the textbox below “Specify the relative path….“, specify: maximouiweb.war/WEB-INF/web.xml
In the “Specify the path to the file“, choose “Local file system“, and click on “Choose file” to browse and select the updated web.xml file, click Next.
Click OK on the next screen to deploy. Click Save when the deployment process is completed.
Wait for a minute for the new settings to be propagated to all nodes, then restart Maximo.

Node Agent cannot start after configuring LDAP

3 September 2021 / Viet Tran / 0 Comments

Just to document a weird issue I had today. I was attempting to configure LDAP (Microsoft AD) for Maximo/Websphere. After it is configured (and I’ve tested that it can query AD data), Application Server security was enabled. Then the server is rebooted to refresh the new configuration.

After a restart, ctgNode01 (node agent) service cannot start. Node Agent log shows the error below:

WSVR0100W: An error occurred initializing, nodeagent [class com.ibm.ws.runtime.component.ServerImpl]

com.ibm.ws.exception.ConfigurationError: com.ibm.websphere.ssl.SSLException: CWPKI0316E: Cannot get a security object from the configuration. This can indicate that the security.xml file for the cell is corrupt and you must validate the integrity of the file.

I looked it up on the web, there are some suggestions about a corrupted security.xml file. So I checked and found that the security.xml file in ctgAppSrv01 profile is completely empty. I looked at other Maximo servers and found this file has an exact same content with the security.xml file from ctgDmgr01 profile. I copied that file over to AppSrv profile, restarted Websphere and it is able to start again.

Not sure why configuring LDAP would completely wipe out the content of this file.