Message Engine doesn’t start after setting up cluster

This issue hit me a few times and always took me some time to figure out what happened. So I thought it’s a good idea to note it down.

Symptom:

When setting up a cluster environment for Maximo, I will need to set up an integration bus with a message engine for each cluster (IF, UI, Cron etc.)

Each message engine will require its own individual schema (and thus individual user if the Oracle DB is used)

After the Integration Bus was configured and the Maximo cluster was started, we saw a lot of errors in the log file, usually in the Cron or MIF cluster due to the message engine is not available.

When restarting the cluster, we can see that the message engine for that cluster has a partial started status. But a few minutes after the whole cluster is started, the message engine would show an unavailable status.

Troubleshoot:

• Check ffdc log under [Websphere_Home]/AppServer/profiles/ctgAppSrv01/logs/ffdc, check the [MXServer_Name]_exception.log file to see if there are any exceptions related to Integration Bus or Message Engine such as the one below:

com.ibm.ws.sib.msgstore.persistence.DatasourceWrapperStoppedException com.ibm.ws.sib.msgstore.persistence.impl.PersistentMessageStoreImpl.start 1:206:1.47.1.53 D:IBMWebSphereAppServerprofilesctgAppSrv01logsffdcMAXUI-N1-4_58bec328_22.10.24_15.50.24.7956943157914483024680.txt

com.ibm.ws.sib.msgstore.PersistenceException com.ibm.ws.sib.msgstore.impl.MessageStoreImpl.start 755 D:IBMWebSphereAppServerprofilesctgAppSrv01logsffdcMAXUI-N1-4_58bec328_22.10.24_15.50.24.8271423565797649410081.txt

• For each of the above exceptions, open the file it referenced. We might see some detailed error messages which could help us to solve the issue.
• In this case, I have this vague error message below:

CWSIS1501E: The data source has produced an unexpected exception: com.ibm.ws.sib.msgstore.persistence.DatasourceWrapperStoppedException: New connections cannot be provided because the persistence layer has been stopped

Solution:

• For this specific case, it is caused by a failed setup process in a previous setup which left a bunch of tables created under different message engine details. Thus, the current message engine doesn’t like to reuse it. To solve this issue, we can simply change the schema name of the message store so that the next time the cluster starts, it will create a whole new bunch of tables for its own use.
• If Oracle database is used, the schema is linked with the DB account used for login and is more difficult to change. Thus, I’ll just drop all the tables used by that schema. One quick way to identify and delete those tables is by running the query below

select ‘drop table ‘ || table_name || ‘;’ from user_tables;