Port 61616 of Activemq hangs up by itself, and then gets stuck.

Hello, prawns!

activemqactivemq.xmlzk:
    <persistenceAdapter>
    <replicatedLevelDB
      directory="${activemq.data}/leveldb"
      replicas="1"
      bind="tcp://0.0.0.0:0"
      zkAddress="172.1.1.24:2181,172.1.1.25:2181,172.1.1.28:2181"    -sharpzk
      hostname="172.1.1.28"    -sharpMQzk
      sync="local_disk"
      zkPath="/activemq/leveldb-stores"
      />
</persistenceAdapter>

Port 61616 of mq is found to be down this morning. Check the activemq.log, log as follows:

2018-09-21 11:39:11,174 | INFO  | Opening socket connection to server 172.1.1.28/172.1.1.28:2181 | org.apache.zookeeper.ClientCnxn | main-SendThread(172.1.1.28:2181)
2018-09-21 11:39:11,174 | WARN  | unprocessed event state: AuthFailed | org.apache.activemq.leveldb.replicated.groups.ZKClient | main-EventThread
2018-09-21 11:39:11,180 | INFO  | Socket connection established to 172.1.1.28/172.1.1.28:2181, initiating session | org.apache.zookeeper.ClientCnxn | main-SendThread(172.1.1.28:2181)
2018-09-21 11:39:11,193 | INFO  | Session establishment complete on server 172.1.1.28/172.1.1.28:2181, sessionid = 0x365f7dbb9800034, negotiated timeout = 4000 | org.apache.zookeeper.ClientCnxn | main-SendThread(172.1.1.28:2181)
2018-09-21 11:39:11,518 | INFO  | Using the pure java LevelDB implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ BrokerService[test-leveldb] Task-1
2018-09-21 11:39:11,529 | INFO  | Attaching to master: tcp://172.1.1.28:46816 | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | ActiveMQ BrokerService[test-leveldb] Task-1
2018-09-21 11:39:11,537 | INFO  | Slave started | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ BrokerService[test-leveldb] Task-1
2018-09-21 11:39:11,545 | WARN  | Unexpected session error: java.net.ConnectException: Connection refused | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1
2018-09-21 11:39:12,553 | INFO  | Using the pure java LevelDB implementation. | org.apache.activemq.leveldb.LevelDBClient | hawtdispatch-DEFAULT-1
2018-09-21 11:39:12,555 | INFO  | Attaching to master: tcp://172.1.1.28:46816 | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1
2018-09-21 11:39:12,556 | WARN  | Unexpected session error: java.net.ConnectException: Connection refused | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1

unlimited card here in Connection refused, 172.1.1.28 (that is, the one where mq is located) does not have port 46816 at all. I have checked that the firewalls are all good.

check the zk, and find that a follower is dead and there are a lot of reconnections in it:

2018-09-21 11:43:00,047 [myid:2] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x265fa2c442e0003, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:748)
2018-09-21 11:43:00,047 [myid:2] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /172.1.1.28:54869 which had sessionid 0x265fa2c442e0003

so I restart the zk, state is follower, again, but the activemq restart is still stuck in the stage of Attaching to master and then Connection refused. What should I do?

Jul.07,2021
Menu