Avamar - MCS service not starting

After powering on an Avamar, the MCS service will not start.


admin@Avamar:~/>: dpnctl status
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: down.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: disabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: ConnectEMC status: up.
dpnctl: INFO: ddrmaint-service status: up.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

Trying to start the MCS service normally generates an error, and the service will not start.


admin@Avamar:~/>: dpnctl start mcs
dpnctl: INFO: Starting MCS...
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-mcs-start-output-26146
dpnctl: WARNING: 1 warning seen in output of "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start"

admin@Avamar:~/>:

admin@Avamar:~/>: dpnctl status
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: down.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: disabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: ConnectEMC status: up.
dpnctl: INFO: ddrmaint-service status: up.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

Force a MCS restore with:

admin@Avamar:~/>: dpnctl start --force_mcs_restore

  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -  -
Action: starting all
Have you contacted Avamar Technical Support to ensure that this
  is the right thing to do?

Answering y(es) proceeds with starting all;
          n(o) or q(uit) exits

y(es), n(o), q(uit/exit): y
dpnctl: INFO: gsan is already running.
dpnctl: INFO: Restoring MCS data...
dpnctl: INFO: MCS data restored.
dpnctl: INFO: Starting MCS...
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-mcs-start-output-26146
dpnctl: WARNING: 1 warning seen in output of "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start"
dpnctl: INFO: MCS started.
dpnctl: INFO: EM Tomcat is already running, not attempting to restart it
dpnctl: INFO: Resuming backup scheduler...
dpnctl: INFO: Backup scheduler resumed.
dpnctl: INFO: AvInstaller is already running.
admin@Avamar:~/>:
admin@Avamar:~/>: dpnctl status
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: disabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: ConnectEMC status: up.
dpnctl: INFO: ddrmaint-service status: up.

Verify the Avamar is now online and good to  go.

admin@Avamar:~/>: status.dpn
Wed Mar 21 13:03:48 PDT 2018  [AVAMAR.XIOLOGIX.LOCAL] Wed Mar 21 20:03:48 2018 UTC (Initialized Fri Jun 23 19:03:09 2017 UTC)
Node   IP Address     Version   State   Runlevel  Srvr+Root+User Dis Suspend Load UsedMB Errlen  %Full   Percent Full and Stripe Status by Disk
0.0    10.10.10.100   7.4.1-58  ONLINE fullaccess mhpu+0hpu+0hpu   1 false   0.88 9545  2606806   0.1%   0%(onl:40 )  0%(onl:36 )
Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: xxxxxxxxxx@00:oo:00:oo:00:ee

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status: ok
Access-Status: full

No checkpoint yet
No GC yet
No hfscheck yet

Maintenance windows scheduler capacity profile is active.
  The maintenance window is currently running.
  Currently running task(s): script
  Next backup window start time: Wed Mar 21 20:00:00 2018 PDT
  Next maintenance window start time: Thu Mar 22 08:00:00 2018 PDT

If the Scheduler is not started, restart the scheduler service

Maintenance windows scheduler capacity profile is active.
  WARNING: Scheduler is WAITING TO START until Thu Mar 22 08:00:00 2018 PDT.
  Next backup window start time: Thu Mar 22 20:00:00 2018 PDT
  Next maintenance window start time: Thu Mar 22 08:00:00 2018 PDT
admin@NC-Avamar:~/>: dpnctl stop maint
dpnctl: INFO: Suspending maintenance windows scheduler...
admin@NC-Avamar:~/>: dpnctl start maint
dpnctl: INFO: Resuming maintenance windows scheduler...
dpnctl: INFO: maintenance windows scheduler resumed.

Avamar - RabbitMQ errors

If you are seeing RabbitMQ errors in your Avamar logs, you'll want to correct the issue. The RabbitMQ error is often due to a mis-configuration of the BRM Service. If you're not using BRM, but the MCS is expecting to see/send data to BRM you can get errors in the logs for the RabbitMQ.

Check your mcserver.xml file for a line "enableBrmService". If set to "true" and you're not using BRM, then it should be set to "false".

admin@AVAMAR:/usr/local/avamar/var/mc/server_data/prefs/>: grep -i enableBRM mcserver.xml

You should see output similar to

<entry key="enableBrmService" value="true" />

If so, edit the mcserver.xml with vi, and change the entry to "false".

Stop, then start the MCS service

mcserver.sh --stop

mcserver.sh --start

Log back into the GUI and clear out the error logs.

You should not see the RabbitMQ errors any more.