HealthCheck Assistance
The Health Check has detected that either the Buddy, MOS or Archive queue has grown beyond a specified threshold.
Although the queues enable different functionality, the issues have similar causes: a remote system or configuration issue. If the threshold is breached, it means that a large quantity of objects have not been sent to a receiving system.
In the case of the Buddy queue (Common\BUDDY), this would mean that if the Primary Server became unavailable, the data set on the Buddy Server would be incomplete or potentially lost. For the Buddy queue, a configuration option is available that will take a back up of the current queue for investigative purposes. The backup will be placed in <data drive>:\BUDDY-QUEUE-BACKUP\yyyyMMdd-HHmm. See Thresholds section below to enable.
A MOS queue (NOM\MOS) would indicate that an integrated MOS system is not being updated with the correct MOS status of the required ENPS objects.
For both issues, the causes could be similar: configuration, the remote system is unavailable/inaccessible or the processes on the remote system are not operational.
Additionally, with ENPS version 9.3 and newer, if the Buddy queue builds but does not clear, this could require the Solr schema to be regenerated on both servers and the data being recrawled.
If it is the Archive queue (NOM\ARCHIVEQ), there may well be a object that is preventing the archive process from completing the archiving of running orders.
Buddy Queue:
Maximize the NOM application, from the Window option, select Nom Configuration/Status.
If the Buddy Server is showing as down, access the console on the Buddy Server and check that the ENPS processes are running. If not, launch NOM Watcher to start the ENPS server components in order to process the Buddy objects.
If the processes on the Buddy Server are running, open System Maintenance, navigate to the Servers table and check the entry in the "Buddy" column. Check the the entry for that server and ensure that the "IP" column is configured correctly.
Confirm that system resolution is working as expected, run ping <entry from IP column> Note: depending on the operating system, the firewall may not be enabled for ICMP packets.
A further test could be attempted to check that both resolution and the required ports are open using a PowerShell command: Test-NetConnection -ComputerName:<entry from IP/Endpoint column> -Port:<8080> (This checks the resolution and the state of the NOM/Solr service)
If the resolution fails, contact your corporate network infrastructure team.
If the processes are running on the remote server, and the configurations and resolution have been confirmed, the remote server may need to the schema to be rebuilt. This can be achieved by running F:\SearchServer\SolrSchemaManager.exe /regenerate. If your ENPS data drive is not the F: drive, change the drive letter to the appropriate drive.
MOS Queue:
Attempt to move the top item in the MOS queue out of the queue by moving it from the device folder in F:\NOM\MOS\OUT. Save the file for further examination by the AP ENPS Helpdesk. If that does not clear the queue, try the following steps:
Open System Maintenance, navigate to the MOS Configuration table and check the entry in the "IP/Endpoint" column is that of the MOS system.
Confirm that system resolution is working as expected, run ping <entry from IP/Endpoint column> Note: depending on the operating system, the firewall may not be enable for ICMP packets.
A further test could be attempted to check that both resolution and the required ports are open using a PowerShell command: Test-NetConnection -ComputerName:<entry from IP/Endpoint column> -Port:<MOS Port> (Ports 10540 and 10541 for MOS v2.
Port 10542 for federated search. Port 10543 for
MOS v3.)
Ensure the required process/services are running on the MOS system.
ARCHIVEQ Queue:
Restarting the NOM at a convenient time in the broadcast schedule may resolve this but should the issue continue, along with either the buddy or MOS queues, please contact the AP ENPS Helpdesk.
NOTE: If you have any questions, please contact AP ENPS Support before making changes to the HealthCheck configurations.
There are thresholds set within \configurables\hc-custom_config.ps1. With the default being:
BUDDY:
[int]$global:HC_Threshold_Buddy_Queue_Critical = 5000 # Set the threshold to check the buddy queue (Default: 5000)
[int]$global:HC_Threshold_Buddy_Queue_Monitor = 2000 # Set the threshold to check the buddy queue (Default: 250)
To enable the buddy queue backup option in \configurables\hc-custom_config.ps1 should the threshold be breached the following can be enabled but could extend the time the HC runs if the queue is hundreds or thousands of objects, along with the disk space required, so consider this carefully be enabling:
$global:buddy_queue_backup_if_breached = $false # Backup buddy queue if below critical threshold is breached (default = $false)
MOS:
[int]$global:HC_Threhold_Mos_Queue_Critical = 5000 # Set the threshold to check the mos queue (Default: 500)
[int]$global:HC_Threhold_Mos_Queue_Monitor = 4000 # Set the threshold to check the mos queue (Default: 250)
[array]$global:HC_MOS_DEVICES_TO_EXLUDE = "OUT","IN" # List of MOS Devices where the queue will not be checked. If more than one, list as "deviceIdOne","deviceIdTwo","deviceIdThree" (hc_suite_status_enps_queues)
ARCHIVEQ:
[int]$global:HC_Threhold_ARCHIVEQ_Queue_Critical = 8 # Set the threshold to check the ARCHIVEQ queue (Default: 8)(hc_suite_status_enps_queues)