Why are RAID rebuild messages seen on Panorama every Sunday 1:00 AM.
12119
Created On 01/26/22 10:33 AM - Last Modified 04/23/24 03:35 AM
Question
When there is no disk trouble, why does Panorama "rebuild" the disk every Sunday at 1:00 AM?
- RAID rebuild of disk pairs starts on both the members of the cluster without any apparent reason
- This starts Sunday 1:00 AM.
- The following messages are seen in "raid.log" (less mp-log raid.log) taken after rebuild.
Jan 16 01:00:01 DEBUG: raid_util: argv: ['RebuildStarted', '/dev/md2']
Jan 16 01:00:02 INFO: Rebuild of Disk Pair B Partition 1 started.
Jan 16 01:00:07 DEBUG: raid_util: argv: ['RebuildStarted', '/dev/md1']
Jan 16 01:00:08 INFO: Rebuild of Disk Pair A Partition 1 started.
Jan 16 03:11:43 DEBUG: raid_util: argv: ['Rebuild20', '/dev/md2']
Jan 16 03:11:44 DEBUG: Rebuild of Disk Pair B Partition 1 20 percent complete.
Jan 16 03:11:44 DEBUG: raid_util: argv: ['Rebuild20', '/dev/md1']
Jan 16 03:11:44 DEBUG: Rebuild of Disk Pair A Partition 1 20 percent complete.
Jan 16 05:22:50 DEBUG: raid_util: argv: ['Rebuild40', '/dev/md2']
Jan 16 05:22:50 DEBUG: Rebuild of Disk Pair B Partition 1 40 percent complete.
Jan 16 05:23:20 DEBUG: raid_util: argv: ['Rebuild40', '/dev/md1']
Jan 16 05:23:21 DEBUG: Rebuild of Disk Pair A Partition 1 40 percent complete.
Jan 16 07:36:56 DEBUG: raid_util: argv: ['Rebuild60', '/dev/md2']
Jan 16 07:36:57 DEBUG: Rebuild of Disk Pair B Partition 1 60 percent complete.
Jan 16 07:40:57 DEBUG: raid_util: argv: ['Rebuild60', '/dev/md1']
Jan 16 07:40:58 DEBUG: Rebuild of Disk Pair A Partition 1 60 percent complete.
Jan 16 10:11:04 DEBUG: raid_util: argv: ['Rebuild80', '/dev/md2']
Jan 16 10:11:04 DEBUG: Rebuild of Disk Pair B Partition 1 80 percent complete.
Jan 16 10:20:05 DEBUG: raid_util: argv: ['Rebuild80', '/dev/md1']
Jan 16 10:20:05 DEBUG: Rebuild of Disk Pair A Partition 1 80 percent complete.
Jan 16 13:28:54 DEBUG: raid_util: argv: ['RebuildFinished', '/dev/md2']
Jan 16 13:28:54 INFO: Rebuild of Disk Pair B Partition 1 finished.
Jan 16 13:43:55 DEBUG: raid_util: argv: ['RebuildFinished', '/dev/md1']
Jan 16 13:43:56 INFO: Rebuild of Disk Pair A Partition 1 finished.
Environment
- Panorama M-200, M-500, M-600
- PA-5200 series.
- PAN-OS 10.0
- PAN-OS 10.1.0 - 10.1.8-h6
- PAN-OS 10.2.0 - 10.2.3-h9
Answer
- By default, this RAID management utility will schedule a "checking" job at every Sunday 1:00 AM by cron job. This feature is added from version 10.0.
- The "checking" is not "rebuilding", but kernel/mdadm can not distinguish these two cases and so the same message ("Rebuild of Disk Pair ...") in captured in the raid.log.
Additional Information
During the checking RAID disk, show output of "show system raid detail" as the below.
Panorama> show system raid detail
Disk Pair A Available
Status clean, checking <<<<
Disk id A1 Present
model : ST8000NM0055-1RM
size : 7630885 MB
status : active sync
Disk id A2 Present
model : ST8000NM0055-1RM
size : 7630885 MB
status : active sync
Disk Pair B Available
Status clean, checking <<<<
Disk id B1 Present
model : ST8000NM0055-1RM
size : 7630885 MB
status : active sync
Disk id B2 Present
model : ST8000NM0055-1RM
size : 7630885 MB
status : active sync
During the scheduled RAID checking, the status is shown as "clean, checking".
If the actual rebuilding was run, the status was shown as "rebuilding".
The default date and time is not able to be changed.
This schedule is set at every Sunday 1:00 AM.
This behavior is observed not only Panorama but also PA-5200 series.
And, this behavior is fixed in PAN-OS 10.2.4 as PAN-207940 .