Renewal process for Bootstrap VM-auth key for panorama orchestrated VM firewalls in AWS ASG
Symptom
Bootstrap vm-Authkey about to expire and need to be renewed forcefully in a maintenance window.
Environment
**Product_versions**
Panorama version 11.2.x
AWS firewalls
AWS plugin version 5.4.1
Cause
If the vm bootstrap authkey timer expires during the production hours then it will initiate redeployment of all the panorama orchestrated autoscaled firewalls causing a production outage.
Resolution
The AWS plugin automatically generates a new auth key upon expiration, which forces a rolling upgrade. To renew a key before it expires, customers can revoke it using the method below on panorama CLI in a planned maintenance window.
Check the if vm authkey is valid or about to expire with command below
request bootstrap vm-auth-key show
Even if we regenerate the authkey it may not initiate a rolling upgrade process and redeployment of firewalls so existing valid authkey would need to be revoked with command below
request bootstrap vm-auth-key revoke vm-auth-key <value>
After this command is ran, new authkey generation can be triggered with a commit force
>configure
#commit force
Redeployment should start in all regions in parallel.
It can be monitored by running the below command and verifying the logs on console screen.
>tail follow yes plugins-log plugin_aws_deployment.log
Note:
vm auth key is not renewed automatically when once it expires.
Additional Information
There are few known issues that could cause failures
- Hitting the AWS default limit of 20 route tables per TGW
- S3 bucket permission issues ( resolved)- Internal resource access issue.
- When we hit redeploy, It dissociates and associate a new route table to the transit gateway.
- Dissociating the old route table and associating a new route table will mean a significant downtime.
- TGW goes missing from route table for which the fix is in progress
- In upcoming version (version TBD) change will be made to not dissociate (the old route table from the transit gateway) and associate a new route table to the transit gateway.