Error "Failed to retrieve tenantDomain for region" is seen when trying to retrieve CIE groups on Firewall

• Firewall setup to retrieve group mappings from the Cloud Identity Engine
• Error message "Failed to retrieve tenantDomain.." is seen in debug mode of dscd.log (less mp-log dscd.log)

{"level":"debug","time":"2022-12-02T07:56:44.339811754Z","message":"[CFG-DATA] Sending query to fetch tenantDomain for region europe"} 
{"level":"debug","time":"2022-12-02T07:56:44.339862462Z","message":"[CFG-DATA] TenantDomain: Get tenant Domains url: https://app-directory-sync.eu.paloaltonetworks.com/service/directory/v1/tenantdomains"} 
{"level":"debug","time":"2022-12-02T07:56:44.438575944Z","message":"[CERT-VERIFY]Use local source address 10.14.22.240 for http request"}
{"level":"debug","time":"2022-12-02T07:56:44.440328384Z","message":"[CERT-VERIFY]Use local source address 10.14.22.240 for http request"}
{"level":"debug","time":"2022-12-02T07:56:44.442030888Z","message":"[CERT-VERIFY]Use local source address 10.14.22.240 for http request"}
{"level":"debug","time":"2022-12-02T07:56:44.443351098Z","message":"[CERT-VERIFY]Use local source address 10.14.22.240 for http request"}
{"level":"error","time":"2022-12-02T07:56:46.724860938Z","message":"Failed to retrieve tenantDomain for region europe"}

• Palo Alto Firewalls or Panorama
• PAN-OS 10.1 and above.
• PAN-OS Firewall retrieving Group mappings from Cloud Identity Engine.
• Cloud Identity Engine with cloud directory configuration.

• A successful Region fetch needs to take place for retrieving CIE groups.
• dscd logs indicate the region fetch is failing.

1. Enable the dscd debug log:

   > debug user-id dscd on debug

   1. To determine which region and tenant domain URLs the firewall is unable to reach, look for "Failed to retrieve tenantDomain", "Failed to execute request to DSS" and "tenant Domains url" in the logs.
2. For more information on how to identify the destination URL of the traffic to allow, refer to the document: Configure the Cloud Identity Agent and Configure Cloud Identity Engine Authentication on the Firewall or Panorama.
3. The cloud identity engine uses the Palo Alto Networks services as service route.
4. 
   1. Check the configured Palo Alto Networks service route:
      1. For firewall, navigate to DEVICE > Setup > Services then click Service Route Configuration in the UI.
      2. For Panorama, navigate to Templates > DEVICE > Setup > Services then click Service Route Configuration in the UI.
      3. For Strata Cloud Manager, navigate to Manage > Configuration then select the right Configuration Scope then go to Device Settings then click Config Service Route.
   2. Depending on which service route is configured and used for Palo Alto Networks services, Verify that all of the URLs found in step 1 are not blocked on any devices on the path between firewall/Panorama and the CIE infrastructure.
   3. Check if you need to build a security policy to allow the firewal/Panorama to access these domain, to configure or to edit a security policy:
      1. For firewall, navigate to POLICIES > Security in the UI, and
      2. For Panorama, navigate to Device Groups > POLICIES > Security in the UI.
      3. For Strata Cloud Manager, navigate to Manage > Configuration then select the right Configuration Scope then go to Security Services > Security Policy.
5. Use the following workaround of calling the tenantDomain API if still having issue even after ensuring that the firewall is able to reach the tenant domain.

   > debug user-id refresh cloud-identity-engine config-data

6. Revert the dscd debug logs back to "info" level using the command:

   > debug user-id dscd on info

*.apps.paloaltonetworks.com
app-registry.appsvc.paloaltonetworks.com
enforcer.iot.services-edge.paloaltonetworks.com

Also refer to table for specific regions. If possible, allow access to *.paloaltonetworks.com as well.

Note 2: The firewall and panorama must have a thermite certificate which is the device certificate installed for the feature of providing group mapping from cloud directories to firewall/panorama to work. Ensure that the device-certificate is valid using:

> show device-certificate status
Device Certificate information:
Current device certificate status: Valid
Not valid before: 2024/05/22 18:03:18 JST
Not valid after: 2024/08/14 04:33:32 JST
Last fetched timestamp: 2024/05/22 18:07:51 JST
Last fetched status: success
Last fetched info: Successfully fetched Device Certificate

Note3: If you have deployed a Palo Alto Networks firewall between the agent and the Cloud Identity Engine, use the paloalto-cloud-identity App-ID to allow traffic from the Cloud Identity agent to the Cloud Identity Engine. This App-ID requires the ssl and web-browsing application signatures.

Note4:
 The command to refresh all CIE profiles is:

> debug user-id refresh cloud-identity-engine all

The command to reset all CIE profiles is:

> debug user-id reset cloud-identity-engine all

Use the second command with caution as it is disruptive so it is recommended to issue it during a maintenance window and if all the other attempts to address the issue of connectivity between the firewall/Panorama and the tenantDomain URLs have failed.