DotW: Commit process hangs at 99%

Waiting for a commit to finish and not sure if it’s working properly? There are a few commands you can run to help you see what’s going on.

 

 

From time to time, you may find that your commit hangs at 99%.

This recent discussion in the community can help you learn what to do if you come across this scenario. 

 

Panorama commit to PA4060 hangs at "commit" process 99%

 

In this thread, community member "DISA-CONUS-IP-TIERII" talks about the commit times from Panorama to a PA-4060 unit.

There are a few things you can do to help speed up commits that are taking longer than normal to complete, and a few commands you can run that can help you understand what's going on.

 

When you are performing a commit via the webGUI or the CLI, you are shown a status either with a graphic of a "progress bar" in the WebGUI or with dots and a % showing progress. That's the extent of the detail, unless you open an additional CLI window and run "show jobs id x". "X" is the job number assigned to the commit process. To find out the job number of the commit process, run "show jobs all" and look for the commit process.

 

> show jobs all

 

Enqueued ID Type Status Result Completed
--------------------------------------------------------------------------
2016/01/25 13:27:34 9137 Commit ACT PEND 99%
2016/01/25 13:10:00 9136 WildFire FIN OK 13:12:56

 

Then you can take that ID, and finish the commit with "show jobs id 9137," which gives you a little more information from the commit process.

 

> show jobs id 9137

 

Enqueued ID Type Status Result Completed
--------------------------------------------------------------------------
2016/01/25 13:27:34 9137 Commit ACT PEND 99%
Warnings:
Details:

 

The command output does not show all of the details that you need to determine if a specific daemon is causing a problem.

 

Now, there is another command, "show management-clients". You may think that it will return which management clients are logged into the device. However, this command actually shows the status of all of running daemons that are used during the commit process.

 

Take a look at the output while the commit is running. You can see all of the "clients" aka daemons, and the status of each one. There are two phases for each daemon, which you see below.

 

> show management-clients

 

        Client PRI   State  Progress
-------------------------------------------------------------------------
        routed 30    P1-ok     99
      ha_agent 25    P1-ok     99
        device 20    P1-ok     99
        ikemgr 10    P1-ok     99
        keymgr 10    init       0 (op cmds only)
       logrcvr 10    P1-ok     99
         dhcpd 10    P1-ok     99
       varrcvr 10    P1-ok     99
         l3svc 10    P1-ok     99
        sslvpn 10    P1-ok     99
        rasmgr 10    P1-ok     99
       useridd 10    P1-sent   70 <------- Note: this is different
          satd 10    P1-ok     99
       websrvr 10    P1-ok     99
        sslmgr 10    P1-ok     99
         authd 10    P1-ok     99
        pppoed 10    P1-ok     99
     dnsproxyd 10    P1-ok     99
       cryptod 10    P1-ok     99
        dagger 10    init       0 (op cmds only)

 

Overall status: P1-sent. Progress: 0
Warnings:
Errors:

 

If something does not look correct, please ignore the "op cmds only."

 

Look above and see how "useridd" shows 70%, and "p1-sent." If it stayed at that for some time, then that might indicate an issue with the User-ID daemon. In that case, the commit may not complete.

 

You can resolve this by restarting that daemon with this command:

> debug software restart user-id

 

This command will ONLY restart the process that you want. In this case it is the User-ID daemon.

You can then initiate the commit process and it should no longer stop at that one process.
If there are no issues, then the output should look more like this:

 

> show management-clients

 

    Client PRI   State Progress
-------------------------------------------------------------------------
    routed 30    P2-ok   100
  ha_agent 25    P2-ok   100
    device 20    P2-ok   100
    ikemgr 10    P2-ok   100
    keymgr 10    init      0 (op cmds only)
   logrcvr 10    P2-ok   100
     dhcpd 10    P2-ok   100
   varrcvr 10    P2-ok   100
     l3svc 10    P2-ok   100
    sslvpn 10    P2-ok   100
    rasmgr 10    P2-ok   100
   useridd 10    P2-ok   100
      satd 10    P2-ok   100
   websrvr 10    P2-ok   100
    sslmgr 10    P2-ok   100
     authd 10    P2-ok   100
    pppoed 10    P2-ok   100
 dnsproxyd 10    P2-ok   100
   cryptod 10    P2-ok   100
    dagger 10     init     0 (op cmds only)

 

Overall status: P2-ok. Progress: 0
Warnings:
Errors:

 

Our very own community teammate "Reaper" (Tom Piens) recently wrote about reducing management load, which can help with commit times. You can read those articles here:

Tips & Tricks: Reducing Management Plane Load
Tips & Tricks: Reducing Management Plane Load—Part 2

I hope this helps you understand the commit process better and what you need to do if there is a specific process stopping the commit from completing properly.

 

Stay secure!
Joe Delio