ESM workers configuration in multi ESM environment

{This KB applies to Traps version 3.3.1/2/3} 

 

Summary:

Traps agents are sending reports to the ESM server which deliver the information to “request buffer” and “hush buffer” tables in the system DB.
The ESM server will process the buffers once he’ll be available for it by using “workers”.

By default each ESM server have 3 workers, this might cause SQL server performance specially in environments that use more then 2 ESM servers.
In order to prevent SQL server load we’ll have to manually configrure the amount of workers on each ESM server.

Following table descirbe PaloAlto networks best practice information:

EMS Server amount	Worker amount
1 or 2 ESMs	the default should remain 3
3 ESMs	2 workers per ESM
4 ESMs and up	1 worker per ESM

 

Expected issues:

When workers amount are greater then the best practice recommandation:

• Very large HashBuffers\RequestBuffer table
• Agent may loose connection to the ESM server
• ESM service may become unresponsive - could not be restarted or stopped
• ESM server log file locate: c:\ProgramData\Cyvera\Logs shows:

ERROR    CyveraServer     11         Cyvera.Server.Facades.ScheduledTasks.ProcessRequestBuffer   General  "GetWildFireAnalysis request  3428790  from machine: CSS-RGAC-W7 failed. Will reattempt later NHibernate.Exceptions.GenericADOException: could not execute batch command.[SQL: SQL not available] ---> System.Data.SqlClient.SqlException: The INSERT statement conflicted with the FOREIGN KEY constraint ""ProcessHashes_ClienteProcess_Id"". The conflict occurred in database ""palotraps"", table ""dbo.ProcessHashes"", column 'Id'.

Violation of PRIMARY KEY constraint 'PK__ClientPr__D8D41F8AF1609081'. Cannot insert duplicate key in object 'dbo.ClientProcesses'. The duplicate key value is (17, 3989).

Violation of PRIMARY KEY constraint 'PK__ClientPr__D8D41F8AF1609081'. Cannot insert duplicate key in object 'dbo.ClientProcesses'. The duplicate key value is (64897, 3989).

Violation of PRIMARY KEY constraint 'PK__ClientPr__D8D41F8AF1609081'. Cannot insert duplicate key in object 'dbo.ClientProcesses'. The duplicate key value is (5, 3989).

Violation of PRIMARY KEY constraint 'PK__ClientPr__D8D41F8AF1609081'. Cannot insert duplicate key in object 'dbo.ClientProcesses'. The duplicate key value is (1, 4110).

Violation of PRIMARY KEY constraint 'PK__ClientPr__D8D41F8AF1609081'. Cannot insert duplicate key in object 'dbo.ClientProcesses'. The duplicate key value is (298, 4074).

Violation of PRIMARY KEY constraint 'PK__ClientPr__D8D41F8AF1609081'. Cannot insert duplicate key in object 'dbo.ClientProcesses'. The duplicate key value is (38, 3656).

Violation of PRIMARY KEY constraint 'PK__ClientPr__D8D41F8AF1609081'. Cannot insert duplicate key in object 'dbo.ClientProcesses'. The duplicate key value is (950, 4074).

The statement has been terminated.

The statement has been terminated.

The statement has been terminated.

The statement has been terminated.

The statement has been terminated.

The statement has been terminated.

The statement has been terminated.

The statement has been terminated.

   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)

   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)

   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)

   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)

   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds, Boolean describeParameterEncryptionRequest)

   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)

   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)

   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()

   at System.Data.SqlClient.SqlCommandSet.ExecuteNonQuery()

   at NHibernate.AdoNet.SqlClientBatchingBatcher.DoExecuteBatch(IDbCommand ps)

   --- End of inner exception stack trace ---

   at NHibernate.AdoNet.SqlClientBatchingBatcher.DoExecuteBatch(IDbCommand ps)

   at NHibernate.AdoNet.AbstractBatcher.ExecuteBatchWithTiming(IDbCommand ps)

   at NHibernate.AdoNet.SqlClientBatchingBatcher.AddToBatch(IExpectation expectation)

   at NHibernate.Persister.Entity.AbstractEntityPersister.Update(Object id, Object[] fields, Object[] oldFields, Object rowId, Boolean[] includeProperty, Int32 j, Object oldVersion, Object obj, SqlCommandInfo sql, ISessionImplementor session)

   at NHibernate.Persister.Entity.AbstractEntityPersister.UpdateOrInsert(Object id, Object[] fields, Object[] oldFields, Object rowId, Boolean[] includeProperty, Int32 j, Object oldVersion, Object obj, SqlCommandInfo sql, ISessionImplementor session)

   at NHibernate.Persister.Entity.AbstractEntityPersister.Update(Object id, Object[] fields, Int32[] dirtyFields, Boolean hasDirtyCollection, Object[] oldFields, Object oldVersion, Object obj, Object rowId, ISessionImplementor session)

   at NHibernate.Action.EntityUpdateAction.Execute()

   at NHibernate.Engine.ActionQueue.Execute(IExecutable executable)

   at NHibernate.Engine.ActionQueue.ExecuteActions(IList list)

   at NHibernate.Engine.ActionQueue.ExecuteActions()

   at NHibernate.Event.Default.AbstractFlushingEventListener.PerformExecutions(IEventSource session)

   at NHibernate.Event.Default.DefaultFlushEventListener.OnFlush(FlushEvent event)

   at NHibernate.Impl.SessionImpl.Flush()

   at Cyvera.Server.Facades.ScheduledTasks.ProcessRequestBuffer.DeleteBufferAndClearSession(IUnitOfWork unitOfWork, RequestBufferEntity requestBuffer)

   at Cyvera.Server.Facades.ScheduledTasks.ProcessRequestBuffer.ProcessSingle(IServiceProviderProxy serviceProvider, IUnitOfWork unitOfWork, RequestBufferEntity requestBuffer)"

 

Step by step ESM server worker configuration:

1. Log in to each  ESM server 
2. Navigate to the following path in the server file explorer:

C:\Program Files\Palo Alto Networks\Endpoint Security Manager\Server

3. Open “CyveraServer.exe.config” using notepad (or another text edit tool)
4. Add the following property under “AppSettings”
   <add key="RequestBufferIntervalWorkers" value="number of workers" />

Example: