Sizing Storage for the Logging Service
Palo Alto Networks Logging Service exists as a cloud-based storage mechanism for logs generated by the security platform. When purchasing Palo Alto Networks devices or services, log storage is an important consideration. Ensuring sufficient log retention not only enables operations by ensuring data is available to administrators for troubleshooting and incident response, but enables the full suite services provided by the Application Framework.
This document will cover storage sizing for the Logging Service. If you need guidance on sizing for traditional on-premise log collectors, see the following document:
When planning a log collection infrastructure, there are three main considerations that dictate how much storage needs to be provided. These are:
- Average size of a log
- Rate of log generation
- Desired retention period
With 8.0 all firewall logs (including Traffic, Threat, Url, etc) have an average size of 1500 bytes when stored in the logging service. This number may change as new features and log fields are introduced. When this happens, the attached tools will be updated to reflect the current status.
For firewall platforms, both physical and virtual, there are several methods for calculating log rate. These will be covered in more detail below. Sometimes it is not practical to directly measure or estimate what the log rate will be. Examples of these cases are when sizing for GlobalProtect Cloud Service. Each GPCS offering is sold based on different.
Determining Log Retention Requirements
There are several factors that drive log storage requirements. Most of these requirements are regulatory in nature. Customers may need to meet compliance requirements for HIPAA, PCI, or Sarbanes-Oxely:
There are other governmental and industry standards that may need to be considered. Additionally, some companies have internal requirements. For example: that a certain number of days worth of logs be maintained on the original management platform. Ensure that all of these requirements are addressed with the customer when designing a log storage solution.
*Note that some companies have maximum retention policies as well.*
Methods For Sizing
This document will cover storage sizing for the Logging Service using three different methodologies:
- Based on log rate: For any given customer, this will be the most accurate method.
- Based on throughput: Primarily used when sizing storage for GPCS Remote Office.
- Based on user count: Primarily used when sizing storage GPCS Mobile User.
Methods 2 & 3 can be used when sizing for firewalls as well when determining the log rate is impractical.
Sizing for NGFW
Determining Log Rate
- Leverage information from existing customer sources. Many customers have a third party logging solution in place such as Splunk, ArcSight, Qradar, etc. The number of logs sent from their existing firewall solution can pulled from those systems. When using this method, get a log count from the third party solution for a full day and divide by 86,400 (number of seconds in a day). Do this for several days to get an average. Be sure to include both business and non-business days as there is usually a large variance in log rate between the two.
- Use data from evaluation device. This information can provide a very useful starting point for sizing purposes and, with input from the customer, data can be extrapolated for other sites in the same design. This method has the advantage of yielding an average over several days. There are two scripts attached which can assist with gathering and calculating this information. The TS_LPS script can be run against a tech support file pulled from an evaluation device. This script will calculate the average connections per second (with 10 minute granularity), which can then be used to estimate the log rate. The Device_LPS script can be run while the evaluation device is in use and will pull the actual logging numbers based on the traffic that the evaluation device is seeing. To use these scripts, download the one to be used, unpack the zip file and reference the README.txt for instructions.
- If no information is available, use the Device Log Forwarding table above as reference point. This will be the least accurate method for any particular customer.
For existing customers, we can leverage data gathered from their existing firewalls and log collectors:
- To check the log rate of a single firewall, download the attached file named "device_lps.zip", unpack the zip file and reference the README.txt file for instructions. This package will query a single firewall over a specified period of time (you can choose how many samples) and give an average number of logs per second for that period. If the customer does not have a log collector, this process will need to be run against each firewall in the environment.
- If the customer has a log collector (or log collectors), download the attached file named "lc_lps.zip", unpack the zip file and reference the README.txt file for instructions This package will query the log collector MIB to take a sample of the incoming log rate over a specified period.
Calculating Required Storage
The equation to determine the storage requirements for a particular log type is:
In the case of the logging service, all log types have an average size of 1500 bytes. For example, if a customer has an aggregate log rate of 1500 logs per second and wants to keep 30 days of logs, the above formula would yield:
This total of 3.88 Terabytes represents the storage required for detailed logs. The default quotas for the Logging Service allocate 60% of available storage for detailed logs. Total storage required in this case (accounting for detailed, summary, and infrastructure logs) would be:
Because Logging Service is sold in 1TB increments, the customer in this case would need to purchase 7TB to cover the 30 day requirement.
Sizing for GlobalProtect Cloud Service Remote Office
GPCS Remote office is purchased based on throughput. In cases where the Remote Office service will be in use, it isn't practical to determine log rate. Additionally, there is value in a sizing methodology that mirrors how the service is purchased.
Sizing Based On Throughput
Due to the fact that we log on session end (by default), there is no direct causal relationship between throughput and log rate. For example, the following two sessions will generate one traffic log each even though the bytes transferred (and thus the throughput) differ greatly:
|Session Number||Application||Total Bytes||Number of Traffic Logs|
The relationship between log rate and throughput is reliant on the traffic mix seen by the firewall. In practice, enterprise traffic mixes containing traffic that is both user to server and user to internet have a ratio of 1.5 logs per second to 1 megabit per second. With the introduction of DPI logs in 8.1, this ratio climbs to 3.0 lps per 1 Mbps throughput.
This ratio can be used to estimate the Logging Service storage requirements for GlobalProtect Cloud Service Remote Office deployments based on the throughput allocated to the site.
Sizing for GlobalProtect Cloud Service Mobile User
Sizing Based On User Count
Like GPCS Remote Office, log storage sizing for GPCS Mobile User is not based strictly on log rate, but on a measured average. In 8.1, customers can expect to see about 30Mb of logs per day, per user.
Sizing for Traps Endpoint Security Cloud Services
Traps logs consist of Traps agents initiated events such as security events and operational logs from protected endpoints, and Traps management service events- such as policy configuration, audit logs, and service operational events. Traps logs are divided into 4 record types:
- Threat- security events, as reported by both Traps Agents and Traps Management Service
- Config- Traps Management Service’s configuration audit trail such as policy editing and local configuration changes.
- System- operational events from both Traps Agents and Traps Management Service, such as service availability, licenses, and Agents drivers’ operations.
- Analytics- information regarding executed PEs and Office files from protected machines
The Traps management service sends all Traps logs to the Logging Service, and displays the logs in the Traps Management Service user interface. These logs are the establishment of all Traps functionality- from policy and environment management to security events investigations.
Traps Storage Quota Allocation Calculator
The quota allocation calculator for Traps reflects the estimated size of a given deployment scenario, as derived from the number of Traps agents and the allocated storage size.
In the attached calculator, you insert the allocated storage size (Allocated Storage (GB)) and number of Traps agents (# of Agents), and observe the storage duration before the quota will be filled, and new events will start overriding old events.
As the estimated storage duration is divided per record type, you can also change the percentage of storage allocated for each record type to match your organization’s requirement for logs storage.
Note: Maximal duration of logs storage is 2,000 days.
How To Use the Calculator
The 'Options' section allows you to select whether DPI logs will be enabled and an optional 'Future Growth' field for budgetary purposes.
Enabling the checkbox for Magnifier will change the calculations to account for the additional logs generated when DPI is enabled.
The 'future growth' field allows you to enter (as a percentage) a modifier that will be applied to the calculated total. This field can be used to:
- Account for future projects/acquisitions which will increase aggregate number of logs generated
- Account for future PANW services that may not be avialable at the time Logging Service is purchased
- Account for error in measurement of log rate (in the case of NGFW) or if the customer use case is outside of a normal enterprise traffic mix (in the case of bandwidth).
There are two options here, depending on whether storage for Magnifier is enabled. The first option is the traditional log rate based calculator for firewalls. The only required input is the log rate and desired retention (in days):
A second option is available when the checkbox for Magnifier is enabled. This option will allow you to estimate storage based on the number of hosts on the network. This option estimates the required storage based on a per-host value. This per host value is based on an average of samples collected accross different environments:
NOTE: These two options can be used concurrently if you have, for example, a log rate for one or more sites but need to estimate based on hosts for other sites.
GPCS Remote Office
The GPCS Remote Office section will allow you to size based on bandwidth:
This option requires more data to provide an accurate number. GPCS Remote office is sold according to throughput. When, for example, 100Mbps is purchased and allocated to a location, it's not likely that the link will see 100% utilization all of the time. In addition to entering the throughput purchased, the calculator allows you to fine tune what the utilization of the link will be during production and non-production hours.
Exercise: You purchase 10 Mbps for a remote sales office. 10Mbps is needed as the sales personnel regularly upload large documents, but for most of the day there isn't very much traffic on the network. In this case, you can define the number of busy hours to '8' to reflect the businesss hours for the office. During the busy hours you may expect an average utilization of 35%, and off-hours utilization of 5% (to account for non-user traffic like printers, security systems, etc).
Fine tuning these variables allows you to dial in a realistic amount of storage for that office.
GPCS Mobile User
The GPCS Mobile User section will allow you to determine how much storage you need for your mobile users.
Putting It Together
For a traditional NGFW deployment, log rate will still yield the most accurate numbers for log storage. In cases where measuring or estimating the log rate isn't practical, you can size based on bandwidth using the GPCS Remote Office section.
The attached scripts (described under "Sizing for NGFW" above) will assist with gathering data for sizing a Next Generation firewall.