On-Prem Analytics - How Data Retention is determined and how to increase it

11/14/2024 0 People found this article helpful 22 Views

Description

In Analytics 2.5.4 and above, you can check data retained by Analytics On-Prem from its user interface. Go to “Console->Flow Agent-> Devices” and look at value in DATA RETENTION (DAYS) column of device table. The days are calculated automatically using traditional method. To change the data retention days, an alertnate solution has been provided:

1. Log into On-Prem Analytics' Appliance page:

HTTPS://(IP)/appliance

2. change the URL to:

HTTPS://(IP)/appliance/techSupport.html

3. edit appflow.conf file by clicking on 'Edit', then clicking on 'Add' to add new parameter:

appflow_check_db_usage_before_purging

value = true

4. click on 'Update' to commit the change.

5. reboot Analytics server.

Afterwards, it will then display the retention days as "--"

From Jira ANA-66:

************

Current Data Retention Calculation

For each firewall sending flow records

1. Monitor the flow database flow record count.
2. Once current flow database reaches 250,000 (max flows per db), the database is closed
3. Calculate the number of days to retain. It is calculated according to this formula

a. Obtain the maximum allowable disk space to store the flow data (80% of actual disk size, leaving 20% as buffer other system related use)
b. Calculate the total number of days that can be potentially stored in the disk (maximum allowable disk space / max amount of flow data to insert per day)
NOTE: max amount of flow data to insert per day (gigsPerDay) is taken from the license
c. Then to calculate the number of days of data each firewall can store
i. first calculate the firewall count and it is based on the platform type (TZ firewalls has a weight of 1, Special FW has weight 2, NSa has weight of 4 and SuperMassive/Multiblade has weight of 8)
example: 3 TZs and 1 NSa, the total firewall count is (3*1) + (1*4) = 7
ii. Then take the value of totals days in (b) and divide it total firewall count in (c-i)

The result in (c-ii) gives allocated number of days to store/retain per firewall. It defaults to minimum 2 days to retain when the value derived from 3b is zero

4. Once it has gotten the number of days (X) to retain, the flow agent will traverse the flow databases and start purging data older then X number of days
5. Create new flow database and start inserting flow records

Proposed Solution for Data Retention

This alternate solution requires a diagnostic settings change
1. enable/disable the alternate purging method - default is false, set to true to use this method
2. number of files (N) to purge when max disk usage threshold is reached, default is 5

1. Monitor the flow database flow record count.
2. Once current flow database reaches 250,000 (max flows per db), the database is closed
3. The purging of data is as follows:
a. check if max disk usage threshold is reached
b. if (a) is breached, the system will check if there are enough N number of earliest flow database files
c. if (b) is has enough files, then it will delete the identified earliest files otherwise it will do no action and just continue to #4, at some point (in later iterations, see more explanation below) this firewall will have enough files to purge

Some important items to note here:
* each (closed) flow database having 250K flow entries take ~52MB of disk space
* given an example deployment of 10TB space, the flow agent takes 80% of storage space for writing data (max disk threshold), leaving 20% as buffer, in this example, 8TB max disk usage is determined
* In this example. the flow agent can store around 150K+ flow databases at 52MB/db file in the storage, it will take some time for the flow agent to hit max disk space, even if there are multiple firewalls.
* note that each firewall independently runs its own check when max disk space threshold is reached, step (c) above is a validation that a firewall has enough old database files to purge, this is to make sure not to delete all of its data including the current database. If there are not enough files to delete, it will not force the purge operation. As other firewalls will eventually run their own check and may have enough files to delete.
* also note that there is still that 20% disk space buffer, this allows some space for firewalls to go beyond the boundary while waiting for other firewalls to delete their db files to give space.

4. Create new flow database and start inserting flow records

NOTE: this solution is not perfect and may not be effective on some deployments where some firewalls are pushing flow data at high rate and some firewalls are pushing at slower rate, the file purging may not be even across firewalls due to purges at file level.

Not Finding Your Answers?

ASK THE COMMUNITY

Was This Article Helpful?

YES NO

On-Prem Analytics - How Data Retention is determined and how to increase it

Description

Related Articles

Categories

Not Finding Your Answers?

Was This Article Helpful?

Article Helpful Form

Article Not Helpful Form

Follow Us

Subscribe to newsletter

Stay In Touch