Net/FSE User Community Welcome to the Net/FSE user community at NetFSE.org. We will be expanding this forum as the open source version of Net/FSE moves forward. http://netfse.org/support/index.php Tue, 07 Sep 2010 08:00:12 -0500 Phorum 5.2.9a http://netfse.org/support/read.php?4,12,20#msg-20 Re: How to avoid filesystem from filling up http://netfse.org/support/read.php?4,12,20#msg-20 this thread for more information.]]> BenUphoff Installation Mon, 06 Sep 2010 14:12:57 -0500 http://netfse.org/support/read.php?4,19,19#msg-19 Managing Disk Space http://netfse.org/support/read.php?4,19,19#msg-19
AutoMigrate::max
The default setting here is 50, indicating that at most 50 PostgreSQL tables should be used to store data per data type. Each table contains TempTableSize records. These tables are not optimized for storage size and can end up eating up a lot of disk if either value is very high.

Consider reducing AutoMigrate::max to 25 to conserve disk. This will force the migration subsystem to kick in earlier and move data from PostgreSQL to flat file b+ trees, which can then be compressed as needed. There is a trade off however as results stored in the PostgreSQL tables are much faster to search than records in the flat file b+ tree indexing mechanism.

Note that this setting should be larger than AutoMigrate::min (default is 3), which tells the system to keep a minimum number of PostgreSQL tables per type.

AutoMigrate::deletedays (0.3 release only)
This setting is set to -1 by default, indicating that the feature is off. If set to a positive integer, the system will delete data that is older than the number of days specified. This in effect makes the system function as a ring buffer, deleting older data over time. Currently this function only works universally for all data types and cannot be customized, for instance to delete high volume data faster than low volume data.

AutoMigrate::days
This is set to 10 by default, indicating that the migration process should try to migrate data that is older than 10 days. This is a rather conservative setting and probably should not be reduced, as the performance benefits outweigh any small storage savings.]]> BenUphoff Installation Mon, 06 Sep 2010 14:11:38 -0500 http://netfse.org/support/read.php?3,18,18#msg-18 New log formats for 0.3 release http://netfse.org/support/read.php?3,18,18#msg-18
    [*] Snort
    [*] Juniper NetScreen 5400
    [*] Bluecoat web proxy
    [*] Squid web proxy

We have made data type integration a key focus of the latest release. Deploying new formats is easy and we are eager to help people integrate new data feeds into Net/FSE>]]> BenUphoff Data Type Support Mon, 06 Sep 2010 13:55:35 -0500 http://netfse.org/support/read.php?1,17,17#msg-17 Net/FSE 0.3 available http://netfse.org/support/read.php?1,17,17#msg-17 downloads page.]]> BenUphoff Announcements Mon, 06 Sep 2010 13:50:31 -0500 http://netfse.org/support/read.php?4,12,14#msg-14 Re: How to avoid filesystem from filling up http://netfse.org/support/read.php?4,12,14#msg-14
In some cases the disk may fill up due to the PostgreSQL database getting very large. There are parameters in /etc/netfse/0.2/local.cnf that can be changed to address this problem. Try changing TempTableCount to something small. That may allow the DiskManager to work properly.]]> BenUphoff Installation Tue, 27 Jul 2010 12:32:17 -0500 http://netfse.org/support/read.php?4,12,12#msg-12 How to avoid filesystem from filling up http://netfse.org/support/read.php?4,12,12#msg-12
Is there a configuration setting for Net/FSE that prevents from filesystem from being filled up?
I am looking for a ring buffer type of functionality.
If there is no way to do this, what can be done manually?

Thanks in advance.]]> mnejat Installation Tue, 23 Mar 2010 11:51:52 -0500 http://netfse.org/support/read.php?3,5,5#msg-5 New Data Type Requests http://netfse.org/support/read.php?3,5,5#msg-5 BenUphoff Data Type Support Wed, 13 May 2009 21:29:20 -0500 http://netfse.org/support/read.php?3,4,4#msg-4 IPFW Logs http://netfse.org/support/read.php?3,4,4#msg-4 /etc/nfse/local.cnf if it is not already there. The value 3 in the example below is a unique ID for the data type. It doesn't matter what the number is it just has to be unique and can never be changed.

Type::class nfse.dataset.firewall.IPFWRecord 3 IPFW]]> BenUphoff Data Type Support Wed, 13 May 2009 21:04:43 -0500 http://netfse.org/support/read.php?3,3,3#msg-3 IPTables Logs http://netfse.org/support/read.php?3,3,3#msg-3 /etc/nfse/local.cnf if it is not already there. The value 4 in the example below is a unique ID for the data type. It doesn't matter what the number is it just has to be unique and can never be changed.

Type::class nfse.dataset.firewall.IPTablesRecord 4 IPTables]]> BenUphoff Data Type Support Wed, 13 May 2009 20:59:34 -0500 http://netfse.org/support/read.php?3,2,2#msg-2 Apache2 Logs http://netfse.org/support/read.php?3,2,2#msg-2 Apache2 documentation for more information on Apache 2 logging. You will need to add a line like the one below to the end of /etc/nfse/local.cnf if it is not already there. The value 5 in the example below is a unique ID for the data type. It doesn't matter what the number is it just has to be unique and can never be changed.

Type::class nfse.dataset.apache.Apache2AccessRecord 5 Apache2]]> BenUphoff Data Type Support Wed, 13 May 2009 20:51:52 -0500