Disk Purge Batch
==========

Disk Purge Utility is designed to clean up old data collection folders from TrackIQ Analysis software.
The TrackIQ Analysis software creates folders in the structure; \trains\YYYY\MMDD\HHMMSS
This structure is common to both the WPU & Database Server, therefore this Utility can be configured to run on both.
On the WPU the pointer files are also santitised. (file are checked if destination exists).

Installation
============
Disk Purge Batch program runs as a Scheduled Task.

A Batch file "CreateScheduledTask" Creates the Scheduled Task under the system account.
The default configuration will launch Disk Purge Batch to run every 3 minutes (if it is not already running).

The default behaviour of Task Scheduler is to "kill" programs that have been running for longer than 3 days, 
this can be switched off but will not cause issue if the default remains. (As the task is configured to run every 5 mins).


Configuration
=============

In the Disk Purge Batch "DiskPurgeBatch.bat" file the following paramaters are used;
:: CONFIGURATION AREA

set DIR_TO_MONITOR=d:\server
set DISK_SPACE_IN_MEGS=4096
set CLEANUP_PERIOD_IN_MINUTES=3

IF NOT DEFINED DPOutputLogFiles SET DPOutputLogFiles=28

:: set AUXCLEAN=Do

:: set SERVER_MONTHS_TO_KEEP=12


:: END CONFIGURATION AREA

The "DIR_TO_MONITOR" is the root folder for the train storage (default for WPU) if used on a server, this
is the base folder of RailBAM data files.

The "DISK_SPACE_IN_MEGS" is the minimum free space required. (4096MB is default)

The "CLEANUP_PERIOD_IN_MINUTES" is the wait period between hard drive space checks. (3 minutes is default)

The "DPOutputLogFiles" is the number of Daily Disk Purge Output Log files that is desired to keep. (28 is default)
The longer period is useful for Trubbleshooting as only one day is required for Monitoring.

The "AUXCLEAN" parameter is specified to run the Auxiliary clean up to occur at first run regardless of DISK_SPACE_IN_MEGS. 
This is useful for an old site where Auxiliary files have not been cleaned up for a long time.

The "SERVER_MONTHS_TO_KEEP" is a configuration parameter for use on servers which will not allow data less than 12 months old to be removed, 
hence it will still be possible to fill up a drive if we

The "WPU_RAW_DATA_DAYS_TO_KEEP" is how many days the WPU will keep the RAW data files for. 
The period beyond WPU_RAW_DATA_DAYS_TO_KEEP will be cleared always once per day. (RailBAM, WCM2 and RailCAM system RAW file types).



Trubbleshooting / Monitoring
============================

The daily log file in sub folder "log" and named DiskPurge.<DATE>.log contains all outputs of this script.

This file can be used to monitor Disk Purge Batch.

Alternativly  the scheduled task can be disabled and the "DiskPurgeBatch" batch file can be launched and you'll 
be able to see the current output in a console window.

Both launch methods will still produce the log file.

If upgrading from an older Clean Up utility, sometimes the trainlist folder has not been maintained, 
run "TrainList_Cleanup.bat" Once to maintain this folder.

Detailed Methods of Functions
============================
Main loop
      Check free space, if less than specified run the clean up otherwise maintain the log files (delete older files) then wait defined period.
      
Clean Up
      Check folder structure, if exist <DIR TO MONITOR>"trains" then use WPU Cleanup, otherwise use Server Cleanup
      
     Server
          Find oldest folder for each subfolder that also has a trains sub folder. i.e. <DIR TO MONITOR><SITE NAME>"trains", then delete. 
          If delete fails, mark folder (archive attribute) to be excluded next time.
     WPU
          Find oldest train folder, then delete. 
          If delete fails, mark folder (archive attribute) to be excluded next time.
          Matching Trainslist folder file is also removed.
          Enable Auxiliary file Cleanup once free space is reached.
          
          Auxiliary File Clean Up
              for all the pointers, check if matching train folder exists, if not delete. 
              (Pointers are only links to files in train folders and they provide no useful information on their own.)
