Validation process

Validation process consists of three component: (1) WSC, (2) ILM, (3) validation.

Validation data includes two parts of data: (1) policy files (each company/hurricane pair) for both wind speed calculation (WSC) and modeled loss calculation (ILM), (2) claim file (same pairs as the policy files) for modeled loss and actual loss comparison (validation).

The validation results will be used to complete Submission Form-S4, two tables in Standard S5 (refer to the submission document (page 184-187) at http://fphlm.cs.fiu.edu/files/wind_certification/v6.2Submission/FCHLPM/FPHLM_2015_Submission_Document_RevJan2017_Track_Changes.pdf)

All the codes related to the validation process can be found in the code directory:

/home/mitch-a/dmis-projects/fphlm/proc/2017_standards_v7.0/code/validation

The last submission’s runs and environments can be found in the validation_runs directory:

/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/validation_runs/

1.- Prepare the files and folders in the new submission’s folder (/home/mitch-a/dmis-projects/fphlm/proc/201X_standards_vX.X/validation_runs) (one copy for all members)

  • Autovalidation code

/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/validation_runs/autovalidation

  • Policy files (Exposures for each company/hurricane pair)

/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/validation_runs/policies_bak

  • Claim files (Actual losses for each company/hurricane pair)

/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/validation_runs/claim_file_bak

Note: Each person who processes a set of validation companies should create their own autovalidation running environment with specific config files

2.- List of policies you plan to run (Can be modified to rerun specific companies/hurricanes)

/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/validation_runs/policies.txt

3.- Update demand surge factors in the excel file Load_all_correct_V5.0_DS.xls using the new simulation_ds_all.txt

  • Excel files:

autovalidation/Data/Validation/[PLH]R/Results/Load_all_correct_V5.0_DS.xls

  • Update values can be found in (please make sure to use the updated file in the new submission folder):

/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/demand_surge_factors/catfundPR2007_v6.2_validation_ds/pr_unmatched/env/Results/simulation_ds_all.txt

[1-11].dat refer to the following hurricanes:

    • 1. Andrew
    • 2. Charley
    • 3. Dennis
    • 4. Erin
    • 5. Frances
    • 6. Ivan
    • 8. Jeanne
    • 9. Katrina
    • 10. Rita
    • 11. Wilma

Note: Data folder contains (1) all the excel files for validation post-processing result (2) WSC setups and (3) ILM setups for creating the running environment of each company/hurricane pair.

4.- Edit the configure file (/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/validation_runs/autovalidation/Configure/Config.xml)

Autovalidation integrate the codes for WSC, ILM and validation (post-processing)

  • <WSCCode>/home/mitch-a/dmis-projects/fphlm/proc/2017_standards_v7.0/code/validation/v6.0_wsc_environment</WSCCode>
  • <ILMPrecleanPR>/home/mitch-a/dmis-projects/fphlm/proc/2017_standards_v7.0/code/validation/v6.0_validation_env_PR/pre_clean_2008zips/</ILMPrecleanPR>
  • <ILMPrecleanHR>/home/mitch-a/dmis-projects/fphlm/proc/2017_standards_v7.0/code/validation/v6.0_validation_env_HR/pre_clean</ILMPrecleanHR>
  • <ILMPrecleanLR>/home/mitch-a/dmis-projects/fphlm/proc/2017_standards_v7.0/code/validation/v6.0_validation_env_LR/pre_clean</ILMPrecleanLR>
  • <VLDCode>/home/mitch-a/dmis-projects/fphlm/proc/2017_standards_v7.0/code/validation/VLDCode/code</VLDCode>
Keep all the paths not change, except:
UserDir: The directory you create for run validation
CollectionDir: The directory of policies
ClaimDir: The directory of claim files
SourceFile: The list of policy names
UserFileName: Same as the name of SourceFile
PRStart, PREnd, HRStart, HREnd, LRStart, LREnd: the starting and end line number in the source file
  • Update output directory with a new folder name: <UserDir>/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/validation_runs/[name of an empty folder]</UserDir>
  • Location of policy files: <CollectionDir>/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/validation_runs/policies</CollectionDir>
  • Location of claim files: <ClaimDir>/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/validation_runs/claim_file</ClaimDir>
  • List of policies need to be processed: <SourceFile>/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/validation_runs/policies.txt</SourceFile>
  • Specific the PR/HR/LR policies’ start and end line numbers in the policies.txt
    • <PRStart>1</PRStart>
    • <PREnd>73</PREnd>
    • <HRStart>86</HRStart>
    • <HREnd>95</HREnd>
    • <LRStart>74</LRStart>
    • <LREnd>85</LREnd>
    • should be ordered continuously for each policy type (PR policies, HR policies, and LR policies)

5.- Edit WSCSetup.xml in the same directory

  • Note: WSC code for validation is modified to use only two servers. 3 files are different from the regular WSC code.  (servers.txt, merge_multiserver.sh, wsc_multiserver.sh)
  • WSCRoot: <The directory you create for running validation>/WSC
    EmailToNotify: input your email
    check RoughTiles, DistanceFolder, SwathFiles for each submission
  • Same new folder name as specified in configure.xml and add ‘/WSC’ at the end: <WSCRoot>/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/validation_runs/[UserDir]/WSC</WSCRoot>
  • <RoughTiles>/home/mitch-a/dmis-projects/fphlm/proc/2011_standards_v5.0/met/20120702_filesSentByDrCocke/code.20120702/rough_popwt_2012</RoughTiles>
  • <DistanceFolder>/home/mitch-a/dmis-projects/fphlm/proc/2011_standards_v5.0/met/20120702_filesSentByDrCocke/code.20120702/distance_popwt_2012</DistanceFolder>
  • <SwathFiles>/home/mitch-a/dmis-projects/fphlm/proc/2013_standards_v6.0/met/swath_files/baseset_20120806_validation_storms_swaths</SwathFiles>

6.- Edit ILMSetup.xml in the same directory

  • EmailToNotify: input your email
    PRFlag, HRFlag, LRFlag: Set up the type of validation you want to run
    Matrices: Update matrices directory for each submission
    No Demand Surge applied during ILM
    PR policy flag set to 0
  • Match with the line index specified in configure (Which type of company you are going to process). The java program will provide the options for company type with 1 flag only.
    • <PRFLAG>1</PRFLAG>
    • <HRFLAG>1</HRFLAG>
    • <LRFLAG>1</LRFLAG>
  • Update matrices for PR, HR, and LR
    • <Matrices>/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/data/matrices_curves/v6.2_pr_matrices/</Matrices>

7.- type ./autovalid.sh

  • Select available option (E.g., process PR policies)
  • Start with manipulating the running environment.
  • Note: validation process needs python2
  • If the autovalidation program has no output for a long time after step 3 (ILM) has finished
    • terminate the program
    • go into the validation folder in your running environment, use ./run_validation.sh to generate the validation results
    • restart the autovalidation program:
      •  select 3. execute HR task (if you are running HR policies for example)
      • select 5. to continue generating the final results.
8.- Post processing

(1)use the result files from LR and HR to manually generate the CR combined result.
(2)use template to create form S4, validation results
/home/mitch-a/dmis-projects/fphlm/proc/2015_standards_v6.2/forms/s4
validation PR: update sheet1, Loss Data Complete, and Loss Data
validation CR: update both sheets
formS4 temp and plot:template for PR and CR plots using highlighted company files
update formS4 using corresponding data

Download PDF
Sidebar