Wednesday, March 25, 2015

Loading Metadata: Improve Performance by using file compare utility

There are many reasons for an organisation to refresh metadata daily in Hyperion Planning and now a days you will generally have at least 1-2 hierarchies which could have 1000s of members. Not to mention if there are alternate hierarchies with shared members then outline load utility might take a while to delete and reload hierarchies.

There was a situation where my client had very tight schedule within which all the hierarchies needed to be reloaded and we were not able to fit that due to the source system’s variations in delivering the files.

So to resolve this issue and/or reduce the impact, it was decided to check every hierarchy whether there is any change compared to previous day’s file. If change exits then load it else skip and go to next. This was achieved by using Windows FileCompare utility and worked quiet well.

Following is the extract from the file which calls outline load utility to load hierarchies one after another.

LoadHierarchies.Bat (extract):
SET DimName=Account
SET InputFileName=%DimName%_dim.csv
SET LoadLogFile=FC_%DimName%_Load.log
CALL D:\script\Custom_FileCompare %InputFileName% %LoadLogFile%
if %CUSTOM_RESULT% EQU 0 goto :LOAD_Another

SET LoadLogFile=%LogFilePath%\%DimName%_Load.log
SET LoadExceptionFile=%LogFilePath%\%DimName%_Load.err

CALL Outlineload -f:%PasswordPath% /A:%App% /U:%UserName% /I:%DimFilePath%\%InputFileName% /D:%DimName% /L:%LoadLogFile% /X:%LoadExceptionFile%

:LOAD_Another
Other Code goes here...



In the following file “PreFilePath” is the path where previous day’s file(s) are placed while “DimFilePath” points to current or latest file which shall be loaded based on comparison.

Custom_FileCompare.bat:
SET CUSTOM_RESULT=0

SET CompFileName=%~1
SET FCLogFileName=%~2
SET File1=%PreFilePath%\%CompFileName%
SET File2=%DimFilePath%\%CompFileName%

FC %File1% %File2% > %LogFilePath%\%FCLogFileName%
SET CUSTOM_RESULT=%ERRORLEVEL%
if %CUSTOM_RESULT% EQU 0 ECHO Same files. Ignore this hierarchy load.