1.
Finit Lumberjack
July 2018
Frank Zidar, Finit Solutions

2.
OUR HISTORY
A brief history of our partnerships
Oracle Gold Partner Finit
• Founded in 2002
• 90+ employees in 20+ states
• Mostly CPAs, MBAs
• 300+ Clients – all CPM / EPM
Focused
• 1,000+ CPM / EPM projects
• 20+ Fortune 100, 60+
Fortune 500, 100+ Fortune
1000
OneStream Platinum Partner
• 150+ HFM Projects
• Managed largest roll-out of
HFM in North America
• 175+ FDM Projects
• World leader in FDM
implementations and
custom solutions
• 100+ Planning / Essbase
projects
• 40+ OneStream Clients
(Market Leader)
• #1 Implementer of
OneStream
• Platinum Partner
• First OneStream
MarketPlace App
• RequestIT

3.
The Finit Model
Fully aligned with
our clients, not just
bottom line $
No debt or external
ownership
Work with Finit
employees, no
subcontractors
Compensation
based on CLIENT
SATISFACTION
PRIVATELY
OWNED
CLIENT
SATISFACTION
NO
SUBCONTRACTORS
DEBT FREE

4.
Our values, culture, and approach to
becoming a trusted advisor to our
customers has led to
100% CUSTOMER SUCCESS
for every project since our inception
in 2002
300+ Clients
1,000+ Projects

5.
Some Finit Clients

6.
Questions

7.
Agenda
• Why did we build this tool and what is it good for?
• Why is this tool different than other Finit Project Tools
• How does it work and how long to install and bring online?
• Common questions it helps solve and other use case detail
• How much does it cost?

8.
About the Presenter
Frank Zidar
(fzidar@finit.com)
Experience
• Director of Infrastructure
• 7+ years at Finit
• Over 50 Oracle EPM installations
• Over 20 years experience with all aspects of
the software development lifecycle
Certification/Education
Bachelor of Science in Computer Science and
Mathematics, Bowling Green State University
Professional Summary
Frank lives in Cleveland, OH and has been part of Finit since 2011 and is focused on the
installation, migration, and server architecture of Oracle EPM systems. Frank has provided
these services and been involved with over 50 installations during the last 5 years and is the
Director of the Infrastructure group at Finit.

9.
Why did Finit build Lumberjack?

10.
Why did Finit build Lumberjack?
• EPM is complex and Finit infrastructure gets involved with a very common set
of questions and support situations
• Challenge:
• Most problems occur during the close and IT, in interest of time, must simply reboot
• Often very challenging (especially with performance issues) to piece back together a root cause
long after the fact
• Performance must be captured continuously
• Log files roll over and are automatically removed
• In the past we have traveled to the client site or worked over VPN to actively monitor the
system during a close cycle

11.
Why did Finit build Lumberjack?
• Tried other utilities in the past with limited success
• Time consuming and complex to setup as well as expensive
• Started as an internal project and has grown over time
• Goals of the system:
• Quick and easy setup
• No additional processes on EPM servers
• Relies on technology built into the operating system as much as possible

12.
How Does It Work?
• All that is required is Microsoft .NET Framework 4.5.2 (built into Windows OS)
• No installation - unzip several standard Windows executable files
• Add configurations for each of the environments (PROD, DEV, etc.)
• Add the list of servers into each configuration
• Lumberjack will automatically determine which EPM services are installed on each server
and where the logs are located

13.
How Does It Work?
• Setup the service to execute in unattended mode every 5 minutes
(configurable)
• Normally installed on a non-EPM server/computer in the datacenter to avoid network
latency and stealing CPU cycles from EPM servers
• The Lumberjack service will process all of the information and send it to a central server
located in Finit’s datacenter
• This allows us to be able to provide support without VPN and/or meetings
• Entire setup process can be completed in 15-30 minutes
• Use the Lumberjack viewing application to see the captured information from
any computer

14.
How Does it Help?

15.
EPM Log Files
• Log files!
• EPM creates tons of log files
• Many servers, many directories
• C:OracleMiddlewareuser_projects
epmsystem1_HFMFDMdiagnosticslogshfm
• C:OracleMiddlewareuser_projectsdomains
EPMSystemserversErpIntegrator0logs
• Where are they all located?
• Which logs are important?
• What should I look for?
• What does this stuff mean?

16.
EPM Log Files
• Lumberjack filters out the unimportant logs
• Can configure to gather everything or only errors
• Centralizes everything into one place and displays in Excel-like
format
• Ability to drill into the details of the message

17.
EPM Log Files
• There are many ways to filter and sort the data (Excel-like features)
• Text searching (all servers, all logs):

18.
Performance
• What is causing our consolidations to slow?
• Is the system optimized for best performance?
• We have many times where we get the “spinning wheel of death”, why and
can it be prevented?
• Performance has significantly degraded recently, why?
• In order to provide answers to these types of questions we gather from the
server(s) the following types of information:

19.
Performance
• Total CPU performance of HFM/FDMEE server for a several week period

20.
Performance
• Comparison of HFM app server and the HFM and FDMEE web processes

21.
Performance
• Total system memory vs. HFM app process

22.
Performance
• Ability to view consolidation performance over time
• This list can be used to see problems in your consolidations and learn more
about them

23.
Performance
• Consolidations grouped by: Entity, Type, Period shows something interesting:
• Drilling down we find a situation where the same consolidation executed on
the same day has very different execution times:

24.
Performance
• Performance of the system during the first consolidation:
• Drilling down into the details:

25.
Performance
• Further analysis shows system available memory at that time is a potential
culprit:

26.
Performance
• Comparison to the consolidation which was executed at 11pm:
• No 100% spike in CPU
• Much more system memory
available

27.
Essbase Performance
• Display a listing of all the logins to the system

28.
Essbase Performance
• Ability to display the Smart View retrieves across all cubes (similar filtering and
sorting options are available):

29.
Essbase Performance
• Display a listing of all calculations
• Very easy to find slow running calcs or ones which have degraded over time.

30.
Other Details
• HFM Database Analysis

31.
Other Details
• WebLogic Server Analysis

32.
Crash Analysis
• Armed with all of the log and performance data, someone can determine why
problems have happened after the fact.
• Provide root cause for questions like:
• The system is currently down – nobody can connect to HFM (spinning wheel) and some
users aren’t even able to log in.
• The system has been unstable recently. What is the cause?

33.
Crash Analysis
• We are monitoring 3 systems for a client and the morning of 6/26 it was
reported to Finit infrastructure that a significant issue occurred in the
development EPM environment the prior evening
• Around 7pm the system became unusable. No one able to log in and anyone in the system
was unable to open forms and HFM applications.
• 2 server configuration
• Server performance
showed nothing
unusual overall:

34.
Crash Analysis
• Log analysis for the same timeframe displayed normal and expected system behavior until
6:46pm when the web server started reporting errors:

35.
Crash Analysis
• Continued searching shows this classic error starting at 7:52pm:
• At this point, FDMEE would be unusable and would have been so for at least 20 minutes

36.
Crash Analysis
• Continuing to search for the root cause:
• Ultimately, we learned the issue was with the database running out of
transaction log space and suffering performance issues leading to the full EPM
crash and restart.

37.
Crash Analysis
• We have performed similar analysis at other clients and ended up
recommending tuning changes to dramatically increase HFM consolidation
performance
• Another time we discovered that I.T. had reduced the amount of RAM available
to the Essbase server VM leading to degraded performance

38.
Financial Report History
• What Financial Reports do we actually utilize?
• Can be used to show if report execution is slowing over time

39.
System Information
• Easy to see all of the EPM patches installed on the system

40.
Alerts
• While monitoring the EPM environment, Lumberjack can be configured to
send out alerts
• The messages can be easily configured to go to multiple addresses or internal
distribution lists.

41.
Stress Testing
• Will the system handle the expected concurrent users?
• Will the system perform well if we add additional users?
• Easy to record user actions in the system
• Once recorded, they are applied to a stress test to be played back to simulate
stress on the system
• All performance and log data is captured during playback/testing
• Additional testing can be performed with Python-based scripting utilities
• FDM data loads, multiple concurrent HFM consolidations

42.
Customized Scripting
• We built a layer into Lumberjack which exposes the HFM programmer
interface (API) in a different way
• Oracle provides a Java based interface and we have created an extra layer accessible via
Python scripting
• Python is very popular and also the foundation of the scripting used in FDMEE (Jython)
• Many easy to use extensions to perform anything imaginable
• Oracle completely changed the HFM API from 11.1.2.3 to 11.1.2.4
• The added layer allows scripts built for 11.1.2.3 to be reused without modification in 11.1.2.4
and potentially beyond
• Currently analyzing the potential for use with Oracle’s cloud solutions

43.
Customized Scripting
• FDM
• Import data file to a specified location in FDM
• Validate and export the mapped data
• Example: We used this at a client to perform FDM stress testing to simulate the processing
of 75 concurrent end users loading multiple GL files

44.
Customized Scripting
• HFM
• Execute calculations, consolidations, and on demand rules
• Process management
• Automate promotion process for all phases
• Extract and load data files
• Execute Extended Analytics exports and work with the resulting data
• Easier to work with and more powerful than Task Flows
• Example: Lights out automation of HFM data extracts for several scenarios with a 5-day
rolling backup maintenance
• Example: Data “recast” process where data extracted from HFM according to values
entered on an Excel sheet. Converted a CUSTOM field from one value to another and
reload the data back to HFM and kick off a consolidation

45.
Customized Scripting
• LCM
• Script commands to run LCM exports
• Normally done with DOS batch files which are difficult to work with
• Common problem of long file names with Financial Reporting – Lumberjack can delete
these folders without issue and with a single command
• EPM patch installation
• Single script to analyze the environment and update every component to the latest Oracle
EPM patch
• Takes a very time consuming and tedious process from hours to minutes

46.
Cost
• The Finit infrastructure and integration teams have been using these tools for
several years and they have recently been packaged into a suite of software
products
• $12,000/year which includes:
• Initial setup and configuration
• Ongoing support and upgrades
• Performance monitoring and health alerts
• Monthly reports
• System performance charts
• Memory utilization charts
• Disk space utilization
• HFM consolidations executed with average execution times
• Financial report execution history with counts and average execution times
• Essbase calculation count and average execution times
• Significant error and restart counts
• HFM database error, task, and audit log counts
• Oracle EPM patch recommendations (as needed)
• Additional system tuning recommendations (as needed)

47.
Questions?

48.
Upcoming Webinars
• Friday, August 17– Cash Flow 2D vs 3D – Part I
• Presented by Matt Spencer
• Friday, August 24– Cash Flow 2D vs 3D – Part II
• Presented by Matt Spencer

49.
Thank you for Attending!
Today’s Presenter:
Frank Zidar
fzidar@finit.com
Greg Barrett
gbarrett@finit.com
Copy of the slides or Recording:
Email us for a copy of the slides or a
link to the recording
insights@finit.com
Past webinars:
www.finit.com/webinars
Follow us on Twitter for updates:
@Finit_Solutions