One of the many challenges to working as a Hyperion Admin is diagnosing issues. Many factors complicate troubleshooting efforts, such as the number of users, geographic location of users, varying experience levels, products in use (Hyperion and third -party), cryptic error messages, and the lack of built-in diagnostic tools. With the use of the proper tools and troubleshooting techniques, administrators can reduce the time required to handle many troubleshooting issues. This powerpoint presentation will demonstrate how to use a variety of third-party tools to assist troubleshooting efforts. Troubleshooting techniques will be discussed and demonstrated using real world issues. Finally, the presentation walks through solving a fairly tricky technical problem.

1. #Kscope
Case of the Unexplained Support Issue
Charles Beyer
Troubleshooting steps for diagnosing HFM, Workspace, and FDM

2. #Kscope
About the Speaker
 15+ years professional development
experience in various languages/technologies
(6 years w/ Hyperion products)
 Significant experience reverse engineering
software
 Loves to help others solve problems
 Posts under beyerch2 on OTN Discussion
Boards

3. #Kscope
Before we Begin…..
 To expedite session, we will use both screen
caps and live demos
 ‘All-in-one’ Virtual Machine created for session,
available upon request*
 Feel free to ask questions

4. #Kscope
Session Goals
 Discover some new tools to be used for
troubleshooting (Oracle / Third Party)
 Acquire some new troubleshooting ideas

5. #Kscope
Challenges
 Responsible for a large amount of products!
● Foundation Services (OHS, Weblogic, Shared Services,
Workspace, EPMA, Calc Manager, LCM)
● HFM (HFM Web, Client, Smartview), FDM
● Reporting and Analysis (FR, IR, SQLPR, WA)
● Planning, Performance Scorecard, Strategic
Finance, Probability Cost, Disclosure Management,
Financial Close Management, Data Relationship
Management, Essbase, Crystal Ball, ……..
Hyperion Kitchen Sink.

6. #Kscope
Challenges (continued)
 Need to understand underlying technologies!
● OS Servers (Windows/Unix)
● Application Servers (Weblogic, Tomcat), Web Servers
(IIS, Apache, OHS)
● Authentication Options (AD, Native, Custom)
● Languages (Java, .NET, ASP Classic, JSP, VBScript)
● Communication Frameworks (DCOM, Web Services)
● Database (SQL Server, Oracle, Berkeley DB)
● Network Principles (Routing, Firewall, VPN)

7. #Kscope
Challenges (continued)
 End Users!
● Quantity – Intermittent issues due to number of users
stressing system
● Internal vs. External
● Less control over environment
● More points of failure
● Remote troubleshooting more difficult
● User Experience
● New Users - More hands on, but easier problems.
● Power Users – Likes to experiment

8. #Kscope
Challenges (continued)
 Internal Issues a.k.a. Politics!
● Separation of Duties
● Complicates troubleshooting efforts due to lack of access
● Delays resolution due to co-ordination efforts
● Restrictive Policies
● Could prevent/limit certain types of troubleshooting
● Network/HTTP sniffing
● No monitor programs on production

9. #Kscope
Challenges (continued)
 Cryptic Error Messages!
But I didn’t click on a folder!

10. #Kscope
Challenges (continued)
 Cryptic Error Messages!

11. #Kscope
Challenges (continued)
 Cryptic Error Messages!

12. #Kscope
Challenges (continued)
 Cryptic Error Messages!

13. #Kscope
Challenges (continued)
 Cryptic Error Messages!

14. #Kscope
Challenges (continued)
 Cryptic Error Messages!

15. #Kscope
Challenges (continued)
 Limited OOTB (Out Of The Box) Tools!
● Mostly consists of Log Files
● System 11 has expanded diagnostic capability
● Support.Oracle.Com

16. #Kscope
Troubleshooting Strategy

17. #Kscope
Troubleshooting Strategy
Final Solution
Follow
Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem

18. #Kscope
Troubleshooting Strategy (continued)
 Clearly Define the Problem
● Ask the right questions
● Verify Information Where Possible
 Research, Prioritize, and Evaluate potential
solutions
● Use Initial Information and Resources to propose
possible causes/solutions
● Prioritize in order of likelihood
● Evaluate Potential Solutions
● Determine Final Solution or Escalate

19. #Kscope
Troubleshooting Strategy (continued)
 Close Out The Problem
● Document / Get Sign-Off
● Add Information To Internal Knowledgebase

20. #Kscope
Troubleshooting Strategy (continued)
 Notes
● Process Is Easier As You Learn More About The
System
● Development / Test Environment Makes This Easier
● Don’t Fear Leveraging Other Resources
● Practice, Practice, Practice

21. #Kscope
Troubleshooting Strategy (continued)
 Clearly Define The Problem – Answer the Who /
What / When / Where / How
● Who? (is affected)
● All Users or Select Group?
● If subset, is there commonality? (Geographic,
Hardware/Software, Data Provider)
● What ? (is happening / not happening)
● What was the attempted procedure?
● What has changed?

22. #Kscope
Troubleshooting Strategy (continued)
 Clearly Define The Problem – Answer the Who /
What / When / Where / How
● When? (does the issue occur)
● Consistently? Intermittently? Pattern?
● Where ? (are failures occurring)
● Client Machine(s)?
● Database Server(s)?
● Web Server(s)?
● App Server(s)?

23. #Kscope
Troubleshooting Strategy (continued)
 Clearly Define The Problem – Answer the Who /
What / When / Where / How
● How? (should the process work)
● Do you know how it should work?
● Important to understand expected operation to help
determine point of failure, probable cause, and solution.
● Research this if you are not sure!

24. #Kscope
Troubleshooting Strategy (continued)
 Research, Prioritize, and Evaluate potential
solutions
● Iterative Process
● Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools =
Potential Solutions
● Prioritize Potential Solutions
● KISS / Occam’s Razor – Start with most likely / obvious
first!

25. #Kscope
Troubleshooting Strategy (continued)
 Research, Prioritize, and Evaluate potential
solutions
● Evaluate Potential Solutions
● In Test/Dev if possible
● Promote to production after verifying your fix doesn’t cause
additional problems
● If you exhaust all possible causes, escalate. Don’t spin
wheels.
● Do not discount workarounds!

26. #Kscope
Troubleshooting Strategy (continued)
 Close Out The Problem
● Get Positive Confirmation That Issue Is Resolved
● Add Issue to Internal Knowledgebase For Future
Review
● SharePoint
● Track-It
● Shared Outlook Folder
● Share Information When Necessary

27. #Kscope
Internal Tools

28. #Kscope
Troubleshooting Tools - Internal
 Knowledge Tools
● http://support.oracle.com – Knowledgebase, Patches &
Update, and “Communities”

29. #Kscope
Troubleshooting Tools – Internal (cont)
 Knowledge Tools
● OTN Discussion Forum (http://forums.oracle.com)

30. #Kscope
Troubleshooting Tools – Internal (cont)
 Knowledge Tools
● Product Documentation
● System 11.1.2.1 - http://docs.oracle.com/cd/E24674_01/index.htm
● System 9.3.x - http://docs.oracle.com/cd/E10530_01/welcome.html
 NOTE : These move periodically! Google
‘Hyperion documentation library’ if you cannot
find what you are looking for

31. #Kscope
Troubleshooting Tools - Internal

32. #Kscope
Troubleshooting Tools – Internal (cont.)
 Knowledge Tools
● Tips and Tricks from EPM System Infrastructure
Development
● Issues 1 to 72 -
http://www.oracle.com/technetwork/middleware/bi-
foundation/epm-tips-issues-1-72-427329.pdf
● Issues 73 and up -
http://www.oracle.com/technetwork/middleware/bi-
foundation/epm-tips-issues-73-up-399995.pdf
● NOTE: Constantly updating, so there will be more, etc.

33. #Kscope
Troubleshooting Tools – Internal (cont.)

34. #Kscope
Troubleshooting Tools – Internal (cont.)
 Data Gathering
● Log Files (see Table)
● HFMErrorLookup.exe – Resolves Error Codes to Text
Description
● System 11- <EPM ORACLE
HOME>EPMSystem11R1productsFinancialManagementConsultant
Utilities
● System 9 - <HYPERION HOME>FinancialManagementConsultant
Utilities

35. #Kscope
Troubleshooting Tools – Internal (cont.)
 Data Gathering
● HFMErrorLogViewer.exe – Utility to view HFM error logs

36. #Kscope
Troubleshooting Tools – Internal (cont.)
 Data Gathering
● FRConfig.cmd – FR utility that shows app Mem / CPU
utilization
● System 11 - <EPM_ORACLE_HOME>productsfinancialreportingbin

37. #Kscope
Troubleshooting Tools – Internal (cont.)

38. #Kscope
Troubleshooting Tools – Internal (cont.)
 Data Gathering
● ManageUserPOV.cmd – View/Edit POV for users

39. #Kscope
Troubleshooting Tools – Internal (cont.)
 Data Gathering
● Remote Diagnostic Agent (RDA) – Gathers system
information about environment. [See KB ID 1304885.1]
● oPatch [System 11 Only] – Install and list installed patches
● Location - <EPM ORACLE HOME>Opatch
● List Installed products = Opatch lsinventory
● NOTE: If you get an error running about OUI Home,
make sure environment variable pointing to EPM
ORACLE HOME which is most like
OracleMiddlewareEPMSystem11R1

40. #Kscope
Troubleshooting Tools – Internal (cont.)

41. #Kscope
Troubleshooting Tools – Internal (cont.)
 Data Gathering
● Comps.xml [System 11 Only] – List of all installed components
on machine
● <EPM ORACLE HOME>inventoryContentsXML

42. #Kscope
Troubleshooting Tools – Internal (cont.)
 Data Gathering
● Epmsys_registry.bat [System 11 Only] – Generates a detailed
report of everything registered in Shared Services Registry.
● Program - <MIDDLEWARE HOME>user_projectsepmsystem1bin
● Reports - <MIDDLEWARE HOME>user_projectsdiagnosticsreportsregistry.html

43. #Kscope
Troubleshooting Tools – Internal (cont.)

44. #Kscope
Troubleshooting Tools – Internal (cont.)
 Data Gathering
● ‘About Us’ pages – Shows currently installed programs and
specific version info. (Applies to Workspace, FDM, Shared
Service web pages, etc.)
● Click on the ‘Show Details’ button to see version info per product
● NOTE : This version info *MAY NOT* change even if you have
installed patches!

45. #Kscope
Troubleshooting Tools – Internal (cont.)

46. #Kscope
Troubleshooting Tools – Internal (cont.)
 Data Gathering
● Informational Web Pages
● http://hostname:port/workspace/browse/configInfo - Returns
detailed information about products installed and links to
other configuration files

47. #Kscope
Troubleshooting Tools – Internal (cont.)
 Data Gathering
● Informational Web Pages
● http://<shared_services_server>:28080/interop/frameowrk/g
etCSSConfigFile – Returns configuration information for
Shared Services
● System 9 – Port Number is 58080
● System 11 – You may need to administratively enable this
page for it to be visible!

48. #Kscope
Troubleshooting Tools – Internal (cont.)

49. #Kscope
Troubleshooting Tools – Internal (cont.)
 Data Gathering
● Informational Web Pages
● http://<hfm_web_server>:80/hfm – Returns ‘hfm’ if HFM is
running properly and integrated with workspace. (else
opens HFM Web)

50. #Kscope
Troubleshooting Tools – Internal (cont.)
 Utility Programs
● Validate.bat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation.
Generates detailed report.
● Can be run on product by product basis or all at once
● Program - <MIDDLEWARE HOME>user_projectsepmsystem1bin
● Report - <MIDDLEWARE
HOME>user_projectsepmsystem1diagnosticsreports
● NOTE : If you are building an environment, you should be
running this after your complete install/config!

51. #Kscope
Troubleshooting Tools – Internal (cont.)

52. #Kscope
Troubleshooting Tools – Internal (cont.)

53. #Kscope
Troubleshooting Tools – Internal (cont.)
 Utility Programs
● DB_Verify / DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues.

54. #Kscope
3rd Party Tools

55. #Kscope
Troubleshooting Tools – 3rd Party
 Data Gathering
● WebEx / GoTo Meeting / Log Me In / VNC / Lync –
Remote Access Software. A must for remote
support.
● Windows Event Log

56. #Kscope
Troubleshooting Tools – 3rd Party
 Basic Connectivity Tools
● Ping / Tracert – Simple connectivity test tools

57. #Kscope
Troubleshooting Tools – 3rd Party
 Basic Connectivity Tools
● Telnet – Connectivity tool for any port

58. #Kscope
Troubleshooting Tools – 3rd Party (cont.)
 Advanced Connectivity Tools
● Fiddler – Web proxy that allows you to monitor (and
simulate) HTTP/HTTPS traffic.
http://fiddler2.com/fiddler2/
● Great Tool for HTTP/HTTPS traffic
● Warning : You can sniff / decode HTTPS traffic
● Warning : Some programs may send logins /
passwords in plain text.

59. #Kscope
Troubleshooting Tools – 3rd Party (cont.)

60. #Kscope
Troubleshooting Tools – 3rd Party (cont.)

61. #Kscope
Troubleshooting Tools – 3rd Party (cont.)
 Advanced Connectivity Tools
● Wireshark – Network traffic analyzer –
http://www.wireshark.org
● Network packet analyzer
● Monitors ALL network traffic, not just HTTP/HTTPS
● Warning : Can intercept all types of traffic such as:
● Streaming Audio / Video
● VoiP
● Types of Encrypted Data
● Plain Text User Name / Password

62. #Kscope
Troubleshooting Tools – 3rd Party (cont.)
 Advanced Connectivity Tools
● Wireshark – Network traffic analyzer –
http://www.wireshark.org
● Network packet analyzer
● Monitors ALL network traffic, not just HTTP/HTTPS
● Warning : Can intercept all types of traffic such as:
● Streaming Audio / Video
● VoiP
● Types of Encrypted Data
● Plain Text User Name / Password

63. #Kscope
Troubleshooting Tools – 3rd Party (cont.)

64. #Kscope
Troubleshooting Tools – 3rd Party (cont.)
 Server Tools
● Service Manager (services.msc) – View services to see
which are running
● Process Monitor – Allows you to monitor processes’ file
use, registry access, memory utilization, etc.
http://technet.microsoft.com/en-us/sysinternals/bb896645

65. #Kscope
Troubleshooting Tools – 3rd Party (cont.)
 Server Tools
● SQL Profiler – Allows you to monitor database activity in
SQL Server – http://msdn.microsoft.com/en-
us/library/ms181091.aspx
● [Oracle Enterprise Manager has similar functionality for Oracle
databases]

66. #Kscope
Troubleshooting Tools – 3rd Party (cont.)

67. #Kscope
Troubleshooting Problems

68. #Kscope
Migrate VM Server Error

69. #Kscope
Migrate VM Server Error
 To aid with session, built VM/Virtual PC System
11 “All In One” Environment

70. #Kscope
Migrate VM Server Error (cont)
 After copying VM to another machine…

71. #Kscope
Migrate VM Server Error (cont)
 But not everything was dead… VM to another
machine…

72. #Kscope
Migrate VM Server Error (cont)
 IDEAS??
●What changed?

73. #Kscope
Migrate VM Server Error (cont)
 Lets get some machine info

74. #Kscope
Migrate VM Server Error (cont)
 Lets get some config (epmsys_registry.bat)

75. #Kscope
Migrate VM Server Error (cont)
 Lets get some config (epmsys_registry.bat)

76. #Kscope
Migrate VM Server Error (cont)
 IDEAS??
●Anyone see anything suspect so far?
●I did, and have a suspicion, but need
more information to confirm.

77. #Kscope
Migrate VM Server Error (cont)
 Lets do some sniffing…(Wireshark)

78. #Kscope
Migrate VM Server Error (cont)
 IDEAS??
●Why would there be calls to an IP
address that does not belong to the
machine?

79. #Kscope
Migrate VM Server Error (cont)
 Lets do some sniffing…(WinGrep)

80. #Kscope
Migrate VM Server Error (cont)
 Lets do some more sniffing…(SQLLocator)

81. #Kscope
Migrate VM Server Error (cont)
 Lets do some more sniffing…(SQLLocator)

82. #Kscope
Migrate VM Server Error (cont)
 IDEAS??
●Looks pretty reasonable that hardcode
IP addresses are breaking things since
the machine no longer has the IP
address buried in config / database!

83. #Kscope
Migrate VM Server Error (cont)
 How To Fix
●Reinstall?
●Reconfigure?
●Manually Search and Replace?

84. #Kscope
Manual Repair!
(Because we can)

85. #Kscope
Migrate VM Server Error (cont)
 Lets fix the SQL Damage First

86. #Kscope
Migrate VM Server Error (cont)
 Lets fix the SQL Damage First

87. #Kscope
Migrate VM Server Error (cont)
 Lets fix config file damage
● Content.XML
● Config.XML
● GlobalContext.XML
● httpd.conf
● mod_wl_ohs.conf
● Server.xml
 Warning: In some places you DO NOT want to
replace the hardcoded IP address as it is part
of a Unique Identifier. That could be bad.

88. #Kscope
Migrate VM Server Error (cont)

89. #Kscope
Migrate VM Server Error (cont)
 Reboot

90. #Kscope
Migrate VM Server Error (cont)
 Mission Accomplished!

91. #Kscope
Questions?
 If you have any questions with the sample code
or presentation, feel free to ask/contact me
● charles@charlescbeyer.com