Investigative Debugging of ManageIQ Automation by Peter McGowan at ManageIQ Design Summit 2016 Demo 1: https://youtu.be/5zDU2Fz7pq4 Demo 2: https://youtu.be/NNd8sJJ57O4 Demo 3: https://youtu.be/k0b6wLaDLwE Demo 4: https://youtu.be/mZYz4ZM1hpA Demo 5: https://youtu.be/LVUAN3tkrP8 Demo 6: https://youtu.be/lyLxanQPkhs

1. Investigative Debugging of
ManageIQ Automation
Peter McGowan

2. (An Automation
Scripter’s Guide to
Overcoming
Frustrations)

3. Agenda
• Making sense of the log files
• $evm & Service Model refresher
• Examining objects
• Inspecting objects
• Walking objects
• Object analysis mid-workflow

4. Takeaways
How to use the logs effectively
Understand when to use $evm.root or $evm.object
Understand how to examine Service Model objects
Understand how to inspect/debug objects mid-workflow

5. Use the Logs, Luke

6. Log Files
[----] I, [2016-05-23T11:34:08.351374 #3880:51197c] INFO -- : MIQ(MiqQueue.put) Message id: [2679], id: [], Zone: [default], Role: [smartstate], Server: [], Ident:
[generic], Target id: [], Instance id: [], Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout: [600], Priority: [20], State: [ready], Deliver On: [],
Data: [], Args: []
[----] I, [2016-05-23T11:34:09.144251 #3655:51197c] INFO -- : MIQ(MiqServer#populate_queue_messages) Fetched 2 miq_queue rows for queue_name=generic,
wcount=4, priority=200
[----] I, [2016-05-23T11:34:09.754587 #3857:51197c] INFO -- : MIQ(MiqPriorityWorker::Runner#get_message_via_drb) Message id: [2679], MiqWorker id: [4], Zone:
[default], Role: [smartstate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout:
[600], Priority: [20], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [1.404928107] seconds

7. Log Files
# rake evm:status
Checking EVM status...
Zone | Server Name | Status | ID | PID | SPID | URL | Started On | ...
default | EVM | started | 1 | 3655 | 3685 | druby://127.0.0.1:60715 | 2016-05-23T05:58:07Z | ...
Worker Type | Status | ID | PID | SPID | Queue Name / URL --------------------------------------------------------------+---------+----+-------+-
------+----------------------- ManageIQ::Providers::Openstack::CloudManager::EventCatcher | started | 16 | 4760 | 4764 | ems_1 |
ManageIQ::Providers::Openstack::CloudManager::RefreshWorker | started | 20 | 34230 | 34234 | ems_1 |
ManageIQ::Providers::Openstack::NetworkManager::EventCatcher | started | 17 | 4767 | 4771 | ems_2 |
ManageIQ::Providers::Openstack::NetworkManager::RefreshWorker | started | 21 | 34235 | 34239 | ems_2 |
MiqEventHandler | started | 1 | 3841 | 3845 | ems |
MiqGenericWorker | started | 3 | 3851 | 3855 | generic |
MiqGenericWorker | started | 2 | 3846 | 3850 | generic |
MiqPriorityWorker | started | 4 | 3857 | 3861 | generic |
MiqPriorityWorker | started | 5 | 3863 | 3868 | generic |
MiqReportingWorker | started | 7 | 3874 | 3878 | reporting |
MiqReportingWorker | started | 6 | 3869 | 3873 | reporting |
MiqScheduleWorker | started | 8 | 3880 | 3885 | |
MiqUiWorker | started | 10 | 3904 | | http://127.0.0.1:3000 |
MiqWebServiceWorker | started | 11 | 3928 | | http://127.0.0.1:4000 |
MiqWebsocketWorker | started | 9 | 3892 | | http://127.0.0.1:5000 |

8. Log Files
grep -i "3846:|3851:|3857:|3863:" log/evm.log
[----] I, [2016-05-23T11:52:15.090823 #3846:51197c] INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [2845], MiqWorker id: [2], Zone:
[default], Role: [automate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [MiqAeEngine.deliver], Timeout: [3600], Priority: [20], State:
[dequeue], Deliver On: [], Data: [], Args: [{:object_type=>"AutomationRequest", :object_id=>11, :attrs=>{:event_type=>"request_created",
"EventStream::event_stream"=>100, :event_stream_id=>100}, :instance_name=>"Event", :user_id=>1, :miq_group_id=>2, :tenant_id=>1, :automate_message=>nil}],
Dequeued in: [3.039205536] seconds
[----] I, [2016-05-23T11:52:16.166956 #3851:51197c] INFO -- : Q-task_id([automation_task_11]) <AutomationEngine> Invoking [inline] method [/Bit63/Stuff/Methods/test]
with inputs [{}]
[----] I, [2016-05-23T11:52:16.167584 #3851:51197c] INFO -- : Q-task_id([automation_task_11]) <AutomationEngine> <AEMethod [/Bit63/Stuff/Methods/test]> Starting
[----] I, [2016-05-23T11:52:16.241517 #3846:b1083c] INFO -- : <AutomationEngine> <AEMethod parse_provider_category> Parse Provider Category Key: nil Value:
unknown
[----] I, [2016-05-23T11:52:16.290814 #3846:51197c] INFO -- : <AutomationEngine> <AEMethod /ManageIQ/System/Process/parse_provider_category]> Ending
[----] I, [2016-05-23T11:52:16.290992 #3846:51197c] INFO -- : <AutomationEngine> Method exited with rc=MIQ_OK

9. Use Following…Followed
Following Relationship [miqaedb:/cloud/VM/Lifecycle/Provisioning#create]
Following Relationship
[miqaedb:/Cloud/VM/Provisioning/StateMachines/VMProvision_vm/template#create]
…
Followed Relationship
[miqaedb:/Cloud/VM/Provisioning/StateMachines/VMProvision_vm/template#create]
Followed Relationship [miqaedb:/cloud/VM/Lifecycle/Provisioning#create]

10. Extracting $evm.log Output
grep and awk are your friends:
grep "AEMethod test” automation.log |
awk 'BEGIN {FS="AEMethod test>"}; {print $2}'

11. $evm & Service Model Refresher

12. $evm
$evm.log(:info, "my_method started")
vm_name = $evm.root['dialog_vm_name']
username = $evm.object['username']
password = $evm.object.decrypt('password')
$evm.set_state_var(:this_or_that, "the_other")

13. $evm
“Handle” into the Automation Engine
Attributes (e.g. $evm.current_instance, $evm.current_method)
Methods (e.g. $evm.log, $evm.vmdb, $evm.instantiate)
Workspace (e.g. $evm.root, $evm.parent, $evm.object)

14. Instance Object Hierarchy
/ManageIQ/System/Process/AUTOMATION ($evm.root)
| /ManageIQ/cloud/VM/Lifecycle/Provisioning
| | /ManageIQ/Cloud/VM/Provisioning/Profile/EvmGroup-super_administrator
| | /Bit63/Cloud/VM/Provisioning/StateMachines/VMProvision_vm/template ($evm.parent)
| | | /ManageIQ/Cloud/VM/Provisioning/StateMachines/Methods/CustomizeRequest
| | | /Bit63/Stuff/Methods/my_method ($evm.object)

15. The Sandpit
MiqAeMethodService::MiqAeService.new(MiqAeEngine::MiqAeWorkspaceRuntime.new)
Automation Engine
MiqAeMethodService::MiqAeService.new(MiqAeEngine::MiqAeWorkspaceRuntime.new)
Automate Method
$evm (DRb::DRbObject)
druby://127.0.0.1:54787
Service Model
Service Model
Service Model

16. Service Models
MiqAeServiceSomething
(Mostly) read-only abstractions of Active Records
• RHEV Virtual Machine (MiqAeServiceManageIQ_Providers_Redhat_InfraManager_Vm)
• User (MiqAeServiceUser)
• Provision Request (MiqAeServiceMiqProvisionRequest)
Retrievable using $evm.vmdb

17. Service Model Object Properties
Attributes (e.g. vm.name, vm.raw_power_state)
Virtual Columns (e.g. vm.cpu_total_cores, vm.ipaddresses)
Associations (e.g. vm.availability_zone, vm.ext_management_system)
Methods (e.g. vm.start, vm.stop)

18. Examining Objects
Rails Console
(https://github.com/georgegoh/cloudforms-util)
Demo

19. Inspecting Objects
InspectMe
Demo

20. Inspecting Objects
An Improved InspectMe
(https://github.com/ramrexx/CloudForms_Essentials)
Demo

21. Walking Objects
object_walker
(https://github.com/pemcg/object_walker)
Demo

22. Object Analysis Mid-Workflow – Scenario 1
How can I see what my provisioning dialog selection
looks like in the options hash?
Demo

23. Object Analysis Mid-Workflow – Scenario 2
What changes do the DialogParser and CatalogItemInitialization methods make to my
service_template_provision_task object?
Demo

24. Summary
• Use the logs to your advantage
• Understand the relationships between instances
• Understand what a Service Model is
• Use the Inspection tools