Maryna Sokyrko & Oleksandr Chugui: Building Product Passion: Developing AI chatbot – lessons learned (UA) UA Online PMDay 2024 Spring Website – www.pmday.org/online Youtube – https://www.youtube.com/startuplviv FB – https://www.facebook.com/pmdayconference

1. fluxon.com
Let’s build
the future together

2. fluxon.com
● WHO WE ARE
● AI SOLUTIONS OVERVIEW
● CHALLENGES FOR A PROJECT MANAGER
● HOW TO PLAN IN AMBIGUOUS
ENVIRONMENT
● HOW TO SET EXPECTATIONS WITH THE
STAKEHOLDERS
● Q&A
CONTENTS:

3. Who we are.
fluxon.com
3

4. fluxon.com
4
Oleksandr Chugui
PMP, Senior Technical Project
Manager
Maryna Sokyrko
Senior Technical Project Manager,
certified Agile Leader

5. fluxon.com
We are
Fluxon
A team of tech leaders from Google,
Stripe and top startups trusted by the
most innovative companies to build
world-class software products.
a unit of change

6. fluxon.com
Fluxon brings
deep expertise
across diverse
industries and
product types.

7. Let’s warm-up!
fluxon.com
7

8. AI Solutions Overview.
fluxon.com
8

9. AI intro:
fluxon.com
9
● Inferior to human intelligence.
● Lacks artificial consciousness or
cognitive abilities.
● Can't solve unfamiliar problems.
● Siri, Alexa, Google Translate, ChatGPT
Narrow AI
● Similar to human intelligence.
● Has human-like consciousness and
cognitive abilities.
● Can solve unfamiliar problems
● R2D2 from Star Wars, Wall-E.
● Yet to become a reality.
General AI (or AGI)

10. AI intro:
fluxon.com
10

11. AI intro:
fluxon.com
11
● Deep neural
network is under
the hood
● Small LLM
visualization

12. AI Products - Wrappers:
fluxon.com
12
● UI - Your Grammar Teacher
● API - Send your query to server
● Server - Prompt “You are English teacher”,
call GPT
● Return result

13. AI Products - for Engineers:
fluxon.com
13
● 83% of Engineers already use
AI tools to write code (2023
Stackoverflow survey)
● the adoption of AI tools in
corporations is still low.

14. AI Products - Chatbots:
fluxon.com
14
● Dialog trees (NPC in games)
● Anything can be a Chatbot, if
you are brave enough

15. AI Products - Chatbots:
fluxon.com
15

16. AI Products - Other:
fluxon.com
16
● Entertainment (voice, video,
audio, image)
● Harder to find its niche in
enterprise companies
○ Example: using text to video for
training materials: Synthesia.io
AI Avatar generation example ->

17. Challenges for a Project
Manager.
fluxon.com
17

18. Challenges:
fluxon.com
18
● Unpredictable Timelines: Traditional
planning struggles with AI projects. It's
hard to know how long things take.

19. Challenges:
fluxon.com
19
● Shifting Features: AI chatbots can evolve
as you build them, making it tough to define
features upfront.
● Prompt Engineering: It’s unpredictable,
hard to estimate, hard to track.

20. Challenges:
fluxon.com
20
● Data ingestions: Good training data is
crucial, ingesting data from multiple
sources might be a challenge.
● Testing: Testing often relies on manual
work. Debugging is also manual and not
trivial.

21. Challenges: Summary
fluxon.com
21
● Hard to estimate things, which makes it hard to plan and build a roadmap.
● Creative work, prompt tuning is more an art than a tech skill.
● Data should be verified, well-structured, up-to-date, depends on a third-party often.
● Testing and debugging are manual and time consuming.

22. How to plan in ambiguous
environment.
fluxon.com
22

23. What’s the plan?
fluxon.com
23
● Embrace Experimentation: Since traditional planning is tricky, introduce such term
as ‘experimenting’ with different approaches.
● Time-Boxed Exploration: Set clear time boxes for your experiments.
● Lessons learnt: Work with historical data.

24. What’s the plan?
fluxon.com
24
Author P Idea Value High-level scope Document Estimate Notes Decision
Alex M. P2 Use ready valuation
framework to
improve
effectiveness of the
chatbot testing
process
Optimise time
spent on testing,
have a baseline,
benefit from a
third-party toll
which is well-
supported and
evolving
Research options
Develop POC
Prepare
implementation
plan
Estimate
Integrate solution
into chatbot
ecosystem
Link Research and
POC: 1w
Known
framework:
Ragas
Go

25. What’s the plan?
fluxon.com
25
● Document Everything: Technical design, experimenting results, decision
making, architecture, etc.
● Motivational Mix: Combine working on core features with response
improvements.

26. What’s the plan?
fluxon.com
26
● Systematic Improvement: Develop a
systematic approach for analyzing results and
making improvements based on data.
● Measure & Track: Use a testing framework to
analyze chatbot performance metrics.
Ragas framework: https://docs.ragas.io/en/stable/

27. Our experience: Summary
fluxon.com
27
● Time-boxed and documented experiments.
● Plan by small iterations, plan based on the knowledge, research, documents.
● Mix tasks, keep the team motivated.
● Optimise your testing process by all means.

28. How to set expectations with
the Stakeholders.
fluxon.com
28

29. Stakeholders Management:
fluxon.com
29
● Transparency is Key: Explain the ambiguous nature of AI chatbot development
to stakeholders.
○ Kick-off call
○ Demo
○ Share materials async

30. Stakeholders Management:
fluxon.com
30
● Show, Don't Just Tell: Regularly showcase demos using real user questions,
especially when it comes to challenges and areas where you struggle.
○ Prepare test cases and dataset references
○ Share environment to test with the instructions

31. Stakeholders Management:
fluxon.com
31
● Data-Driven Decisions: Use clear metrics in your reports to track progress and
demonstrate the impact of the chatbot development. For example, we used:
○ Custom metric, like pass rate.
○ And these:

32. Stakeholders: Summary
fluxon.com
32
● Show and explain everything: process, progress, challenges, fails and success.
● Use data driven approach, think how to demonstrate your testing results and how
the chatbot is evolving.
● Give them access to try it out and setup clear feedback process.

33. fluxon.com
Thank you
for your
attention!