Performance Profiling and Numeric Python

•Download as PPTX, PDF•

0 likes•10 views

Python often gets a bad wrap for performance, specifically regarding numeric computations. In this presentation I've dug in to Numba and how I got around an error I was having, solving a performance bottleneck I had.

Software

Performance
Profiling and
Numeric Python
BEN WELLER

Brief Problem Statement
 Reading DIRECTLY off the slides
 Never posts the slides
 Video is available
 Automatically grab all the slides
 Quickly underwhelmed with speed
 How to speed this up
 Speed has never been an issue previously

Python isn’t the Problem*
 The fault dear programmer isn’t in Python
 But sometimes it is
 Get around it
 Async
 C extension
 Pypy (JIT everything)
 Numba (just JIT certain methods)

Before Anything Else:
 Software Hierarchy of needs
Fast enough?
Tests
Does the code
work?

Initial Code/Data
 Reading the frames:
 Data:
 1920 x 1080 x 3
 All pixels recorded in the video, then 3 values for each pixel (RGB)
 Test for validation:

Version 0.0
 Establish a baseline
 Time: 788.85

Version 0.1
 Remove all the redundant lines:
 Time: 728.88

Version 0.15
 Apply Numba
 Time: 200.35

Version 0.175
 Realize your calculation is wrong!
 Redo with new formula
 Time: 582.29

Version 0.2
 Apply numba
 Very slow odd results
 Time: 1020.90
 Results: [4458465513457839.0, 2137656293717757.8, 3461590301822020.5,
2435933916096979.0, 1020423444981550.0 …]
 Not bounded between 0 and 100

Version 0.75
 Realize there’s something wrong with broadcast operations?
 Time: 549.63
 Output was all in the valid range

Version 1.0
 Apply numba in full force
 Time: 186.37

What I learned
 Follow the hierarchy of needs!
 If you don’t you will get burned
 Tests/initial benchmarks are key
 Slowest code first
 Incrementally solve the next bit
 Very rarely, break out specialized tools
 Know your tools
 Has anyone ran in to the problem I had with broadcasting?

Similar to Performance Profiling and Numeric Python

ELC-E 2010: The Right Approach to Minimal Boot Times

andrewmurraympc

Big Data Expo 2015 - Anchormen Distributed video analysis

BigDataExpo

Keynote AST 2016

Kim Herzig

Zipline is Airbnb’s data management platform specifically designed for ML use cases. Previously, ML practitioners at Airbnb spent roughly 60% of their time on collecting and writing transformations for machine learning tasks. Zipline reduces this task from months to days – by making the process declarative. It allows data scientists to easily define features in a simple configuration language. The framework then provides access to point-in-time correct features – for both – offline model training and online inference. In this talk we will describe the architecture of our system and the algorithm that makes the problem of efficient point-in-time correct feature generation, tractable. The attendee will learn Importance of point-in-time correct features for achieving better ML model performance Importance of using change data capture for generating feature views An algorithm – to efficiently generate features over change data. We use interval trees to efficiently compress time series features. The algorithm allows generating feature aggregates over this compressed representation. A lambda architecture – that enables using the above algorithm – for online feature generation. A framework, based on category theory, to understand how feature aggregations be distributed, and independently composed. While the talk if fairly technical – we will introduce all the concepts from first principles with examples. Basic understanding of data-parallel distributed computation and machine learning might help, but are not required.

Zipline—Airbnb’s Declarative Feature Engineering Framework

Databricks

LabVIEW Report

Manish Kumar

Most high-level programming languages run on top of a virtual machine (VM) to abstract away from the underlying hardware. To reach high-performance, the VM typically relies on an optimising just-in-time compiler (JIT), which speculates on the program behavior based on its first runs to generate at runtime efficient machine code and speed-up the program execution. As multiple runs are required to speculate correctly on the program behavior, such a VM requires a certain amount of time at start-up to reach peak performance. The optimising JIT itself is usually compiled ahead-of-time to executable code as part of the VM. The dissertation proposes Sista, an architecture for an optimising JIT, in which the optimised state of the VM can be persisted across multiple VM start-ups and the optimising JIT is running in the same runtime than the program executed. To do so, the optimising JIT is split in two parts. One part is high-level: it performs optimisations specific to the programming language run by the VM and is written in a metacircular style. Staying away from low-level details, this part can be read, edited and debugged while the program is running using the standard tool set of the programming language executed by the VM. The second part is low-level: it performs machine specific optimisations and is compiled ahead-of-time to executable code as part of the VM. The two parts of the JIT use a well-defined intermediate representation to share the code to optimise. This representation is machine-independent and can be persisted across multiple VM start-ups, allowing the VM to reach peak performance very quickly. To validate the architecture, the dissertation includes the description of an implementation on top of Pharo Smalltalk and its VM. The implementation is able to run a large set of benchmarks, from large application benchmarks provided by industrial users to micro-benchmarks used to measure the performance of specific code patterns. The optimising JIT is implemented according to the architecture proposed and shows significant speed-up (up to 5x) over the current production VM. In addition, large benchmarks show that peak performance can be reached almost immediately after VM start-up if the VM can reuse the optimised state persisted from another run.

Ph.D Defense Clément Béra

Clément Béra

Squeezing Blood From a Stone V1.2

Jen Costillo

IMAGE PROCESSING

ABHISHEK MAURYA

智慧檢測技術與工業自動化

CHENHuiMei

Move out from AppEngine, and Python PaaS alternatives

tzang ms

Circular barcode design resistant to linear motion blur (preliminary slides)

Naoki Shibata

Unit testing for the TYPO3 4.x core

Oliver Klee

IoT evolution

Yoss Cohen

Continuous Delivery for Python Developers – PyCon Otto

Peter Bittner

Raspberry pi jam july

Jon Ho

Elizabeth Perry: Processing programming language, part 2

ÚISK FF UK

Python testing like a pro by Keith Yang

PYCON MY PLT

HW Emulators: Does it Belong in your Verification Tool Chest?

DVClub

Capturing and Displaying Digital Image

Integrated Circuit Design Research & Education Center (ICDREC)

Cuda meetup presentation 5

Rihards Gailums

Similar to Performance Profiling and Numeric Python (20)

ELC-E 2010: The Right Approach to Minimal Boot Times

Big Data Expo 2015 - Anchormen Distributed video analysis

Keynote AST 2016

Zipline—Airbnb’s Declarative Feature Engineering Framework

LabVIEW Report

Ph.D Defense Clément Béra

Squeezing Blood From a Stone V1.2

IMAGE PROCESSING

智慧檢測技術與工業自動化

Move out from AppEngine, and Python PaaS alternatives

Circular barcode design resistant to linear motion blur (preliminary slides)

Unit testing for the TYPO3 4.x core

IoT evolution

Continuous Delivery for Python Developers – PyCon Otto

Raspberry pi jam july

Elizabeth Perry: Processing programming language, part 2

Python testing like a pro by Keith Yang

HW Emulators: Does it Belong in your Verification Tool Chest?

Capturing and Displaying Digital Image

Cuda meetup presentation 5

Recently uploaded

How To Troubleshoot Collaboration Apps for the Modern Connected Worker

ThousandEyes

HR Software Buyers Guide in 2024 - HRSoftware.com

Fatema Valibhai

A great deal of attention in medical devices has shifted towards cybersecurity with the ratification of section 524B of the FD&C act. This new law enables the FDA to enforce cybersecurity controls in any medical device that is capable of networked communications or that has software. In this webinar we will recap the process for managing vulnerabilities, identify categories of vulnerabilities and solutions and more.

The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...

ICS

Software Quality Assurance Interview Questions

Arshad QA

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

masabamasaba

MakeMyPass" Online Bus Pass Management System illustrates the flow of activities and actions that occur within the system to accomplish specific tasks or use cases. This type of diagram focuses on representing the sequence of activities and decision points involved in a particular process. Below is an example outline and description of key elements that could be included in an Activity Diagram for the system:

BUS PASS MANGEMENT SYSTEM USING PHP.pptx

alwaysnagaraju26

VTU technical seminar 8Th Sem on Scikit-learn

AmarnathKambale

In the realm of real-time applications, Large Language Models (LLMs) have long dominated language-centric tasks, while tools like OpenCV have excelled in the visual domain. However, the future (maybe) lies in the fusion of LLMs and deep learning, giving birth to the revolutionary concept of Large Action Models (LAMs). Imagine a world where AI not only comprehends language but mimics human actions on technology interfaces. For example, the Rabbit r1 device presented at CES 2024, driven by an AI operating system and LAM, brings this vision to life. It executes complex commands, leveraging GUIs with unprecedented ease. In this presentation, join me on a journey as a software engineer tinkering with WebRTC, Janus, and LLM/LAMs. Together, we’ll evaluate the current state of these AI technologies, unraveling the potential they hold for shaping the future of real-time applications.

Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications

Alberto González Trastoy

In the past six months, the AI landscape has undergone a massive transformation, ushering in a new era of productivity with the latest in Large Language Models (LLMs) and AI technology. This deep dive unlocks how to: Create CustomGPT Models: No coding needed to tailor AI for your unique projects. Integrate your own data, including PDFs and Excel sheets, making information handling a breeze. Plus, discover how to call your own actions/integrations for even more personalized utility. Navigate Advanced Prompting: Overcome AI's memory limits and utilize Retrieval-Augmented Generation for accessing your personalized data, streamlining how you interact with AI. Stay Ahead with AI Trends: Peek into the evolving world of LLMs, featuring newcomers like Google Gemini, Anthropic Claude, Open Sora, and Twitter Grok, and understand what their advancements mean for your productivity. Witness Real-Life Transformations: Through examples and prompt demonstrations, see firsthand how these AI strategies revolutionize routine tasks, from data analysis to content creation. Learn to leverage image output and input for advanced practical use cases, adding a new dimension to your productivity toolkit. No previous coding or AI experience is needed for this talk. Stay ahead in the fast-evolving world of work. Embrace the AI revolution and transform your workflow with advanced LLM techniques. Join us to ensure you're not left behind in the productivity race.

AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques

VictorSzoltysek

Investing in AI transformation today The modern business advantage: Uncovering deep insights with AI Organizations around the world have come to recognize AI as the transformative technology that enables them to gain real business advantage. AI’s ability to organize vast quantities of data allows those who implement it to uncover deep business insights, augment human expertise, drive operational efficiency, transform their products, and better serve their customers

Microsoft AI Transformation Partner Playbook.pdf

Willy Marroquin (WillyDevNET)

8257 interfacing 2 in microprocessor for btech students

HimanshiGarg82

At the recent Microsoft Ignite 2023 conference, Microsoft unveiled a groundbreaking strategy that will redefine enterprise work management. The plan involves integrating Microsoft’s key planning tools, Microsoft To Do, Microsoft Planner, and Microsoft Project for the web into a unified experience called “Microsoft Planner.” What does this new strategy from Microsoft mean for current users? Join us and learn how best to take advantage of this announcement while gaining a clear path on how to elevate the current state of Microsoft Planner from a basic task manager to a comprehensive tool for Enterprise Work Management using OnePlan. Learn how OnePlan’s integration with Microsoft Planner allows for strategic alignment with business goals through advanced features like strategic planning, portfolio management, resource management, financial management, and more!

Introducing Microsoft’s new Enterprise Work Management (EWM) Solution

OnePlan Solutions

Model Call Girl Services in Delhi reach out to us at 🔝 9953056974 🔝✔️✔️ Our agency presents a selection of young, charming call girls available for bookings at Oyo Hotels. Experience high-class escort services at pocket-friendly rates, with our female escorts exuding both beauty and a delightful personality, ready to meet your desires. Whether it's Housewives, College girls, Russian girls, Muslim girls, or any other preference, we offer a diverse range of options to cater to your tastes. We provide both in-call and out-call services for your convenience. Our in-call location in Delhi ensures cleanliness, hygiene, and 100% safety, while our out-call services offer doorstep delivery for added ease. We value your time and money, hence we kindly request pic collectors, time-passers, and bargain hunters to refrain from contacting us. Our services feature various packages at competitive rates: One shot: ₹2000/in-call, ₹5000/out-call Two shots with one girl: ₹3500/in-call, ₹6000/out-call Body to body massage with sex: ₹3000/in-call Full night for one person: ₹7000/in-call, ₹10000/out-call Full night for more than 1 person: Contact us at 🔝 9953056974 🔝. for details Operating 24/7, we serve various locations in Delhi, including Green Park, Lajpat Nagar, Saket, and Hauz Khas near metro stations. For premium call girl services in Delhi 🔝 9953056974 🔝. Thank you for considering us!

CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE

9953056974 Low Rate Call Girls In Saket, Delhi NCR

Looking for an efficient way to manage your finances? Look no further than our money management app. With easy-to-use features, you can track your expenses, create budgets, and monitor your savings goals all in one place. Our app provides real-time updates on your spending habits and helps you make smarter financial decisions. Take control of your finances today with our user-friendly money management app.

Right Money Management App For Your Financial Goals

Jhone kinadey

Data spaces in distributed environments should be allowed to evolve in agile ways providing data space owners with large flexibility about which data they store. Agility and heterogeneity, however, jeopardize data exchanges because representations may build on varying ontologies and data consumers may not rely on the semantic correctness of their queries in the context of semantically heterogeneous, evolving data spaces. Graph data spaces are one example of a powerful model for representing and querying data whose semantics may change over time. To assert and enforce conditions on individual graph data spaces, shape languages (e.g SHACL) have been developed. We investigate the question of how querying and programming can be guarded by reasoning over SHACL constraints in a distributed setting and we sketch a picture of how a future landscape based on semantically heterogeneous data spaces might look like.

Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...

Steffen Staab

Many specialized tools cater to distinct stages within the software development lifecycle (SDLC). These tools target various aspects of development, delivery, and operations, each with its unique strengths. Uniting these diverse testing needs into a single continuous testing platform presents several challenges. Such a platform must seamlessly integrate with various development tools and environments, accommodate different testing methodologies, and remain flexible to adapt to organizational processes and quality standards.

The Guide to Integrating Generative AI into Unified Continuous Testing Platfo...

kalichargn70th171

Azure Native Qumulo scales elastically for common High Performance Compute (HPC) workloads based on application requirements for: Financial Services, Automotive, Genomics / Life Sciences, Media and Entertainment, Energy, Oil and Gas, etc. Performance can be dialed UP (and back down) much higher than the examples shown here. These slides offer a glimpse into ANQ's HPC capabilities, although at a smaller scale. We invite YOU to do your own testing (with a free ANQ trial) and work with us to test your HPC workloads in Azure.

Azure_Native_Qumulo_High_Performance_Compute_Benchmarks.pdf

ryanfarris8

Direct Style Effect Systems -The Print[A] Example- A Comprehension Aid

Philip Schwarz

%in kempton park+277-882-255-28 abortion pills for sale in kempton park

masabamasaba

The title is not connected to what is inside

shinachiaurasa2

Recently uploaded (20)

How To Troubleshoot Collaboration Apps for the Modern Connected Worker

HR Software Buyers Guide in 2024 - HRSoftware.com

The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...

Software Quality Assurance Interview Questions

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

BUS PASS MANGEMENT SYSTEM USING PHP.pptx

VTU technical seminar 8Th Sem on Scikit-learn

Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications

AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques

Microsoft AI Transformation Partner Playbook.pdf

8257 interfacing 2 in microprocessor for btech students

Introducing Microsoft’s new Enterprise Work Management (EWM) Solution

CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE

Right Money Management App For Your Financial Goals

Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...

The Guide to Integrating Generative AI into Unified Continuous Testing Platfo...

Azure_Native_Qumulo_High_Performance_Compute_Benchmarks.pdf

Direct Style Effect Systems -The Print[A] Example- A Comprehension Aid

%in kempton park+277-882-255-28 abortion pills for sale in kempton park

The title is not connected to what is inside

Performance Profiling and Numeric Python

1. Performance Profiling and Numeric Python BEN WELLER

2. Brief Problem Statement  Reading DIRECTLY off the slides  Never posts the slides  Video is available  Automatically grab all the slides  Quickly underwhelmed with speed  How to speed this up  Speed has never been an issue previously

3. Python isn’t the Problem*  The fault dear programmer isn’t in Python  But sometimes it is  Get around it  Async  C extension  Pypy (JIT everything)  Numba (just JIT certain methods)

4. Before Anything Else:  Software Hierarchy of needs Fast enough? Tests Does the code work?

5. Initial Code/Data  Reading the frames:  Data:  1920 x 1080 x 3  All pixels recorded in the video, then 3 values for each pixel (RGB)  Test for validation:

6. Version 0.0  Establish a baseline  Time: 788.85

7. Version 0.1  Remove all the redundant lines:  Time: 728.88

8. Version 0.15  Apply Numba  Time: 200.35

9. Version 0.175  Realize your calculation is wrong!  Redo with new formula  Time: 582.29

10. Version 0.2  Apply numba  Very slow odd results  Time: 1020.90  Results: [4458465513457839.0, 2137656293717757.8, 3461590301822020.5, 2435933916096979.0, 1020423444981550.0 …]  Not bounded between 0 and 100

11.

12. Version 0.75  Realize there’s something wrong with broadcast operations?  Time: 549.63  Output was all in the valid range

13. Version 1.0  Apply numba in full force  Time: 186.37

14. What I learned  Follow the hierarchy of needs!  If you don’t you will get burned  Tests/initial benchmarks are key  Slowest code first  Incrementally solve the next bit  Very rarely, break out specialized tools  Know your tools  Has anyone ran in to the problem I had with broadcasting?

Performance Profiling and Numeric Python

Recommended

Recommended

More Related Content

Similar to Performance Profiling and Numeric Python

Similar to Performance Profiling and Numeric Python (20)

Recently uploaded

Recently uploaded (20)

Performance Profiling and Numeric Python