Python Performance Profiling: The Guts And The Glory

01:45 PM - 02:10 PM on July 16, 2016, Room CR4

A. Jesse Jiryu Davis

Audience level:

intermediate

Watch:

https://www.youtube.com/watch?v=EJ87Kfzvnbs

Description

Your Python program is too slow, and you need to optimize it. Where do you start? Optimization is a scientific investigation, just like debugging is: you generate hypotheses about how to make your program faster, then use benchmarks as the experiments that test your hypotheses. With the right tools, you can optimize your code where it counts. With the Python profiler “Yappi” to understand its features and limitations we’ll learn how to find the maximum performance wins with minimum effort.

Abstract

My boss alerted me to an article on a popular site, which claimed to show that my open-source Python client for MongoDB is three times slower than the Javascript client. Anxiety immediately set in: Was this true? Could I improve it? What should I tell my boss?

I'll describe three open-source profilers for Python, and explain why Yappi is the right profiler for this job.

Optimization is like debugging: we use the scientific method. We form a hypothesis about what changes we can make to our code to make it faster, and we estimate how much improvement an optimization can yield. Then comes the second part of the scientific method: we test the hypothesis with an experiment.

And now, let's return to the problem at hand: Is my code slow? We'll use Yappi to form some hypotheses, then validate them with tests. Finally, I have an answer to my boss's question, and better yet, I can back up my answer with hard evidence.