Timestamped outline for Continuous Performance Regression Testing with JfrUnit by Gunnar Morling at jChampionsConference 2022

• By Gunnar Morling
  • Follow him on Twitter: @gunnarmorling
  • Find JfrUnit on Github: github.com/moditect/jfrunit

Outline

• 1:06 Objectives
  • Implementing performance regression tests
  • Mission Control
  • Emitting JFR events
• 3:10 Challenges of Performance Test
  • Requires prod-like setup
  • Impact by unrelated concurrent load
• 6:05 JfrUnit
  • Proxy metrics
    • memory, number of objects, etc
    • increase -> performance penalty likely
    • Benefit: not environment-based
  • Plain unit test
  • Uses JFR/Mission Control
• 8:21 Hello World
  • JUnit5
  • @JfrEventTest on class
  • @EnableEvent(event type) on function enables collection
  • jfrEvents.awaitEvents() blocks till all events are recorded
    • 41:47 required because some events are emitted, but finish later
  • can assert on jfrEvents
• Tools
  • 11:27 Flight Recorder
    • efficient event recorder
    • compact binary format
    • custom events possible
    • used to be propriety, is OSS since Java 11, also backported
    • Low overhead (<1%), meant for always-on
    • recent GraalVM, too
  • 14:58 Event Types
    • 150+ types
    • Website with organized list
  • 15:58 JFR Event Streaming
    • Since Java 14
    • React to events -> metrics
    • Predictive analytics
    • 19:34 How to code example
  • 20:17 Mission Control
    • OSS
    • Analyze, visualize JFR recordings
• 21:45 Demo
  • SocketIO: verify number of jdk.SocketRead / jdk.SocketWrite events
  • 27:24 Filter events by port to get DB IO only
  • JfrUnit persists JFR per test-case -> enables further analysis
    • Many findById in stacktrace
    • Points to code, additional @Lob, maybe lazy fetching is better here
• 30:26 JMC Agent
  • Instrument 3rd party code
  • Emit event when configured classes are invoked
  • 32:34 Configuration example for DB Connection Pool
  • Can capture values, too
  • Before, after, wrap
    • Wrap captures duration
• 35:08 Demo with JMC Agent
  • Test produces JFR recording again
  • Custom event found in Event Browser
  • Custom Events contain complete Query and stacktrace
  • Stacktrace points to problematic N+1 code
• 39:19 JFR Analytics (WIP)
  • Querying JFR using SQL
  • Pull-based for automated analysis
  • Streaming Queries
  • 45:04 Demo
  • 48:38 Example: Top 10 allocating stacktraces
• 51:33 Discussion and Limitations
  • Robust and fast
  • But requires good understanding of Proxy Metrics
    • Increased allocation may be fine
  • Won’t easily show sync/locking issues
  • Won’t help everywhere, e.g. bad execution plan
• 54:27 Take Aways
  • It’s a complement
  • JMC Agend for 3rd-party
  • Move beyond Java 1.8
• 55:39 Outlook
  • Historic Trends when things get gradually worse
• 57:00 Resources / further reading
  • ContainerJFR for k8s
• Q&A
  • 1:00:03 Would parallelism / asynchrouncy affect test results?
    • It depends, but parallelism is happening in the examples already, generally it works, when you are interested in certain threads that might require more thinking
  • 1:01:48 Can you comment on how this level of testing accounts for scalar replacement?
    • Unrelated, but maybe JIT compiler events are affected?
  • 1:02:33 What is scalar replacement?
    • Compiler can inline objects that don’t leave the method’s scope

Further Reading

• README on https://github.com/moditect/jfrunit
• Towards Continuous Performance Regression Testing
• Introducing JfrUnit 1.0.0.Alpha1
• Asserting JDK Flight Recorder Events with JfrUnit
• Continuous Performance Regression Testing with JfrUnit
• Keep Your SQL in Check With Flight Recorder, JMC Agent and JfrUnit