Mystifying microbenchmark result for stream API on Java 12 vs. Java 8 with -gc true
Solution 1:
Thanks, everyone for the help and especially to @Aleksey Shipilev!
After applied changes to JMH benchmark, the results look more realistic (?)
Changes:
-
Change the setup method to be executed before/after each iteration of the benchmark.
@Setup(Level.Invocation)->@Setup(Level.Iteration) -
Stop JMH forcing GC between iterations. Forcing Full GC before each iteration is quite likely to throw off GC heuristics. (c) Aleksey Shipilev
-gc true->-gc false
Note: gc false by default.
Comparison tables
Based on new performance benchmarks there is no performance degradation on Java 12 compare to Java 8.
Note: After those changes, the throughput error for a small array size significantly increased for more than 100%, for a large dataset remain the same.
Raw results
Java 8
# Run complete. Total time: 04:36:29
Benchmark (arraySize) Mode Cnt Score Error Units
FilterBenchmark.complexFilter 10 thrpt 50 5947577.648 ± 257535.736 ops/s
FilterBenchmark.complexFilter 100 thrpt 50 3131081.555 ± 72868.963 ops/s
FilterBenchmark.complexFilter 1000 thrpt 50 489666.688 ± 6539.466 ops/s
FilterBenchmark.complexFilter 10000 thrpt 50 17297.424 ± 93.890 ops/s
FilterBenchmark.complexFilter 100000 thrpt 50 1398.702 ± 72.820 ops/s
FilterBenchmark.complexFilter 1000000 thrpt 50 81.309 ± 0.547 ops/s
FilterBenchmark.complexFilterParallel 10 thrpt 50 24515.743 ± 450.363 ops/s
FilterBenchmark.complexFilterParallel 100 thrpt 50 25584.773 ± 290.249 ops/s
FilterBenchmark.complexFilterParallel 1000 thrpt 50 24313.066 ± 425.817 ops/s
FilterBenchmark.complexFilterParallel 10000 thrpt 50 11909.085 ± 51.534 ops/s
FilterBenchmark.complexFilterParallel 100000 thrpt 50 3260.864 ± 522.565 ops/s
FilterBenchmark.complexFilterParallel 1000000 thrpt 50 406.297 ± 96.590 ops/s
FilterBenchmark.multipleFilters 10 thrpt 50 3785766.911 ± 27971.998 ops/s
FilterBenchmark.multipleFilters 100 thrpt 50 1806210.041 ± 11578.529 ops/s
FilterBenchmark.multipleFilters 1000 thrpt 50 211435.445 ± 28585.969 ops/s
FilterBenchmark.multipleFilters 10000 thrpt 50 12614.670 ± 370.086 ops/s
FilterBenchmark.multipleFilters 100000 thrpt 50 1228.127 ± 21.208 ops/s
FilterBenchmark.multipleFilters 1000000 thrpt 50 99.149 ± 1.370 ops/s
FilterBenchmark.multipleFiltersParallel 10 thrpt 50 23896.812 ± 255.117 ops/s
FilterBenchmark.multipleFiltersParallel 100 thrpt 50 25314.613 ± 169.724 ops/s
FilterBenchmark.multipleFiltersParallel 1000 thrpt 50 23113.388 ± 305.605 ops/s
FilterBenchmark.multipleFiltersParallel 10000 thrpt 50 12676.057 ± 119.555 ops/s
FilterBenchmark.multipleFiltersParallel 100000 thrpt 50 3373.367 ± 211.108 ops/s
FilterBenchmark.multipleFiltersParallel 1000000 thrpt 50 477.870 ± 70.878 ops/s
FilterBenchmark.oldFashionFilters 10 thrpt 50 45874144.758 ± 2210325.177 ops/s
FilterBenchmark.oldFashionFilters 100 thrpt 50 4902625.828 ± 60397.844 ops/s
FilterBenchmark.oldFashionFilters 1000 thrpt 50 662102.438 ± 5038.465 ops/s
FilterBenchmark.oldFashionFilters 10000 thrpt 50 29390.911 ± 257.311 ops/s
FilterBenchmark.oldFashionFilters 100000 thrpt 50 1999.032 ± 6.829 ops/s
FilterBenchmark.oldFashionFilters 1000000 thrpt 50 200.564 ± 1.695 ops/s
Java 12
# Run complete. Total time: 04:36:20
Benchmark (arraySize) Mode Cnt Score Error Units
FilterBenchmark.complexFilter 10 thrpt 50 10338525.553 ? 1677693.433 ops/s
FilterBenchmark.complexFilter 100 thrpt 50 4381301.188 ? 287299.598 ops/s
FilterBenchmark.complexFilter 1000 thrpt 50 607572.430 ? 9367.026 ops/s
FilterBenchmark.complexFilter 10000 thrpt 50 30643.286 ? 472.033 ops/s
FilterBenchmark.complexFilter 100000 thrpt 50 1450.341 ? 3.730 ops/s
FilterBenchmark.complexFilter 1000000 thrpt 50 138.996 ? 2.052 ops/s
FilterBenchmark.complexFilterParallel 10 thrpt 50 21289.444 ? 183.245 ops/s
FilterBenchmark.complexFilterParallel 100 thrpt 50 20105.239 ? 124.759 ops/s
FilterBenchmark.complexFilterParallel 1000 thrpt 50 19418.830 ? 141.664 ops/s
FilterBenchmark.complexFilterParallel 10000 thrpt 50 13874.585 ? 104.418 ops/s
FilterBenchmark.complexFilterParallel 100000 thrpt 50 5334.947 ? 25.452 ops/s
FilterBenchmark.complexFilterParallel 1000000 thrpt 50 781.046 ? 9.687 ops/s
FilterBenchmark.multipleFilters 10 thrpt 50 5460308.048 ? 478157.935 ops/s
FilterBenchmark.multipleFilters 100 thrpt 50 2227583.836 ? 113078.932 ops/s
FilterBenchmark.multipleFilters 1000 thrpt 50 287157.190 ? 1114.346 ops/s
FilterBenchmark.multipleFilters 10000 thrpt 50 16268.016 ? 704.735 ops/s
FilterBenchmark.multipleFilters 100000 thrpt 50 1531.516 ? 2.729 ops/s
FilterBenchmark.multipleFilters 1000000 thrpt 50 123.881 ? 1.525 ops/s
FilterBenchmark.multipleFiltersParallel 10 thrpt 50 20403.993 ? 147.247 ops/s
FilterBenchmark.multipleFiltersParallel 100 thrpt 50 19426.222 ? 96.979 ops/s
FilterBenchmark.multipleFiltersParallel 1000 thrpt 50 17692.433 ? 67.606 ops/s
FilterBenchmark.multipleFiltersParallel 10000 thrpt 50 12108.482 ? 34.500 ops/s
FilterBenchmark.multipleFiltersParallel 100000 thrpt 50 3782.756 ? 22.044 ops/s
FilterBenchmark.multipleFiltersParallel 1000000 thrpt 50 589.972 ? 71.448 ops/s
FilterBenchmark.oldFashionFilters 10 thrpt 50 41024334.062 ? 1374663.440 ops/s
FilterBenchmark.oldFashionFilters 100 thrpt 50 6011852.027 ? 246202.642 ops/s
FilterBenchmark.oldFashionFilters 1000 thrpt 50 553243.594 ? 2217.912 ops/s
FilterBenchmark.oldFashionFilters 10000 thrpt 50 29188.753 ? 580.958 ops/s
FilterBenchmark.oldFashionFilters 100000 thrpt 50 2061.738 ? 8.456 ops/s
FilterBenchmark.oldFashionFilters 1000000 thrpt 50 196.105 ? 3.203 ops/s