Efficiently compute Intersection of two Sets in Java?

Run some tests with the posted approach and versus constructing a new HashSet. That is, let A be the smaller of the sets and B be the larger set and then, for each item in A, if it also exists in B then add it to C (a new HashSet) -- for just counting, the intermediate C set can be skipped.

Just as the posted approach, this should be a O(|A|) in cost as the iteration is O(|A|) and probe into B is O(1). I have no idea how it will compare vs. the clone-and-remove approach.

Happy coding -- and post some results ;-)


Actually, on further thinking, I believe this has slightly better bounds than the method in the post: O(|A|) vs O(|A| + |B|). I have no idea if this will make any difference (or improvement) in actuality and I would only expect it to be relevant when |A| <<< |B|.


Okay, so I was really bored. At least on JDK 7 (Windows 7 x64), it appears the method in presented in the post is slower than the above approach -- by a good (albeit what appears to be mostly constant) factor. My eye-ball guesstimate says it is about four times as slow than the above suggestion that just uses a counter and twice as slow of when creating a new HashSet. This seems to be "roughly consistent" across the different initial set sizes.

(Please keep in mind that, as Voo pointed out, the numbers above and this micro-benchmark assume a HashSet is being used! And, as always, there are dangers with micro-benchmarks. YMMV.)

Here are the ugly results (times in milliseconds):

Running tests for 1x1
IntersectTest$PostMethod@6cc2060e took 13.9808544 count=1000000
IntersectTest$MyMethod1@7d38847d took 2.9893732 count=1000000
IntersectTest$MyMethod2@9826ac5 took 7.775945 count=1000000
Running tests for 1x10
IntersectTest$PostMethod@67fc9fee took 12.4647712 count=734000
IntersectTest$MyMethod1@7a67f797 took 3.1567252 count=734000
IntersectTest$MyMethod2@3fb01949 took 6.483941 count=734000
Running tests for 1x100
IntersectTest$PostMethod@16675039 took 11.3069326 count=706000
IntersectTest$MyMethod1@58c3d9ac took 2.3482693 count=706000
IntersectTest$MyMethod2@2207d8bb took 4.8687103 count=706000
Running tests for 1x1000
IntersectTest$PostMethod@33d626a4 took 10.28656 count=729000
IntersectTest$MyMethod1@3082f392 took 2.3478658 count=729000
IntersectTest$MyMethod2@65450f1f took 4.109205 count=729000
Running tests for 10x2
IntersectTest$PostMethod@55c4d594 took 10.4137618 count=736000
IntersectTest$MyMethod1@6da21389 took 2.374206 count=736000
IntersectTest$MyMethod2@2bb0bf9a took 4.9802039 count=736000
Running tests for 10x10
IntersectTest$PostMethod@7930ebb took 25.811083 count=4370000
IntersectTest$MyMethod1@47ac1adf took 6.9409306 count=4370000
IntersectTest$MyMethod2@74184b3b took 14.2603248 count=4370000
Running tests for 10x100
IntersectTest$PostMethod@7f423820 took 25.0577691 count=4251000
IntersectTest$MyMethod1@5472fe25 took 6.1376042 count=4251000
IntersectTest$MyMethod2@498b5a73 took 13.9880385 count=4251000
Running tests for 10x1000
IntersectTest$PostMethod@3033b503 took 25.0312716 count=4138000
IntersectTest$MyMethod1@12b0f0ae took 6.0932898 count=4138000
IntersectTest$MyMethod2@1e893918 took 13.8332505 count=4138000
Running tests for 100x1
IntersectTest$PostMethod@6366de01 took 9.4531628 count=700000
IntersectTest$MyMethod1@767946a2 took 2.4284762 count=700000
IntersectTest$MyMethod2@140c7272 took 4.7580235 count=700000
Running tests for 100x10
IntersectTest$PostMethod@3351e824 took 24.9788668 count=4192000
IntersectTest$MyMethod1@465fadce took 6.1462852 count=4192000
IntersectTest$MyMethod2@338bd37a took 13.1742654 count=4192000
Running tests for 100x100
IntersectTest$PostMethod@297630d5 took 193.0121077 count=41047000
IntersectTest$MyMethod1@e800537 took 45.2652397 count=41047000
IntersectTest$MyMethod2@76d66550 took 120.8494766 count=41047000
Running tests for 100x1000
IntersectTest$PostMethod@33576738 took 199.6269531 count=40966000
IntersectTest$MyMethod1@2f39a7dd took 45.5255814 count=40966000
IntersectTest$MyMethod2@723bb663 took 122.1704975 count=40966000
Running tests for 1x1
IntersectTest$PostMethod@35e3bdb5 took 9.5598373 count=1000000
IntersectTest$MyMethod1@7abbd1b6 took 2.6359174 count=1000000
IntersectTest$MyMethod2@40c542ad took 6.1091794 count=1000000
Running tests for 1x10
IntersectTest$PostMethod@3c33a0c5 took 9.4648528 count=733000
IntersectTest$MyMethod1@61800463 took 2.302116 count=733000
IntersectTest$MyMethod2@1ba03197 took 5.4803628 count=733000
Running tests for 1x100
IntersectTest$PostMethod@71b8da5 took 9.4971057 count=719000
IntersectTest$MyMethod1@21f04f48 took 2.2983538 count=719000
IntersectTest$MyMethod2@27e51160 took 5.3926902 count=719000
Running tests for 1x1000
IntersectTest$PostMethod@2fe7106a took 9.4702331 count=692000
IntersectTest$MyMethod1@6ae6b7b7 took 2.3013066 count=692000
IntersectTest$MyMethod2@51278635 took 5.4488882 count=692000
Running tests for 10x2
IntersectTest$PostMethod@223b2d85 took 9.5660879 count=743000
IntersectTest$MyMethod1@5b298851 took 2.3481445 count=743000
IntersectTest$MyMethod2@3b4ac99 took 4.8268489 count=743000
Running tests for 10x10
IntersectTest$PostMethod@51c700a0 took 23.0709476 count=4326000
IntersectTest$MyMethod1@5ffa3251 took 5.5460785 count=4326000
IntersectTest$MyMethod2@22fd9511 took 13.4853948 count=4326000
Running tests for 10x100
IntersectTest$PostMethod@46b49793 took 25.1295491 count=4256000
IntersectTest$MyMethod1@7a4b5828 took 5.8520418 count=4256000
IntersectTest$MyMethod2@6888e8d1 took 14.0856942 count=4256000
Running tests for 10x1000
IntersectTest$PostMethod@5339af0d took 25.1752685 count=4158000
IntersectTest$MyMethod1@7013a92a took 5.7978328 count=4158000
IntersectTest$MyMethod2@1ac73de2 took 13.8914112 count=4158000
Running tests for 100x1
IntersectTest$PostMethod@3d1344c8 took 9.5123442 count=717000
IntersectTest$MyMethod1@3c08c5cb took 2.34665 count=717000
IntersectTest$MyMethod2@63f1b137 took 4.907277 count=717000
Running tests for 100x10
IntersectTest$PostMethod@71695341 took 24.9830339 count=4180000
IntersectTest$MyMethod1@39d90a92 took 5.8467864 count=4180000
IntersectTest$MyMethod2@584514e9 took 13.2197964 count=4180000
Running tests for 100x100
IntersectTest$PostMethod@21b8dc91 took 195.1796213 count=41060000
IntersectTest$MyMethod1@6f98c4e2 took 44.5775162 count=41060000
IntersectTest$MyMethod2@16a60aab took 121.1754402 count=41060000
Running tests for 100x1000
IntersectTest$PostMethod@20b37d62 took 200.973133 count=40940000
IntersectTest$MyMethod1@67ecbdb3 took 45.4832226 count=40940000
IntersectTest$MyMethod2@679a6812 took 121.791293 count=40940000
Running tests for 1x1
IntersectTest$PostMethod@237aa07b took 9.2210288 count=1000000
IntersectTest$MyMethod1@47bdfd6f took 2.3394042 count=1000000
IntersectTest$MyMethod2@a49a735 took 6.1688936 count=1000000
Running tests for 1x10
IntersectTest$PostMethod@2b25ffba took 9.4103967 count=736000
IntersectTest$MyMethod1@4bb82277 took 2.2976994 count=736000
IntersectTest$MyMethod2@25ded977 took 5.3310813 count=736000
Running tests for 1x100
IntersectTest$PostMethod@7154a6d5 took 9.3818786 count=704000
IntersectTest$MyMethod1@6c952413 took 2.3014931 count=704000
IntersectTest$MyMethod2@33739316 took 5.3307998 count=704000
Running tests for 1x1000
IntersectTest$PostMethod@58334198 took 9.3831841 count=736000
IntersectTest$MyMethod1@d178f65 took 2.3071236 count=736000
IntersectTest$MyMethod2@5c7369a took 5.4062184 count=736000
Running tests for 10x2
IntersectTest$PostMethod@7c4bdf7c took 9.4040537 count=735000
IntersectTest$MyMethod1@593d85a4 took 2.3584088 count=735000
IntersectTest$MyMethod2@5610ffc1 took 4.8318229 count=735000
Running tests for 10x10
IntersectTest$PostMethod@46bd9fb8 took 23.004925 count=4331000
IntersectTest$MyMethod1@4b410d50 took 5.5678172 count=4331000
IntersectTest$MyMethod2@1bd125c9 took 14.6517184 count=4331000
Running tests for 10x100
IntersectTest$PostMethod@75d6ecff took 25.0114913 count=4223000
IntersectTest$MyMethod1@716195c9 took 5.798676 count=4223000
IntersectTest$MyMethod2@3db0f946 took 13.8064737 count=4223000
Running tests for 10x1000
IntersectTest$PostMethod@761d8e2a took 25.1910652 count=4292000
IntersectTest$MyMethod1@e60a3fb took 5.8621189 count=4292000
IntersectTest$MyMethod2@6aadbb1c took 13.8150282 count=4292000
Running tests for 100x1
IntersectTest$PostMethod@48a50a6e took 9.4141906 count=736000
IntersectTest$MyMethod1@4b4fe104 took 2.3507252 count=736000
IntersectTest$MyMethod2@693df43c took 4.7506854 count=736000
Running tests for 100x10
IntersectTest$PostMethod@4f7d29df took 24.9574096 count=4219000
IntersectTest$MyMethod1@2248183e took 5.8628954 count=4219000
IntersectTest$MyMethod2@2b2fa007 took 12.9836817 count=4219000
Running tests for 100x100
IntersectTest$PostMethod@545d7b6a took 193.2436192 count=40987000
IntersectTest$MyMethod1@4551976b took 44.634367 count=40987000
IntersectTest$MyMethod2@6fac155a took 119.2478037 count=40987000
Running tests for 100x1000
IntersectTest$PostMethod@7b6c238d took 200.4385174 count=40817000
IntersectTest$MyMethod1@78923d48 took 45.6225227 count=40817000
IntersectTest$MyMethod2@48f57fcf took 121.0602757 count=40817000
Running tests for 1x1
IntersectTest$PostMethod@102c79f4 took 9.0931408 count=1000000
IntersectTest$MyMethod1@57fa8a77 took 2.3309466 count=1000000
IntersectTest$MyMethod2@198b7c1 took 5.7627226 count=1000000
Running tests for 1x10
IntersectTest$PostMethod@8c646d0 took 9.3208571 count=726000
IntersectTest$MyMethod1@11530630 took 2.3123797 count=726000
IntersectTest$MyMethod2@61bb4232 took 5.405318 count=726000
Running tests for 1x100
IntersectTest$PostMethod@1a137105 took 9.387384 count=710000
IntersectTest$MyMethod1@72610ca2 took 2.2938749 count=710000
IntersectTest$MyMethod2@41849a58 took 5.3865938 count=710000
Running tests for 1x1000
IntersectTest$PostMethod@100001c8 took 9.4289031 count=696000
IntersectTest$MyMethod1@7074f9ac took 2.2977923 count=696000
IntersectTest$MyMethod2@fb3c4e2 took 5.3724119 count=696000
Running tests for 10x2
IntersectTest$PostMethod@52c638d6 took 9.4074124 count=775000
IntersectTest$MyMethod1@53bd940e took 2.3544881 count=775000
IntersectTest$MyMethod2@43434e15 took 4.9228549 count=775000
Running tests for 10x10
IntersectTest$PostMethod@73b7628d took 23.2110252 count=4374000
IntersectTest$MyMethod1@ca75255 took 5.5877838 count=4374000
IntersectTest$MyMethod2@3d0e50f0 took 13.5902641 count=4374000
Running tests for 10x100
IntersectTest$PostMethod@6d6bbba9 took 25.1999918 count=4227000
IntersectTest$MyMethod1@3bed8c5e took 5.7879144 count=4227000
IntersectTest$MyMethod2@689a8e0e took 13.9617882 count=4227000
Running tests for 10x1000
IntersectTest$PostMethod@3da3b0a2 took 25.1627329 count=4222000
IntersectTest$MyMethod1@45a17b4b took 5.8319523 count=4222000
IntersectTest$MyMethod2@6ca59ca3 took 13.8885479 count=4222000
Running tests for 100x1
IntersectTest$PostMethod@360202a5 took 9.5115367 count=705000
IntersectTest$MyMethod1@3dfbba56 took 2.3470254 count=705000
IntersectTest$MyMethod2@598683e4 took 4.8955489 count=705000
Running tests for 100x10
IntersectTest$PostMethod@21426d0d took 25.8234298 count=4231000
IntersectTest$MyMethod1@1005818a took 5.8832067 count=4231000
IntersectTest$MyMethod2@597b933d took 13.3676148 count=4231000
Running tests for 100x100
IntersectTest$PostMethod@6d59b81a took 193.676662 count=41015000
IntersectTest$MyMethod1@1d45eb0c took 44.6519088 count=41015000
IntersectTest$MyMethod2@594a6fd7 took 119.1646115 count=41015000
Running tests for 100x1000
IntersectTest$PostMethod@6d57d9ac took 200.1651432 count=40803000
IntersectTest$MyMethod1@2293e349 took 45.5311168 count=40803000
IntersectTest$MyMethod2@1b2edf5b took 120.1697135 count=40803000

And here is the ugly (and possibly flawed) micro-benchmark:

import java.util.*;

public class IntersectTest {

    static Random rng = new Random();

    static abstract class RunIt {
        public long count;
        public long nsTime;
        abstract int Run (Set<Long> s1, Set<Long> s2);
    }

    // As presented in the post
    static class PostMethod extends RunIt {
        public int Run(Set<Long> set1, Set<Long> set2) {
            boolean set1IsLarger = set1.size() > set2.size();
            Set<Long> cloneSet = new HashSet<Long>(set1IsLarger ? set2 : set1);
            cloneSet.retainAll(set1IsLarger ? set1 : set2);
            return cloneSet.size();
        }
    }

    // No intermediate HashSet
    static class MyMethod1 extends RunIt {
        public int Run (Set<Long> set1, Set<Long> set2) {
            Set<Long> a;
            Set<Long> b;
            if (set1.size() <= set2.size()) {
                a = set1;
                b = set2;           
            } else {
                a = set2;
                b = set1;
            }
            int count = 0;
            for (Long e : a) {
                if (b.contains(e)) {
                    count++;
                }           
            }
            return count;
        }
    }

    // With intermediate HashSet
    static class MyMethod2 extends RunIt {
        public int Run (Set<Long> set1, Set<Long> set2) {
            Set<Long> a;
            Set<Long> b;
            Set<Long> res = new HashSet<Long>();
            if (set1.size() <= set2.size()) {
                a = set1;
                b = set2;           
            } else {
                a = set2;
                b = set1;
            }
            for (Long e : a) {
                if (b.contains(e)) {
                    res.add(e);
                }           
            }
            return res.size();
        }
    }

    static Set<Long> makeSet (int count, float load) {
        Set<Long> s = new HashSet<Long>();
        for (int i = 0; i < count; i++) {
            s.add((long)rng.nextInt(Math.max(1, (int)(count * load))));                     
        }
        return s;
    }

    // really crummy ubench stuff
    public static void main(String[] args) {
        int[][] bounds = {
                {1, 1},
                {1, 10},
                {1, 100},
                {1, 1000},
                {10, 2},
                {10, 10},
                {10, 100},
                {10, 1000},
                {100, 1},
                {100, 10},
                {100, 100},
                {100, 1000},
        };
        int totalReps = 4;
        int cycleReps = 1000;
        int subReps = 1000;
        float load = 0.8f;
        for (int tc = 0; tc < totalReps; tc++) {
            for (int[] bound : bounds) {
                int set1size = bound[0];
                int set2size = bound[1];
                System.out.println("Running tests for " + set1size + "x" + set2size);               
                ArrayList<RunIt> allRuns = new ArrayList<RunIt>(
                        Arrays.asList(
                                new PostMethod(),
                                new MyMethod1(),
                                new MyMethod2()));
                for (int r = 0; r < cycleReps; r++) {
                    ArrayList<RunIt> runs = new ArrayList<RunIt>(allRuns);
                    Set<Long> set1 = makeSet(set1size, load);
                    Set<Long> set2 = makeSet(set2size, load);
                    while (runs.size() > 0) {
                        int runIdx = rng.nextInt(runs.size());
                        RunIt run = runs.remove(runIdx);
                        long start = System.nanoTime();
                        int count = 0;
                        for (int s = 0; s < subReps; s++) {
                            count += run.Run(set1, set2); 
                        }                       
                        long time = System.nanoTime() - start;
                        run.nsTime += time;
                        run.count += count;
                    }
                }
                for (RunIt run : allRuns) {
                    double sec = run.nsTime / (10e6);
                    System.out.println(run + " took " + sec + " count=" + run.count);
                }
            }
        }       
    }
}

Just use Google Guava's Sets#intersection(Set, Set) method.