Can I escape Kerbin's gravitational pull?
Technically, you can escape Kerbin's gravity at any height, as long as you're going fast enough.
To orbit the planet at any given altitude, you only need to be travelling at the corresponding horizontal speed, as shown in the table below. The only reason that you can't orbit the planet consistently at low altitudes is that it's difficult to maintain your orbital velocity in the presence of atmospheric drag.
For example, to enter an orbit at 100km above the surface, you need a horizontal speed of 2245.8 meters per second.
Kerbin's atmosphere tapers off at approximately 68-70 km above the surface. Any orbit below this altitude will require corrections due to atmospheric drag, or else your ship will begin losing orbital speed and eventually deorbit. You can also do this intentionally if you want to deorbit your spacecraft - reduce your speed enough to drop below the minimum horizontal speed given in the table, and your spaceship will begin to descend.
I've snipped portions of this table to save space. The full thing can be found on the Kerbal Space Program Wiki:
Altitude (m) Horizontal Speed (m/s) Orbital Period (min)
35000 2357.9 28.20
40000 2348.7 28.54
45000 2339.6 28.87
50000 2330.6 29.21
55000 2321.7 29.54
60000 2312.8 29.88
65000 2304.1 30.22
70000 2295.5 30.56
75000 2287.0 30.91
80000 2278.6 31.25
85000 2270.3 31.60
90000 2262.0 31.94
95000 2253.9 32.29
100000 2245.8 32.64
110000 2229.9 33.34
120000 2214.4 34.05
130000 2199.2 34.76
140000 2184.3 35.48
150000 2169.6 36.20
160000 2155.3 36.93
170000 2141.3 37.66
180000 2127.5 38.39
190000 2114.0 39.13
200000 2100.7 39.88
300000 1980.6 47.59
400000 1879.0 55.73
500000 1791.5 64.30
600000 1715.3 73.26
700000 1648.0 82.61
800000 1588.0 92.32
900000 1534.2 102.39
1000000 1485.5 112.79
2,868,378 1008.9 6 hours
8,140,000 635.4 24 hours
Once you hit roughly 100 km above Kerbin, the effect that gravity plays on your ship is minimal. Problem is, once you leave the planet's sphere of influence, the sun will continue to exert it's influence on you.
It is (remotely) possible to leave the solar system, but most of it just involves adding as much fuel as you can, and building the most momentum possible. Once you run out of fuel, just turn up the game speed, and perhaps you got enough to leave the solar system. If not, your Kerbal is doomed to orbit the sun. Forever.
For an amusing twist (since Kerbals are all about fun), if you manage to fail spectacularly enough, it's possible to have it orbit the planet, and you can leave him up there if you so choose. He then becomes a navigation hazard for subsequent launches.
Just as a side note about the last post talking about orbit. It's never optimal to go full blast out off the earth once oyu have the capability of using liquid fuel over solid fuel boosters. 300m/s to 15,000m, then 45* at the same speed till 50,000m. You will practically be in a full orbit by that point and you will need to make anywhere from a 5-20 second burn depending on how well you executed the launch. But more engines and more speed is almost never better in KSP.