My first impression when manipulating my BeagleBone Black was a mix of excitation and a pinch of doubt. Excitation, when seeing the possibilities of such a small piece of hardware. Doubt, when getting in touch with the default Angstrom OS - not bad per se, but leaving a disappointing feeling of sluggishness when it came to X applications.
So I went on trying and installing several versions of Linux: an up-to-date Angstrom, several Debian builds, Ubuntu... Good, but still slow. The use of hardware acceleration was a theorical possibility, but at the time, noone seemed to have managed to get it working.
Rather than wait for a hypothetical saviour, I decided to go the other way around: is X too slow? Drop X. I would focus on the framebuffer device: a few tests managed to convince me that this was viable performance-wise, and lots of emulators are based on SDL, which allows fb as its device.
So, in the end, I opted for a Debian image: BBB-eMMC-flasher-debian-7.4-2014-03-04-2gb.img. You can check for more recent images at http://elinux.org/BeagleBoardDebian#eMMC:_BeagleBone_Black , but my experience is on this precise build only. For the install procedure on the eMMC, just check the official wiki: http://www.crashcourse.ca/wiki/index.php/BBB_software_update_process.
I removed the following packages which were obviously useless (use apt-get or aptitude):