OpenGL itself is written for C/C++ programmers, and is in fact more efficiently implemented in that language. Serious games and animations are written that way. However, you can still do snazzy things with it from Java using the Java OpenGL (JOGL) library and the associated JNI/utilities library GLU. Perhaps an encouraging fact is that JOGL is the result of a joint effort between Sun Microsystems (makers of Java) and SGI (makers of OpenGL) to produce Java bindings and extensions for OpenGL. I also looked at Maya (the top GUI-oriented 3D package out there) and various incarnations of JOGL and Java 3D (native to the JDK), but decided that to learn the most about OpenGL, I should learn to code everything from scratch. Also considered was DirectX; I discourage this because it is targeted at Windows systems and thus not as portable as OpenGL is.
To get started, you need JOGL binaries (JARs) that you can download from java.net. Choose the apprpriate package for your OS and extract it to a location on your computer. For Windows, copy all DLLs into the %system32% directory, and the *.jar files into a location that your IDE can access to include in compile and build classpaths.
If you'll be using Eclipse, they claim to support OpenGL out-of-the-box (since Callisto, v3.2), but I couldn't get a simple application going until I added the JOGL libraries to the project. If you use NetBeans, there is a plugin for OpenGL available through the Update Center, but I couldn't get that going as well. So I decided to do things manually i.e. copy DLLs to appropriate location, then manually include JOGL libraries to the project path.
Here's the source code of a template you can use to start OpenGL projects. Included is a class (JoglTester) that tests whether your DLLs loaded correctly, and whether the core library is accessible. It'll also open a window (JoglApp) with dots of colors across the application, demonstrating how a canvas can be initialized for display. It is a good idea to have your own event handler that implements a GL listener (JoglGLEventListener). Most of your code changes will happen in the listener's overridden methods.
This work is based on Gene Davis' article at Java World. He does a good job of maintaining his code samples there (see comments/updates at bottom of article). When you set up your projects, create them as "Java Applications".