http://www.salesforce.com/us/developer/docs/apexcode/Content/apex_classes_keywords_sharing.htm

Let's see if I can summarize:

System Context: the code has full access to all records and fields in the system.

User Context: the code can only access records (and possibly fields, such as executeAnonymous calls) available to the user.

Without Sharing: the code runs in system context; all records and fields are available.

With Sharing: the code runs in user context; DML operations may fail because of sharing settings, and queries will only return values available to the user.

The default mode for most code transactions is "without sharing" (meaning, the code is running in system context). This means the user may be able to see data they could not normally see, or update records not normally accessible to them. A class without a sharing modifier will take on the sharing of the current context (meaning the code may be running in "with sharing" or "without sharing" depending on the source). A class with a modifier will cause the sharing mode to change as defined while inside that class.

For example:

public with sharing class A {
  public static void B {
    System.debug('I am in sharing mode.');
    C D = new C();
    c.E();
    F G = new F();
    G.I();
  }
}

public without sharing class C {
  public void E() {
    System.debug('I am not in sharing mode.');
    F H = new F();
    H.I();
  }
}

public class F {
  public void I() {
    System.debug('I might be in sharing mode (see callee).');
  }
}

Note that class F is "confused." It doesn't know (nor can know) if sharing is applied to it. It is best to avoid ambigious situtions by specifically marking a class as one or the other. However, this can be useful if, for example, you have a utility class that might be called from a trigger (and thus needs no sharing) or might be called from a Visualforce page (in which case, it may need sharing). This is a flexibility feature that allows reusable code, but should be used sparingly. In almost all cases, "with sharing" should be used if the user would be able to otherwise access data they shouldn't, and rarely should "without sharing" be used, but keep in mind that this mode has fewer constraints and therefore runs faster, so if you can avoid using sharing, you can make the application faster (but you shouldn't do so at the expense of security).

Just so we're clear, the example class you mentioned is the same as F (above). It doesn't know if it's in sharing mode, and will either be sharing-enabled or sharing-disabled depending on the caller.