You have to setup the standalone unit ahead of time.
In Apogee Control > I/O Page > You will see the Routing area to choose what goes to each output.
You can see the Analog Outputs, Guitar Outputs and SPDIF Coax across the top row, and Optical Outputs across the bottom.
Thus you can choose what is routed to each output.
In a standard Standalone AD/DA converter setup you would route the Analog Inputs to Optical Outputs and vice versa.
So on Optical Outs choose:
Hardware Inputs Analog 1-2 > Optical Out 1-2
Hardware Inputs Analog 3-4 > Optical Out 3-4
Hardware Inputs Analog 5-6 > Optical Out 5-6
Hardware Inputs Analog 7-8 > Optical Out 7-8
This gives you the A/D, Analog inputs to your Optical Outs on the Ensemble, so you can connect it to another unit with Optical Inputs.
For D/A go to the upper Analog Outs section and choose:
Hardware Inputs Optical 1-2 > Analog 1-2
Hardware Inputs Optical 3-4 > Analog 3-4
Hardware Inputs Optical 5-6 > Analog 5-6
Hardware Inputs Optical 7-8 > Analog 7-8
Also, be sure if you wish to send D/A and have full line out that you set the Main Output Format to Line, so it is not attenuated by the Main Out Knob.