Misc tips and notes
In theory, can edit .theanorc (in home folder): change cnmem to whatever % of memory you want your process to use. (In practice, this doesn't seem to work terribly well in limiting memory; analyses seem to take as much memory as they need if they need more than has been allocated.)
nvidia-smi shows current load statistics for all your GPUs (as well as stuff like temperature and fan speed)
To alter Theano configuration just for the current run (i.e. without editing .theanorc), prepend your command with THEANO_FLAGS, e.g.: THEANO_FLAGS=lib.cnmem=.40 /opt/anaconda/bin/python3 my_analysis_script.py