Fast or Focused? Making Low Latency Accurate