Breast cancer is one of the most common occurring cancers in women. Residual cancer burden measures tumor response to the therapy and is shown to be prognostic for long term survival. The percentage of invasive or in situ tumor cellularity is an important component of tumor burden assessment. In the current clinical practice, tumor cellularity is manually estimated by pathologists on hematoxylin and eosin (H&E) stained slides. In this paper, we present a deep ordinal regression framework to automatically assess cellularity from pathological images. We formulate the cellularity assessment as an ordinal regression problem and address by an end-to-end learning approach using deep convolutional neural networks. We evaluated the proposed methods on the SPIE BreastPathQ challenge dataset and achieved significant higher agreement with expert pathologist scoring in terms of intraclass correlation (ICC) of 0.94 (vs. 0.89 of the inter-rater agreements between pathologists).