Parkinson's disease (PD) is one of the most severe and common disease globally. PD induces motor system impairment causing symptoms such as shaking, rigidity, slowness of movement, body tremor and difficulty with walking. Clinically, accurately and objectively assessing the severity of PD symptoms is critical in controlling appropriate dosage of Levodopa to prevent unwanted side effect of switching between Dyskinesia and PD. The unified Parkinson's disease rating scale published by the Movement Disorder Society (MDS-UPDRS) is an validated instrument regularly administrated by trained physician to assess the severity of a PD patient's motor disorder. In this work, we aim at advancing vision-based automatic motor disorder assessment, specifically hand tremor and movement, for PD patients during UPDRS. Our proposed method leverages information across the two behavior tasks simultaneously via deep joint training to improve each single task's, i.e., tremor and movement, severity classification rate. We evaluate our framework on a large cohort of 106 PD patients, and with our proposed deep joint training framework, we achieve accuracy of 78.01% and 80.60% in right and left hand movement binary classification; in terms of tremor severity classification, our approach obtains an enhanced recognition rates of 72.20% and 71.10% for right and left hand respectively.