There is a growing number of large scale cross-site database collection of resting-state functional magnetic resonance imaging (rs-fMRI) for studying neurobehavioral diseases, such as ADHD. Although a large amount of data benefits machine learning-based classification methods, the idiosyncratic variability of each site can deteriorate cross-site generalization ability. This challenge creates a bottleneck in requiring a large number of labeled samples of each site. Hence in this research, we utilize an approach of conditional adversarial domain adaptation network (CDAN) to learn a discriminative fMRI representation that is site-invariant for unsupervised transfer of ADHD classification. We evaluate our framework on a multi-site ADHD dataset and achieve improvement in transferring between sites. Further visualization reveals that there indeed exists a substantial site discrepancy and statistically analysis indicates that male's rs-fMRI could be more vulnerable toward site-specific effects.