Smartphone adoption has increased significantly and, with the increase in smartphone capabilities, this means that users can access the Internet, communicate, and entertain themselves anywhere and anytime. However, there is growing evidence of problematic use of smartphones that impacts both social and heath aspects of users' lives. Currently, assessment of overuse or problematic use depends on one-time, self-reported behavioral information about phone use. Due to the known issues with self-reports in such types of assessments, we explore an automated, objective and repeatable approach for assessing problematic usage. We collect a wide range of phone usage data from smartphones, identify a number of usage features that are relevant to this assessment, and build detection models based on Adaboost with machine learning algorithms automatically detecting problematic use. We found that the number of apps used per day, the ratio of SMSs to calls, the number of event-initiated sessions, the number of apps used per event initiated session, and the length of non-event-initiated sessions are useful for detecting problematic usage. With these, a detection model can identify users with problematic usage with 89.6% accuracy (F-score of .707).