Organizational usability is about the match between the user and the system, between the organization and the system, and between the environment and the system. While the first of these matches can, to a large extent, be evaluated in the lab, the two others cannot. Organizational usability must instead be evaluated in situ, that is, while the system is used for real work. We propose three contexts for such evaluation: pilot implementation, technochange, and design in use. Pilot implementation aims to inform the finalization of a system on the basis of testing it in the field prior to go-live. Technochange focuses on shaking down a system during go-live to realize the benefits it was developed to help achieve. Design in use is the tailoring performed by users after go-live to fit a system and its use to their local and emerging needs. For each evaluation context we describe its aim and scope, provide a brief example, and discuss the challenges it presents. To strengthen the focus on evaluation in the three contexts we propose the measurement of specified effects, combined with a sensitivity toward emergent effects. Incorporating effects in the evaluation of organizational usability makes for working systematically toward realizing benefits from the use of a system.