Flexibility of Closure is known as a subordinated ability of visual processing, which is one dimension of human cognitive abilities. Despite its importance as an ability within intelligence, there is a lack of psychometric research on it. Hence, the aim of this study was to generate a pilot version of a Flexibility of Closure test using automatic item generation and to examine item features concerning their psychometric properties and impact on item difficulties.Based on previous research, items for Flexibility of Closure were automatically generated using the dual-component method. In the first component, item features which may contribute to item difficulty were defined. The second component included constraints to avoid test bias. Test results were analyzed with classical test theory as well as item response theory, using a dichotomous Rasch model function for analysis.Out of a pool of 3240 items, 18 items were selected based on (1) a systematic selection according to item features and (2) a subjective assessment of required constraints. The online version of the test was completed by 322 people of whom 50% solved more than 13 items; the average test time was 12 minutes. The Cronbachs value for reliability was .77 for the 18 selected items, indicating an adequate measurement precision. The test did not show a fit on the dichotomous Rasch model function of the item response analysis, not even after excluding five of the original 18 items due to unsatisfying item properties. Rasch homogeneity could not be assumed for the test items. Even though the data of this pilot study showed no fit on the Rasch model, a first step in constructing a Flexibility of Closure test with automatic item generation was taken. Further research is needed to determine how item features contribute to item difficulty and how to adapt the items for a fit on the Rasch model.