This paper presents a concept for building a benchmark dataset to systematically evaluate chatbot responses in e-commerce with respect to ethical quality dimensions such as bias, toxicity and personalization. The core approach involves generating neutral base dialogues of varying lengths, which are then expanded into more problematic variants and enriched with linguistic diversity. The data is annotated by humans through a multi-stage process. The resulting dataset is intended to support the development and calibration of ethical AI components. To increase utility an english translation of the dataset is provided.