Automatic anomaly detection on engineering structures is often carried out using supervised models, raising the issue of anomalous images acquisition and annotation. Unsupervised methods like normalizing flows achieve excellent results while trained with defect-free images only. However, normalizing flows methods, such as MSFlow, are generally applied on features extracted by an encoder pre-trained on datasets that may not be related to engineering structures images. Therefore, we investigate the possibility to derive more discriminative features with an additional fine-tuning of the feature extractor on images with synthetic anomalies. We consider two types of such anomalies and demonstrate their efficiency with MSFlow on the MVTec (Wood/Tile) and Crack500 datasets, with significantly improved predictions. Interestingly, both tasks produce similar results suggesting that pre-training is mainly improved by the healthy part of images and not very sensitive to anomaly realism. Additionally, when comparing our fine-tuned MSFlow with a reference supervised model, CT-CrackSeg, on the Crack500 dataset, we observe similar qualitative behaviours. This open a promising direction towards annotation-free, more scalable alternatives, in particular for anomaly detection in engineering structure applications.