Visual appeal (VA), as judged by users accessing different websites, has been shown to impact overall user preferences and satisfaction. In the context of Web Quality of Experience (QoE), we hypothesize that VA has a non-trivial impact on end user quality perception and hence needs to be accounted for when modeling overall Web QoE. In this paper, we aim to quantify the impact of VA factors on perceived visual design quality for different types of tested websites. We carried out two large-scale (>350 users each) crowd-sourced campaigns to test the influence of several factors often found in design best practices recommendations. While results have identified color goodness and font goodness as Key Influence Factors (KIF) for VA, an analysis of user demographics has further shown a strong impact of user’s origin on subjective ratings. By quantifying the impact of identified KIFs on VA, we provide guidelines for successful VA manipulations to be considered in future Web QoE subjective tests.