There are two positions in the classic debate regarding musical emotion: the cognitivist position and the emotivist position. According to the cognitivist position, music expresses emotion but does not induce it in listeners. So, listeners may recognize emotion in music without feeling it, unlike real, everyday emotion. According to the emotivist position, listeners not only recognize emotion but also feel it. This is supported by their physiological responses during music listening, which are similar to responses occurring with real emotion. When listeners provide emotion appraisals, if the cognitivist position were true, then these appraisals might be based on audio features in the music. However, if the emotivist position were true, then appraisals would be based on the emotion experienced by listeners as opposed to what they perceived in the audio features. We propose a hypothesis combining both positions according to which, listeners make emotion appraisals based on a combination of what they perceive in the music as well as what they experience during the listening process. In this paper, we explore all three positions using connectionist prediction models, specifically four different neural networks: (a) using only audio features as input, (b) using only physiological features as input, (c) using both audio and physiological features as input, and (d) using a committee machine that combines contributions from an audio network and a physiology network. We examine the performance of these networks and discuss their implications as possible cognitive models of emotion appraisal within listeners.