CAA Survey

Computer-Assisted Assessment (CAA)

Full Edition of National Survey Questionnaire 2004

Thank you for taking the time to fill in this questionnaire. All responses will be kept confidential, and the raw data deleted as soon as it has been analysed. If you have any queries about the questions, please contact us on 023 8059 2326 or email wiw@soton.ac.uk.
If you submit a completed questionnaires before the end of August 2004 we will enter your e-mail address into a draw for two £25 gift vouchers.

Personal and Professional Details (optional)

Email address

Job title

Subject specialism, if relevant

(a) Which of the following describe your position? (Please select all that apply)
Part-time Full-time

(b) Role
Academic Pedagogic support ICT support

(c) Contract type
Permanent Short-term contract Hourly paid

(d) Gender
Male Female

(e) Age
21-30 31-40 41-50 51-60 61-70+

(f) Highest qualification held

1. Which of the following statements most accurately describes your experience with CAA?
     a) I currently use CAA, advise on the use of CAA or support CAA
     b) do not use CAA, advise on the use of CAA or support CAA
     c) I have used, advised on, or supported CAA in the past, but no longer do so.

2. Do you intend to use CAA in the future?    Yes   No

3. If you teach, are you currently using computer-aided learning (CAL) in your teaching, excluding word processing and spreadsheets?
Yes No

4. Does your institution provide CAA support for academics? Yes No

(a) If so, where is this located? Please select all that apply.
As a central unit Within faculties Within academic departments

Elsewhere (please specify)

(b) Does your institution have a dedicated CAA officer(s)?
Yes No

(c) Does your institution have a dedicated CAA department?
Yes No

5. If you teach, how likely would you be to use CAA IF …

Please select from the choices presented:

a) You had to create a test AND computerise it YOURSELF	very unlikely	unlikely	neutral	likely	very likely
b) You created a test BUT SOMEONE ELSE computerised it for you	very unlikely	unlikely	neutral	likely	very likely
c) Tests were available BUT you had to computerise them YOURSELF	very unlikely	unlikely	neutral	likely	very likely
d) Tests were available BUT SOMEONE ELSE computerised them for you	very unlikely	unlikely	neutral	likely	very likely
e) Appropriate computerised tests were already available for use	very unlikely	unlikely	neutral	likely	very likely

6. In your view, does CAA save academics’ time? Yes No

     a) If 'yes', has this meant a significant shift in the workloads of other staff?
          Yes No

     b) Could you say for whom, how and roughly the number of extra hours per week?

7. a) Please list any advantages to using CAA that you have identified

7. b) Please list any disadvantages to using CAA that you have identified

8. What do you see as the critical success factors for the implementation of CAA?
     a) At the level of the individual academic

     b) At the level of the institution

9. What do you see as the main obstacles to the successful implementation of CAA?
     a) At the level of the individual academic

     b) At the level of the institution

10. What do you perceive to be the attitude of each of the following groups towards CAA?

a) Students	very negative	negative	neutral	positive	very positive
b) Academic staff	very negative	negative	neutral	positive	very positive
c) Senior management	very negative	negative	neutral	positive	very positive
d) External examiners	very negative	negative	neutral	positive	very positive
e) Technical staff	very negative	negative	neutral	positive	very positive

11. Would any of the following be useful in supporting your use of CAA?
      (For each, please indicate how)

      a) Staff development
      b) Institutional support
      c) National support
      d) Hardware provision
      e) Software provision

12. What future developments in CAA would you like to see?

13. If you CURRENTLY USE CAA, please complete the rest of the questionnaire.
IF YOU DO NOT, please scroll down to Q.23 and the Survey Feedback Details section at the bottom.

14. a) When did you start using CAA?

      Before 1985 - 1989      1990 - 1994      1995 - 1999      2000 - present

      b) Can you say why you started using CAA?

15. a) Do you use question banks?     Yes       No

      b) If so, can you give any details?


16. a) Would you be interested in sharing questions with academics at other institutions?
          Yes      No

      b) If 'yes', can you give any details?


17. a) Have you evaluated your students’ use of CAA?      Yes     No

      b) If so, can you give any details?

18. a) Have you ever used CAA with students with special needs?
           Yes      No

      b) If so, can you give any details?

19. Please state the approximate number of computer-assisted assessments which you have delivered this year.

0-5 6-10 11-20 21-30 31-50 51-100 >100

a) If you have delivered computer-assisted assessments this year, please say briefly what the subjects were

     b) How many assessments were at the level of:

         (i) Undergraduate Year 1?
        (ii) Undergraduate Year 2?
       (iii) Undergraduate Year 3 (or above)
       (iv) Postgraduate level?

     c) How many assessments were:

         (i) Diagnostic - to identify pre-course knowledge?
        (ii) Formative - to assist in the learning process?
       (iii) Summative - to assess post-course knowledge?
       (iv) Other - please specify

     d) How many assessments were invigilated?

    e) Which question types have you used this year?

         Multiple-choice
         Multiple-response
         Text input
         Numeric input
         Graphical hot spots
         Audio
         Video
         Other

     If other question types were used, could you say what kind?

f) Please indicate the typical class size for CAA assessments?

g) What other kinds of assessment are used in the modules?

         Essay – coursework
         Essay – exam
         Short answer questions
         Lab report
         Presentation
         Objective test
         Worksheets
         Other

         If Other, then please specify what

         (h) Please indicate the typical weighting (%) of the CAA within the modules? %

20. Do you receive support from any of the following for your use of CAA?

      Please specify what form this support takes, e.g. funding, time released, staff, additional hardware/software resources.

      a) Department


      b) Faculty


      c) Institution


      d) External

      e) Other


21. What CAA software packages are you using?
     Please list the packages and indicate which delivery method they relate to (summative/formative etc)

22. What do you see as the particular advantages and disadvantages of the CAA software you use?

23. Which of the following statements describe the experience of the staff at your institution with CAA?
         A significant percentage (20% or more) currently uses CAA
         Only a small group of enthusiasts use CAA
         No-one uses CAA
         Some have used CAA in the past, but no longer do so
         Some may be using it, but I am not aware of their use

24. a) Does your institution already provide an institution-wide Virtual Learning Environment (VLE) such as Blackboard or WebCT?
           No        Blackboard         WebCT       Other VLE

      b) If so, do you think it has:
          Encouraged people to use CAA
          Discouraged people from using CAA
          Made no discernable difference to the uptake of CAA

      c) Would you like to be able to export quizzes from your VLE to a CAA system?
          Yes No

      d) What impact is the VLE making on the way you use CAA?

25. Interoperability standards:
      IMS Question and Test Interoperability (QTI) is one specification for a standard way of sharing assessment data.
      IMS QTI is designed to make it easier to transfer information such as questions, tests and results between different software applications.

     a) Are you already aware of CAA interoperability standards such as the IMS QTI specification?
       Very aware
       Moderately aware
       Not really aware

     b) How important to you is the ability to ‘future proof’ CAA interoperability,
         such as the ability to transfer questions and assessments from one CAA system to another?
        Very important
        Moderately important
        Not really important
        Not at all important

     c) What impact is interoperability making on the way you use CAA?

26. To what extent do you agree with the following statements?
Please choose from the choices presented:
SD=strongly disagree, D=disagree, N= neutral, A=agree, SA=strongly agree.

a) CAA provides statistical analysis of examination questions (e.g. facility and discrimination)	SD	D	N	A	SA
b) CAA providesa wide range of reports analysing student performance	SD	D	N	A	SA
c) CAA exams entail fewer security risks than paper exams	SD	D	N	A	SA
d) CAA is more costly than paper-based assessment methods	SD	D	N	A	SA
e) CAA is more time-consuming than paper-based assessment methods	SD	D	N	A	SA
f) Lower order learning (e.g. knowledge and comprehension) can be tested with objective tests	SD	D	N	A	SA
g) Higher order learning (e.g. critical analysis and evaluation) can be tested with objective tests	SD	D	N	A	SA
h) Computerised marking saves academic staff a significant amount of time	SD	D	N	A	SA
i) Training in CAA software and pedagogy should be available centrally within an institution	SD	D	N	A	SA
j) Support staff should be solely responsible for the technical and operational aspects of CAA	SD	D	N	A	SA
k) CAA offers greater objectivity in marking than tutor-marked assessments	SD	D	N	A	SA
l) Student anxiety about using computers is a significant problem with CAA	SD	D	N	A	SA
m) Academic staff anxiety about using computers is a significant problem with CAA	SD	D	N	A	SA
n) Creating good CAA questions requires considerable time and effort	SD	D	N	A	SA
o) Academic staff benefit from writing CAA questions (e.g. enhances question design skills)	SD	D	N	A	SA
p) An important advantage of formative CAA is to allow students to work at their own pace	SD	D	N	A	SA
q) CAA significantly aids learning and revision	SD	D	N	A	SA
r) CAA offers a wide range of question types	SD	D	N	A	SA
s) CAA can only be used to test some disciplines	SD	D	N	A	SA
t) CAA offers the potential to test a broad range of subject knowledge	SD	D	N	A	SA
u) CAA provides tutors with more feedback than paper-based methods	SD	D	N	A	SA
v) CAA improves the speed of feedback of results and comments to students	SD	D	N	A	SA
w) Objective tests result in inflated scores due to guessing	SD	D	N	A	SA
x) Objective testing is a good way to assess material in first and second year U/G modules	SD	D	N	A	SA
y) Objective testing is a good way to assess material in final year U/G or P/Grad. modules	SD	D	N	A	SA
z) It is desirable to incorporate multimedia applications (e.g. video clips) into assessment	SD	D	N	A	SA

This is the end of the 2004 national CAA Survey. We would like to thank you for giving your time and effort. As a small token of our appreciation we will enter your email address into a draw for two £25 Gift Vouchers. It is intended that the draw will take place at ALT-C in September 2004.

The results of this survey will be published in a variety of formats, including electronic publications available from the TOIA website.

Would you like to receive further information about results of the survey, other project activities and CAA developments?

Yes No

If there are any aspects of CAA you are particularly interested in, please list them below

Can we contact you by telephone or email to follow up on your responses?

Yes No

If so, please indicate suitable days and times to call

Telephone number

This is the end of the survey. Please press "Finish" below to submit your answers.