Evaluation Plan Draft

My project consists of a Needs Analysis investigation asking the question:How can an existing predominately F2F Written Communication class use electronic resources to engage and encourage learners.

Presently, I teach a range of F2F business classes. One of the subjects I struggle to make exciting and enthralling for my students is Written Communication, Level 3. It is a core course that all students need to complete to obtain their full qualification. The classes are scheduled as a two hour practical class and a two hour tutorial class running over six weeks. In the 2 hour practical, three areas are covered: CV and Letter Writing, Technical Writing and Report Writing, so any eLearning components I introduce will be given as additional resources rather than classwork resources. I am hoping by giving the students interactive resources I will notice a marked improvement in the quality of the assessment submitted and an enthusiastic (well, more enthusiastic) approach to the subject by the students.

For students with a solid grasp of English, the coursework is a ‘cinch’, but for others the coursework is difficult. Many students find the lessons long and tedious and I often notice a drop off in attendance after week one or two which in turn leads to a frantic panic by the students towards the end of the course when they realise they need to complete Written Com to finish their certificate. Added to this, many of the students taking this level 3 paper, don’t have a thorough knowledge of business writing to begin with so their ‘problems’ are compounded as the course nears the end.

Purpose

The general purpose of this evaluation is to evaluate whether eLearning can easily and effectively be included into this particular unit and whether it will be worthwhile for the students. AudienceThe Needs Analysis assessment will be carried out with the help of current students on the course. There are 9 regular attendees who will be asked to contribute their thoughts and feelings on using supplementary eLearning resources for the course.

Decisions

The findings from this Needs Analysis evaluation will initially assist me in designing and obtaining eLearning resources that are useful for my students. It may provide advice to other lecturers and designers of foundation literacy courses of where to acquire ready made resources or provide new avenues of learning not previously explored.

Questions

I have chosen the following two elearning guidelines to base my Needs Analysis on. The additional comments are my thoughts (but not necessarily the entire sub-questions) on what I want to achieve from each guideline. :

TD1: How can eLearning be used to support students in meeting the intended learning outcomes?
~ How do students feel about using webbased technology? I need to establish whether the students feel comfortable using possibly alternative technology so early in their coursework. While I can assume they will be ‘all for it’, perhaps the pressure of having to potentially learn something new may hinder their positive experience~ How is eLearning currently being used in tertiary education for this subject?
Finding out what other institutes are doing/using to teach this subject will be useful as comparison data. ~ What sort of approaches and resources might be used to motivate learning?

TD7: Can Students practice using the assessment tools used in the summative assessment?
~ Do the templates provided assist the student?At present one template for the written report is provided so I’d like to know if this was helpful for the student, why or why not it was helpful. This may help me to determine whether I need to create more directed resources for the students.
~ Does the ‘potential resource’ provided complement the classroom environment?The potential resource I have in mind is a webbased quiz type application that is almost game like. It takes the student through commonly confused English words i.e. Their, there and they’re. For the student to progress they must complete the previous level. I would like to know whether this type of interactive resource is useful for the student and whether it complements the work we do in class.

Methods

The methods I am using are an informal focus group consisting of some of my current students. The opinions of the focus group will be collated and used in my investigation of how other institutes deliver foundation programmes (namely Written Comm) online or via other eLearning methods and whether this is viable in my programme.
I will conduct email interviews with expert peers (at least 2) at other institutes around the country to gain their knowledge on what has worked for them in the same scenario.
I may have time to visit at least one TAFE in Queensland, Australia on my next visit (July) and speak with lecturers/ course designers to ascertain how Australian Institutes deal with eLearning in foundation courses.

The methodology I am using is a mixed-methods evaluation design which fits under an eclectic-mixed methods-pragmatic paradigm (cited Reeves,T. 1997).
The reason this particular paradigm was chosen is that it provides diversity and incorporates different approaches of a practical nature that is suited to my personality and style of work.

Evaluation in an educational setting is the process whereby we seek evidence that the learning experiences we have designed for students are effective. We evaluate educational activities for two overlapping reasons:
1. to obtain information that can inform the ongoing design and development process (often referred to as formative evaluation);
2. to decide whether an innovation is worth retaining (often referred to as summative evaluation). These forms of evaluation often meld together, and each can be difficult to undertake properly. (Philips, Bain, McNaught, Rice, Tripp, 2000)

Objectives

To complete the evaluation an evaluation plan (this document) will be produced and submitted for feedback. An investigation of resources and methods detailed in this plan will then be undertaken. Following the investigation, an analysis of data and materials will proceed which will conclude in a formal written report for the perusal of my fellow course participants.

Timeline (Adjusted 30th June 2009)

30 June – 3 July. Finish writing evaluation plan and submit
4- 19th July Undertake investigation and gather data.
20-27 July Analyse data, write report and submit by 31st July.

Budget

Evaluator charges at $70 per hour * an estimated time spent on project of 12-15 hours) Total $960 - $1200

References

Reeves, T. C. (1997). Established and emerging evaluation paradigms for instructional design. In C. R. Dills & A. J. Romiszowski (Eds.), Instructional Development Paradigms (pp. 163-178). Englewood Cliffs, New Jersey: Educational Technology Publications.

Phillips, R., Bain, J., McNaught, C., Rice, M. and Tripp, D. (2000). Handbook for Learning-centred Evaluation of Computer-facilitated Learning Projects in Higher Education, Murdoch University and the Australian Society for Computers in Learning in Tertiary Education (ASCILITE). Available http://www.tlc.murdoch.edu.au/archive/cutsd99/handbook/handbook.htmll

Week Two thoughts

Why is evaluation important to you and how do you define it?

I view evaluation as a platform or tool for design, for development or for improvement. Evaluation is a way of determining what is needed, how it will fit with other developments and if something is really needed or just wanted. . Evaluation can easily be incorporated into the design stage of a project via SWOT analysis where the strengths, weaknesses, opportunities and threats are defined. Evaluation throughout development is vital to ensure that a reliable, realistic product is created.

Evaluation comes in many forms and informal talks with peers, stakeholders and industry advisors at conferences etc are just another example of ways in which we can self assess and expand our horizons.

2. What sort of evaluations mentioned on the presentation are familiar to you already and why?

Observation

These are used in the courses I teach to ensure students can actually do the physical tasks asked of them in the unit plan. Additionally, we as lecturers are observed by our peers in the classroom and given feedback on our practices.

Questionnaires

Students are asked to fill in lecturer and programme evaluation forms each semester on each course. This enables the lecturer to take a look at themselves through others eyes.

Discussion forums

In one of my courses I use the discussion board in Blackboard to encourage collaboration amongst the students. This exercise also provides a way for me to be able to assess how the students are coping with the sometimes difficult material. I can then decide whether parts of the subject needs to be taught a different way or new resources created.

Expert Reviews

An example of expert reviews that I can think of is the external moderation of course design and material that happens on all programmes in the school I work in. Stakeholder meetings are also one way that our institute evaluates what is going on in industry and therefore which direction to take our courses.

3. Why is quality important in eLearning?

Quality in elearning is important because the product or resource has to replace (albeit
temporarily sometimes) a face to face lesson. Students will rely on that resource to provide the knowledge and answers they would generally ask of the tutor.
Providing a poor quality elearning product or resource with little or no support (teacher contact, help resources or f2f facilitation) will inevitably result in poor success rates and lower overall course participation numbers perhaps.

Hello

Hi all

I'm Heather Moran and I work at UCOL Palmerston North in the same office as Debra M, although not on the same programme. I've been working at UCOL now for ten years with the past five of them working as a Lecturer in Information Systems.

I teach face to face classes but mark assignments for our blended delivery computing programme and online business programmes on occasion.

What I know of, and about, evaluation is all down to my past experiences as a student and as a tutor. Both of these situations are very obviously different and give very differing views on the relevance of evaluation.

As a student sometimes it feels as if we are just filling in forms or posting opinions and/or ideas to fulfill some assessment criteria within the programme, rather than it being of any relevant use to us. We're given programme evaluation forms to fill in, lecturer evaluation forms to complete, unit or paper evaluations and sometimes units come with assessment criteria that requires us to leave feedback on a discussion board or some other form of chatting facility. Some of the time we are left wondering if anything said in the evaluation is even taken notice of.

As a tutor we see evaluation as a way to gauge students' interest in our papers/units, resources, and programmes. While data gathered can be useful, it is what is done with that information that is important. If we find that we are asking students and co-workers to evaluate situations/resources and then not following up with the data gathered, is it a wasted exercise? And how often are we performing these “wasted exercises?”.

So in reality (my opinion anyway) it's no wonder that some students give responses that are less than honest and authentic. Some people are not naturally outspoken or adept at giving "feedback" so evaluation to them can be a daunting task. Some people love it and relish the opportunity to leave feedback. So what is the balance? Is there a balance between giving evaluation tasks and receiving useful authentic feedback and giving evaluation tasks that result in information that is less useful? Of course there is, but how and when it is done is the key.

What I hope to get from this paper are ideas and/or tools for making evaluation authentic, realistic and relevant. I hope to share my ideas and learn with my fellow course participants the "best practice" for evaluation.