Sunday, May 3, 2009

Evaluation Plan Draft

My project consists of a Needs Analysis investigation asking the question:How can an existing predominately F2F Written Communication class use electronic resources to engage and encourage learners.

Presently, I teach a range of F2F business classes. One of the subjects I struggle to make exciting and enthralling for my students is Written Communication, Level 3. It is a core course that all students need to complete to obtain their full qualification. The classes are scheduled as a two hour practical class and a two hour tutorial class running over six weeks. In the 2 hour practical, three areas are covered: CV and Letter Writing, Technical Writing and Report Writing, so any eLearning components I introduce will be given as additional resources rather than classwork resources. I am hoping by giving the students interactive resources I will notice a marked improvement in the quality of the assessment submitted and an enthusiastic (well, more enthusiastic) approach to the subject by the students.

For students with a solid grasp of English, the coursework is a ‘cinch’, but for others the coursework is difficult. Many students find the lessons long and tedious and I often notice a drop off in attendance after week one or two which in turn leads to a frantic panic by the students towards the end of the course when they realise they need to complete Written Com to finish their certificate. Added to this, many of the students taking this level 3 paper, don’t have a thorough knowledge of business writing to begin with so their ‘problems’ are compounded as the course nears the end.


The general purpose of this evaluation is to evaluate whether eLearning can easily and effectively be included into this particular unit and whether it will be worthwhile for the students. AudienceThe Needs Analysis assessment will be carried out with the help of current students on the course. There are 9 regular attendees who will be asked to contribute their thoughts and feelings on using supplementary eLearning resources for the course.


The findings from this Needs Analysis evaluation will initially assist me in designing and obtaining eLearning resources that are useful for my students. It may provide advice to other lecturers and designers of foundation literacy courses of where to acquire ready made resources or provide new avenues of learning not previously explored.


I have chosen the following two elearning guidelines to base my Needs Analysis on. The additional comments are my thoughts (but not necessarily the entire sub-questions) on what I want to achieve from each guideline. :

TD1: How can eLearning be used to support students in meeting the intended learning outcomes?
~ How do students feel about using webbased technology? I need to establish whether the students feel comfortable using possibly alternative technology so early in their coursework. While I can assume they will be ‘all for it’, perhaps the pressure of having to potentially learn something new may hinder their positive experience~ How is eLearning currently being used in tertiary education for this subject?
Finding out what other institutes are doing/using to teach this subject will be useful as comparison data. ~ What sort of approaches and resources might be used to motivate learning?

TD7: Can Students practice using the assessment tools used in the summative assessment?
~ Do the templates provided assist the student?At present one template for the written report is provided so I’d like to know if this was helpful for the student, why or why not it was helpful. This may help me to determine whether I need to create more directed resources for the students.
~ Does the ‘potential resource’ provided complement the classroom environment?The potential resource I have in mind is a webbased quiz type application that is almost game like. It takes the student through commonly confused English words i.e. Their, there and they’re. For the student to progress they must complete the previous level. I would like to know whether this type of interactive resource is useful for the student and whether it complements the work we do in class.


The methods I am using are an informal focus group consisting of some of my current students. The opinions of the focus group will be collated and used in my investigation of how other institutes deliver foundation programmes (namely Written Comm) online or via other eLearning methods and whether this is viable in my programme.
I will conduct email interviews with expert peers (at least 2) at other institutes around the country to gain their knowledge on what has worked for them in the same scenario.
I may have time to visit at least one TAFE in Queensland, Australia on my next visit (July) and speak with lecturers/ course designers to ascertain how Australian Institutes deal with eLearning in foundation courses.

The methodology I am using is a mixed-methods evaluation design which fits under an eclectic-mixed methods-pragmatic paradigm (cited Reeves,T. 1997).
The reason this particular paradigm was chosen is that it provides diversity and incorporates different approaches of a practical nature that is suited to my personality and style of work.

Evaluation in an educational setting is the process whereby we seek evidence that the learning experiences we have designed for students are effective. We evaluate educational activities for two overlapping reasons:
1. to obtain information that can inform the ongoing design and development process (often referred to as formative evaluation);
2. to decide whether an innovation is worth retaining (often referred to as summative evaluation). These forms of evaluation often meld together, and each can be difficult to undertake properly. (Philips, Bain, McNaught, Rice, Tripp, 2000)


To complete the evaluation an evaluation plan (this document) will be produced and submitted for feedback. An investigation of resources and methods detailed in this plan will then be undertaken. Following the investigation, an analysis of data and materials will proceed which will conclude in a formal written report for the perusal of my fellow course participants.

Timeline (Adjusted 30th June 2009)

30 June – 3 July. Finish writing evaluation plan and submit
4- 19th July Undertake investigation and gather data.
20-27 July Analyse data, write report and submit by 31st July.


Evaluator charges at $70 per hour * an estimated time spent on project of 12-15 hours) Total $960 - $1200


Reeves, T. C. (1997). Established and emerging evaluation paradigms for instructional design. In C. R. Dills & A. J. Romiszowski (Eds.), Instructional Development Paradigms (pp. 163-178). Englewood Cliffs, New Jersey: Educational Technology Publications.

Phillips, R., Bain, J., McNaught, C., Rice, M. and Tripp, D. (2000). Handbook for Learning-centred Evaluation of Computer-facilitated Learning Projects in Higher Education, Murdoch University and the Australian Society for Computers in Learning in Tertiary Education (ASCILITE). Available


  1. This is looking great and I really like your sub-questions and descriptions about how you are going to collect data. As I promised, I will send some questions through to help with a short survey to find out whether students will be skilled enough to use the online resources. For example, how long have they been using computers, what sort of tools are they already using, how confident are they etc.

    You have a good mix of approaches to make this a mixed-methods evaluation design - my thoughts are that it comes under an eclectic-mixed methods-pragmatic paradigm. Meaning diversity and several different approaches of a practical nature - sounds like just your thing. :)

    It is a good size having nine students whom you teach, and having the bond already which will enable you to interact freely with them and organise your evaluation to suit you and the students. One thing you may wish to consider, which an anonymous survey will help alleviate, is their reluctance perhaps to give you honest feedback. This is an ethical dilemma you can discuss in the report - were they honest or just telling you what you wish to hear?

    Here is a resource you might also like to show your students; it is on WikiEducator: and might give them an idea of how online resources could look. It is pretty text-based so it will be good to find out their reactions.

    I am looking forward to reading the next part of your plan and giving feedback. You are moving ahead well.

  2. Hi Heather

    I have read with interest your needs assessment as this is similar to my project, however, at this stage (due to time limitations) I am not going to survey students, however, I will be interested to come back and read some of your comments from students. You raise an interesting point with the technological dilemma whether students will buy into using 'new' technology or whether they will be concerned of the effects it may have on their overall learning experience. You have got me thinking - thanks!

  3. Hi Heather, I like the sound of the game for they, they're etc. I think you are very wise to consider the student's ability to use the online resources. It is a good idea to survey them as Bronwyn suggests.

    Also, find out about the technical support that is available to them through the institution ie computer classes, Blackboard helpdesk, and provide information about how to access those services.

    Cheers Adrienne

  4. Hi Heather since our talk today, I think it will be advisable to do the following:
    As your organisation is going to Moodle in 2010, it would be good to estimate what proportion of the course should be online to suit the student group. As you said some of the students need a lot of support, almost coaching, and were not particularly keen on using online resources for the course though they like using the Internet for their own interests. You can use the information from the discussion you had with students about online resources as part of the data for the evaluation.

    I am glad you are interested in talking to my colleague re the language learning and his class blog he uses with language students. William has two blogs a class blog and a professional blog. The latest post on there lists some of the benefits in using a blog for teaching and is worth a look. (

    As we discussed you will start doing some research about what is going on elsewhere in the same field - online resources and course designs. You will also spend some time talking to colleagues who are teaching similar types of courses using online strategies might be helpful as well with regard to transferring the course to Moodle.

    I think you can still use the first evaluation question and I have modified TD1 to accommodate what we spoke about today, and I have added another couple of sub-questions. Have a think about what you want to find out. One guideline and three sub-questions will be sufficient.

    TD1: How can elearning be used to support students in meeting the intended learning outcomes?
    ~ How do students feel about using web-based technology?
    - How is eLearning currently being used in tertiary education for this subject?
    - What sort of approaches and resources might be used to motivate learning?

  5. Heather
    Hi Heather, this is late in coming, but hopefully you are continuing on with your plan and will find it useful.

    You are conducting a Needs Analysis to investigate how electronic resources could be used to engage and encourage learners in a Written Communication class.

    You have identified some problems with the course,
    -Long, tedious lessons
    -Difficulties for students with English as a second language
    -Difficulties for students with no previous experience of business writing
    -Desire to increase student motivation and engagement
    -Desire to increase quality of student work
    and are wondering if elearning will assist with student motivation and the quality of their work.

    I think your idea of a quiz or game is excellent and it can be a lot of fun for students, however, I don’t believe that a quiz will solve all of your problems. I would also look at:
    -Timetabling – can the course be taught over 12 weeks with a one hour practical and a one hour tutorial?
    -Support for ESL students – peer tutors, international department?
    -Foundation concepts – an easy to understand, self-paced online resource might be useful for this
    -Quality of work – make sure that you only provide qood quality examples from the internet – critique examples that are not up to scratch in class.
    -Motivation - devise interesting and practical exercises during class time – use group work, real-life situations, case studies and the internet. Students can set up their own companies and correspond with other student companies. Make it interesting and relevant for them. Get them using modern business communication technology – email and websites all need impeccably written letters and reports. It can be fun for students to see their work – once it is of good quality - up on a student website. (Yes you will need to do further learning to achieve this, but maybe it would be motivating for those who are finding the course easy).

    A quick Google search will come up with lots of ready-made websites and resources. You will have to assess the quality and relevance of them and adapt them for your own use. Here are some examples:

    (Sorry about these links, I couldn't get them to work for some reason.)

    There are screeds of electronic resources out there, but you also need to look at the whole course and how it is taught. Electronic resources can be a part of this, and they work best when integrated into a course which is well designed overall.

    It is a great idea to do a focus group with students and research how others are teaching this course. I am sure you will get great ideas from your students and experts.

    I hope this is useful feedback for you, let me know.