14. Questions? This is a NSW E-learning Innovations project output, developed by National Training and Business IT Training Specialists (BITTS) P/L, with seed funding from the Framework.
Hinweis der Redaktion
Thank you for attending this discussion on an POV Technology and it’s use in remote assessment …
Our project is focusing on the use of POV for assessment in the context of workplace trainees. We set our focus originally on the TLI industry, but have expanded this to other training packages as well. One reason for this is that our outcomes will not be specific to any particular training package but will be applicable to all training packages and it happened that we had some great quality trainees in other packages that were keen to participate in the trials, so who were we to hold them back? We often use observation as an assessment tool for trainees, but there are a few issues that we commonly come across: It is not always easy to work out mutually agreeable times to come in for demonstration of work tasks as some tasks are not performed every single day or at specific times, and so working out when for example will an IT trainee be installing an operating system or installing a hard drive is not always that simple. We can’t have them tell a client that this job that would normally be done within the same working day will take longer as we are waiting for an assessor to come and observe! The second side to this is that a single observation is not as good as multiple observations – if you can see your trainee do the same or a similar task repeatedly it is so easy to confirm competence. Take a unit out of TLI such as Receive Goods – parts of the elements here cover dealing with problems in the receival. We can’t guarantee that when we turn up to observe a receival there will be a problem to deal with. Now, if the trainee has a pair of glasses with them for a week, imagine how many different receivals and types of issues they can record. The underpinning skills and knowledge are also easier to observe if the trainee can collect video of numerous receivals and solving any issues as you will see them communicate with a wider range of people and cope with a range of situations. Finally travel can be a costly component for workplace trainee assessments in a regional area. We only have so much funding to cover the number of visits we do, so it is not practical to be travelling out to workplaces often to observe only 1 task at a time. The ability to use the glasses to collect evidence during many routine workplace tasks suddenly makes a lot of sense!
A significant part of the beginning of the project was spent in preparing for the workplace trials. We identified 4 main types of documentation that was going to be needed: Digital media release forms ensure that the workplace and all staff are fully aware that digital imagery is going to be captured and used both for assessment and for the purposes of the trial, and that permission has been obtained for this. It is important to remember that the trainee using the glasses is actually the least captured digitally of everyone, as they have the glasses recording what they see rather than recording themselves except for establishing their identity at the beginning of each video. Trial observation assessment tools had to be created and passed through the RTO compliance department. This is particularly important as the tools must map directly to the elements in the unit being assessed. We have developed a range of trial assessment tools for TLI, IT, Business and TAA units of competency. Feedback forms for assessors to complete every time they conduct an assessment via POV video is essential to track successes but even more for tracking issues that arise with this type of remote assessment. This feedback is critical to using a reflective cycle approach to the entire trial so that we can address issues immediately and adjust the trials as needed through this feedback. In this feedback we have been looking mostly at the quality of the evidence the video is capturing and what kind of things affect this quality. Feedback from the trainees for every trial assessment they complete is essential to track successes again, but more importantly to ensure that we identify typical issues that trainees may face using this technology for assessment. Even though the technology is very simple to operate on the one hand, on the other hand there are many small things that can affect the quality of the video that is captured that could render the assessment invalid for being able to capture the evidence an assessor needs
The first critical requirement for success with remote assessment is that your assessors understand the technology and can use it themselves. We have found that an assessor that cannot operate the glasses well themselves is unlikely to have their trainee ready to use the technology effectively on their own! We provided our assessors with a short task to perform using the glasses themselves – making a cup of tea, coffee or similar beverage. The approach we took was to have them perform a very simple task they are highly familiar with, but a task that requires focusing on small objects clearly. They are required to narrate what they are doing whilst completing the task. The entire task takes around 1 minute to complete (you already have the kettle boiled before beginning), so is very short, which is also important if they need to repeat it several times over to get it correct (which is most likely going to be the case). The reason it is likely they will need several attempts to get this right is that the one major issue we have discovered associated with using the glasses from a technical aspect is that the camera sits slightly above the eyes. This really causes no issue at all if you are wearing the glasses and watching something happening a couple of metres away, but takes some getting used to if you are performing a task with your own hands and attempting to focus the camera on your hands. You find that you need to tilt your head slightly down and therefore be looking slightly up with your eyes to capture what you are doing correctly. The simple task of making a cup of tea allows the assessor to get a true ‘feel’ for how to use the glasses correctly, and they are then able to explain that over to their trainee. The trainees then likewise perform this same task in order to become familiar with the glasses on a simple routine task before they attempt to perform a real assessment. The trainee needs to view each assessment before they attempt another assessment task so that they are confident they have captured their first assessment correctly with the critical aspects in focus. If not, they are able to delete that video and record again the next time they perform that task in the workplace. Another great feature of having the glasses is that the trainee may feel that they captured 75% of the task well with the glasses, but know that they did not tilt their head well in some parts or missed narrating important points in others. They may still elect to keep and submit that video, along with further videos of the same type of task knowing that the assessor will see and hear all they need across a range of submitted videos, even if each single video is not perfect. Next it is critical that the trainees understand the process they will undertake and the requirements of the assessment. Now this is a normal part of any assessment and yet I would say that many trainees rely on us to fully understand this and guide them along the way. If a trainee is to conduct these POV assessments alone they need to understand their assessment well to ensure they collect enough evidence – we have found that we encourage the trainees to narrate as much as possible when performing tasks that don’t include communication – eg when installing an operating system, changing a hard drive, performing manual handling operations etc. Finally the first 3 points allow us to get the trainee to a point where they are hopefully therefore capable of conducting the assessment on their own. I know that when I do assessment by observation in the workplace I am always talking to the trainee, prompting that narration and explanation of the task. The ability to do this can be quite a separate ability to their competence in the task as per the elements in the unit.
One of the biggest issues that was raised when POV began to show potential for assessment in the last couple of years has been around the age old question of – but how do I know that my trainee really did the video? This is not a new question for us as assessors – if I am handed a typed essay or project, how do I know my trainee actually did this work? I think this is a good time for you to have a small break from my voice. I’m going to switch to the whiteboard and would ask that you all throw some input in here – it doesn’t just apply to trainees, I could be teaching in a classroom and ask my students to all submit a homework assignment to me the following week. But how do I know they submit their own work? Please raise your hand to offer an answer, and as you put your point forward I will write them on the board here. In our project we have identified 3 methods of authentication and believe that all 3 are needed for each assessment collected. The trainee turns the POV on with the glasses facing them so that their face is captured. They identify themselves by stating their name, the unit they are undertaking and the date. Keeping the glasses running they then turn them around slowly and put them on their face. Narration is a large key – if they are narrating whilst recording the task you are able to asses underpinning knowledge and skills as well as identify your trainee throughout the recording. This narration should include WHY they are doing something, any particular organisational rules involved, any legislation involved or other supporting narration. Finally, after you have watched the assessment videos, you should hold a short discussion with your trainee about their tasks. Whilst watching the video, make note of anything that could become a good discussion topic ready for your next visit or a phone call. For example as they fill out a workplace form you could ask them about some details on the form, or ask them some finer points about using a piece of equipment that was in the video. Would you all agree that you use this type of method with students already, and often without consciously thinking about it very much? How many of you find you often just ask little clarifying questions of your students or trainees that allow you to mentally note things about their underpinning skills and knowledge?
Many skills need demonstration to learn them effectively. Many people simply prefer to learn by demonstration due to their learning style. Whilst some tasks can be video taped to create learning objects, the video camera in someone else’s hands may not always capture the demonstration clearly. Attempting to video a close up of a RAM chip being inserted is going to be difficult for a third party camera operator to capture without pretty much getting in the way of the demonstrator or the demonstrators hands will be in the way. Wearing the glasses means that if you as the demonstrator can clearly see the process you are demonstrating, then the camera can too! We have also found that people don’t always perform well with a camera pointed at them. We have found people quickly forget the glasses are a camera and they stay far more relaxed and natural. Of course the advantage over doing the demonstration in real time for the trainee or in a classroom is that the video can be watched as many times as needed to assist the learning. It is simple to make lots of short, sharp and to the point video demonstrations, each of which may be short and only cover a single element or sub element in a unit. One unit may comprise of a whole suite of video demonstrations, each showing a single task or process allowing the student to grab just the point of learning they need at the time they need it. The video can be published to the web – your own moodle, blog, web site or even you tube if you wish. The video can be placed on CD and given to students or trainees, or put on an internal intranet. You can convert to formats that can then be viewed on mobile phones if desired. The cost of producing video is minimal – in fact you can create the video the next time you are doing a workplace or classroom demonstration!
This first example video was recorded very early in the project. We can’t see what the student is actually doing! We can see the chopping board, and the occasional flash of a knife, but this video would not allow a remote assessor to see what they need to see. It is a great example video and we learned a lot from this early demonstration – what we learned was that we needed to develop a short, simple test task for trainees to perform until they felt they were now well able to use the technology for their assessment task. The narration provided however was excellent, explaining not only what she was doing, but WHY she was doing it – chopping each vegetable into a size that will fit on a spoon.
This video is distributed to all trainees to watch before they use the technology. Along with the video is a hand out sheet that provides the instructions on how to create a similar piece of footage. Trainees are asked to repeat this task until they have produced a video that clearly focuses on what they are doing with their hands. This has proved to be a critical step in the process and has allowed our trainees to become very comfortable using the technology before attempting an assessment task. Something that has shown to be of particular importance in this is that students are nervous when demonstrating an assessment task, so if they are not comfortable with the technology they will produce poor quality video, and will be disheartened if they have to repeat the task due to this rather than due to being not yet competent in the actual task.
This test video was conducted on a motorbike to test how the camera performs in full sunlight, with movement and background noise. We were very pleased with how the camera coped with these conditions, and were testing actions such as showing the speed of the movement by looking down at the instruments.
Flouro lighting is an issue with any video camera and has to be taken into account when trainees are conducting assessments in an environment like this. This example video was taken at a conference, and unfortunately the camera was pointing slightly upwards which has exacerbated the flouro issue.
This is an example section of an assessment conducted in a warehouse environment. We were particularly happy with this capture as the trainee narrates well, and points to documents and computer screens often with his narration. This full assessment took nearly 20 minutes to complete. A copy of the documents used during this task is submitted with the video allowing the assessor to follow the progress of the demonstration task and see clearly where the trainee was marking items off the list and making amendments.
This is an example of a learning object created from video by a recently completed IT trainee. These learning objects could be created either by trainees doing successful assessments, or by trainers demonstrating competencies. We developed 6 example learning object videos for use with IT trainees. The concept we have aimed for is one of having many very short movie demonstrations available – short sharp learning objects. The first one we aimed to develop was to teach how to change RAM sticks on a MacBook Pro. What I loved was that as we went through this process we decided to break it down into 4 separate videos. Battery Removal Open RAM Access door Remove RAM Install RAM We had 2 main reasons for choosing this path – firstly the shorter time length the smaller file size and therefore the faster download time. Secondly if a learner just wants to see how to put a RAM stick in, they don’t need to watch the first 3 steps, so they can choose to watch only the part of the learning they need at the time. In order to create the learning object I did edit the video using iMovie These files will be available for you to watch again from my mobile me site which will be given to you after this presentation.
The conclusions we have come to from the project so far revolve around how essential it is for training in using the technology by the assessors. We were surprised at the level of training needed, maybe because we are tech heads ourselves, but not all assessors who are experts in their fields are terribly comfortable with technology and need to be well supported in becoming comfortable with the technology. We have also concluded that authentication is not a technology issue, but is a process of getting to know your trainees well enough to use this technology as just one tool in a suite of assessment tools.
![Background ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)