CHAPTER 8

Multimedia Beyond Text

Images have played a key role in scientific textbooks for centuries. Galileo used illustrations in his manuscripts during the mid-1530s. Century-old science textbooks include illustrations that are black-line drawings. Some of these are marvelous in the way they illustrate complex concepts. Thirty years ago color in texts was often restricted to one signature, that is, a group of pages that were manufactured together during printing process. Modern textbooks, especially those aimed at large audiences, are usually multicolored affairs. Even the printed text is emphasized using color to code words.

Visual images aid in learning. The capacity for pictorial memory exceeds that for verbal memory [Shepard, 1977; Standing, 1973]. The neurological mechanisms underpinning human storage of memories of pictures do not include either models in which individual images are stored as if in a museum gallery, or strings of 0s and 1s. Biederman [1987] has proposed a recognition-by-components model. Using images to teach is nothing new [Willows & Houghton, 1987]. There are some situations, i.e. with beginning readers, where pictures not only slow reading, but pictures unrelated to the text produce interference [Willows, 1978]. However, the conclusion of considerable research on the use of charts, graphs, and diagrams indicates that they can convey information uniquely in ways not possible for text [Winn, 1987]. Mayer has pursued this area and developed some explicit guidelines regarding the nature of illustrations successful in enhancing learning [Mayer & Gallini, 1990].

Modern Web browsers readily accommodate images and other forms of media. Web-delivered media can be so complex that one needs to be concerned about cognitive load – and to be sure that a learner's cognitive resources are working synergistically rather than antagonistically with the learning materials provided. Sweller and colleagues [Sweller & Chandler, 1994; Mousavi et al., 1995; Cerpa et al., 1996; Sweller, et al., 1998] address the design issue of matching cognitive load to cognitive resources.

Compression

Because computer files of pictures, movies, and sounds are very large compared to text files, developers have designed ways to make the files smaller. Image files are saved by taking the information in the image and compressing it. An image that is 8 pixels by 8 pixels contains 64 total pixels, every pixel has multiple characteristics - color, hue, intensity to name just a few. If every characteristic of every pixel were recorded individually, even an 8 pixel by 8 pixel image would be a large file. The answer is to compress the file. A grossly simplified example would be to record that the first through sixth pixel in the second through fourth rows of pixels are all a particular color, hue, and intensity. Compression procedures can be lossless or lossy.

Lossless compression occurs when every pixel is recorded and the image can be precisely reconstructed from the compressed data. Lossless compression requires the files be sufficiently large to include all the possible data. A compressed file of 64 pixels with all pixels different would be the same size as the non-compressed file. But, since most images actually have large areas that are identical, even lossless compression usually represents a major saving in file size.

Lossy compression goes a step further and saves the image without saving the data for every single pixel. It samples the pixels, and saves a reasonably representative set of pixels. Lossy compression can trade quality of picture for size of file. As the quality of the saved image improves, the size of the file increases. Since lossy compression decreases the total amount of information, files that are compressed and saved have less data than the original. Repeated compression will degrade the image. Nearly all video compression is lossy. Some still image formats are lossy, while others are lossless. Maximizing compression minimizes download time, but maximum compression leads to loss of data.

Thumbnails

When it is necessary to have a large number of images available, one way to minimize the time for download is to create thumbnail versions of the images (see Figure 8.02). The thumbnail images can be created with little concern for loss of data, the user clicks on a hypertext title or even on the thumbnail itself to download the full version.

The GIF format (Graphics Interchange Format, pronounced like the peanut butter), developed by CompuServe, was the original format used for still images on the Web. GIF uses a lossless form of compression; images may be saved, restored, and saved again with no degradation. GIF is excellent for line art (logos and basic illustrations), icons, graphs, and lettering clearly. The GIF format permits up to 256 colors. When interlaced, downloading GIF images and embedded lettering appear fuzzy until nearly finished loading.

The JPEG (Joint Photographic Experts Group, pronounced JAY-peg) format provides the ability to display millions of colors, and to compress the file for speedier transmission across the Web. JPEG was designed mainly for use with photographic and lifelike images. JPEG is a lossy form of compression; the degree of loss can be varied when saving in JPEG format. Quality can be traded for speed, or vice versa. Creation of thumbnail versions of images for indexing on a Web page can allow use of multiple JPEG images without sacrificing download time (see Figure 8.02). These thumbnails can be clickable to access the full size pictures. Since the image compression produced by JPEG is lossy, any editing of the image should either be accomplished before conversion to JPEG, or the file should be converted to PNG and edited within that format before being returned to JPEG. Repeated compression and decompression within JPEG degrades the image rapidly. Software using the PNG (Portable Network Graphics, pronounced "ping") format can be used for editing of images prior to conversion to JPEG or between a decompression and compression.

JPEG uses a two-dimensional interlacing, so print embedded in images becomes clear sooner during the download process. GIF images should not be converted directly to JPEG, too much detail will be lost.

The PNG format uses lossless compression. It can be saved, viewed, and saved again without degradation. It was developed in response to copyright issues concerning GIF, and is widely regarded as an improvement on GIF. PNG has a better maintenance of color between and within platforms. It includes the ability to have variable transparency on the pixel level (254 levels of partial transparency), slightly better compression than GIF (5% to 25%), and two dimensional interlacing so text embedded in a downloading image is readable about twice as fast as in a GIF image. Like JPEG, PNG supports true color. PNG serves as the native format for many of the best new graphics programs (e.g., Fireworks and PhotoShop use PNG, and then allow for export of the image to GIF or JPEG).

To insert an image into an HTML document, use the "image" tag. Within the tag, the source attribute (src) tells the browser where to find the image file. Other attributes, such as width and height in pixels, may also be included in the image tag.

<img src="Ch08-04.gif" width="150" height="150">

Editing is a simple matter of clicking on images and using the "draw tools" to manipulate them.

There are numerous ways to capture photographs for Web use. Photos can be scanned; film processors can provide a digital version of pictures at a minimal cost; frames can be captured from video. Digital photography, however, has revolutionized the use of photos on the Web. If the desired illustration can be photographed, the use of a good digital camera can make the process of acquiring the photos quick and relatively inexpensive. Although digital cameras can be more expensive than film cameras, the savings in time and cost of film and developing can quickly make up the difference. Software such as Adobe Photo Deluxe can make the manipulation of digital images quick and simple.

The digital camera market is changing fast, as most technology markets do. Any specific recommendation which could be made here would be out of date prior to publication. When shopping for a digital camera, several things need to be taken into consideration. The quality of the images is very dependent on the resolution, but if the camera is exclusively for use with a Web site, extremely high resolution cameras may be unnecessary. If images need to have very minute detail, higher resolution may be desired. Camera manufacturers use any of several storage strategies. Some cameras use memory cards; some cameras use floppy discs.

The Kodak Photo CD also offers a simple way to convert slides, negatives, and even flatwork (printed pictures) to digital form. Existing slides or film negatives can be sent to a photo lab for transfer to a Photo CD.

The standard Photo CD can hold about 100 images. It comes with a numbered thumbnail color image that permits quick review of the images to locate any particular desired image. Software on the Photo CD permits you to examine each photo. Five different image resolutions are available. The greatest resolution demands enormous amounts of memory, and provides what is tantamount to magnification of the image. Software designed for a variety of graphical image uses (e.g., Photo Deluxe, Photoshop, or even Fireworks) can be used to adjust and edit the image.

Using an appropriate board or other converting device, single frame video images (analog output) can be converted into digital still images. This technology used to be a mainstay, and may be the best way to extract still images from existing video resources. With digital video, frames are captured using the editing software.

An interesting, popular, and inexpensive solution for some situations is a small camera mounted atop the computer. These devices usually come with software that permits capturing a single frame. A computer mounted Webcam can provide intermittent pictures of something you want to keep an eye on, like this construction site. (Figure 8.06.)

Visuals developed for the Web can be information rich, esthetically rich, and interactive. Graphs, once created with great effort, are now handled easily through desktop computers. Morphs, multiple images representing the metamorphosis of one image into another, once were essentially impossible to create. Overlays and multiple exposures were possible, but stepwise transformations of one image into another required careful artwork and enormous effort. Morphs now are created easily on a desktop computer.

Spreadsheet programs are the computer tools most commonly used to construct graphs. Data which has been entered in the spreadsheet can be organized into a wide variety of graphs and charts with just a few clicks. Graphs may be captured from screens, edited in a program like Fireworks, and pasted into a Web page as an image.

For those who need more specialized graphing capabilities, high power graphing applications such as DeltaGraph, or Kaleidagraph are designed to construct multiple forms of complex graphs. Kaleidagraph, available for both Macintosh and Windows (files may be transferred cross-platform), provides a powerful, but relatively inexpensive 2-D graphing tool for the scientific community. It includes several built in macros to perform operations such as the generation of random numbers. It includes built-in fits for linear regression and smoothing, plus over 100 industry-specific formulas. DeltaGraph has similar capabilities to Kaleidagraph, but puts more emphasis on the presentation aspects of charts and graphs.

Several applications dedicated to specific subject areas include charting and graphing capabilities. These images can sometimes be added directly to a Web page, otherwise they may be captured as a screen shot, edited, and pasted in as an image.

Programs are available that allow students to download software to their graphing calculators (TI-Graphlink), either from other calculators or from computers.

Virtual reality can take the form of images stitched together to form a 3D view, or images forming movies, allowing the viewer to "move" through the scenes. Petrakis placed a digital still camera on a special tripod and captured images sequentially as the camera was rotated through 15-degree increments. These images were then stitched together to create a file that is interpreted by QuickTime. By dragging the cursor, a virtually continuous display is presented as if one were standing in the center of the laboratory and continuously turning around. The rate is determined by the extent of the mouse gesture. This is something you must see to believe. It can crawl, or it can spin at a dizzying rate.

There are several chemistry programs for creating images of molecular structures. These may be represented in any of several ways, such as with letters

(alphameric characters) representing atoms and lines representing the chemical bonds between atoms, or with three-dimensional spheres touching overlapping spheres representing three-dimensional molecular structures.

Creating images such as the one on the left of Figure 8.10, used to involve literally constructing physical models and photographing them. Today building such a representation using software takes an experienced user seconds. Chime allows the same information file used to create the images at the left to be manipulated using the Chime plug-in. So, the user has a great deal of control over the format and aspect for viewing the structure. To manipulate the image in

3-D; CAD

Computer-assisted design (CAD) programs can create very useful images for teaching, especially for technical disciplines. One can capture one or a few such images, and display them on the Web as GIF or JPG images. Modern CAD packages allow basic design in two dimensions with extensions into three dimensions, including rendering. Most modern architectural designs are rendered with CAD programs. Inexpensive CAD programs with limited features are available.

Some basic 3-D affects are now available in the drawing tools included in some wordprocessing applications. (Microsoft Word for Windows can take an oval and turn it into a tin can telephone! See Figure 3.01 in Chapter 3.) Companies marketing 3D and/or CAD programs are flooding the marketplace with products aimed at the Web. Some of these focus on the creation of images for use in Web documents.

The quality of the movies produced for the Web continues to grow as the result of improved hardware and software. However, video is still very expensive in terms of system resources.

In order to work on the Internet, video must be compressed. Uncompressed, a 3 minute movie would be approximately 4 gigabyte. MPEG compression can reduce the file size by up to 99% depending on the images and the degree of lossiness that is acceptable. Uncompressed video takes so much bandwidth that accessibility would need to be restricted. There are several video compression standards commonly in use. MPEG, the film industry standard for video compression, stitches together JPEG images, retaining all the individual frames. Cinepak is the compression format used for QuickTime files. MPEG

We use the term movie to describe moving images that are packaged into a file that will transfer as a package from the server to the client.

The impact that digital technologies have had on media is nothing short of remarkable. The surrealism that greets viewers, made possible by computers, is spectacular. Also spectacular has been the decrease in video production costs. Not only is desktop video an achievable reality, but portable hardware and software packages are more than up to the task of creating outstanding video. One person can do in minutes what used to take several persons many hours, or even days, to accomplish. With digital video, desktop movie making is rapidly coming into its own. Excellent digital video camcorders are now very affordable.

Desktop video is not something to enter into casually. There is more to making movies than simply running a camera. A sense of production quality appropriate to the task does not come packaged with the equipment. Regardless, many teachers may know their own subject, but creation of quality video requires a larger set of design skills. Many teachers are undertaking their own video production. What teachers do bring to the task is a sense of the troubles students have when they encounter material. This knack for identifying the stumbling blocks is key in many situations.

Video must capture what is needed. The key visual information must nearly fill the video screen, and be devoid of any distracting visual information.

Streaming video is a file sent from a server and captured in a buffer but not stored permanently by the client. As soon as enough information has been received, the video starts. Usually, starting is delayed until a combination of the amount in the buffer and the rate of data reception suggest that the video will be played to completion without interruption. Streaming video usually is played from a server. Sometimes, however, live video is compressed and delivered nearly immediately – just seconds elapse between the live act and the information stream sent to the Internet.

Some sort of software player that decompresses the data stream that is arriving and displays it as video is necessary. A player may be included in browser software, or can be downloaded from the Internet.

An animation is a sequence of graphic images stitched together to form a movie. Animations are probably used as often as movies in teaching. Animated graphs, for example, provide clear indications of phenomena.

One of the simplest forms of animation for the Web is the animated GIF, which consists of a series of GIF images stitched together using appropriate software. Figure 8.14 is an animated GIF taken from one of the author’s Web sites.

Once the files of digitized video are on your hard drive, you’re ready to begin the editing process. You will need to use some nonlinear videoediting software. Drive space and RAM can limit what you are able to do. For most operations, a one gigabyte drive and 48 megabytes of RAM specify a small system.

Many visual effects are available to go from one segment to another as you edit them together. For example, you can use a cross-fade – where the first image fades to black as the second image comes up from black. Also, you usually can control several soundtracks at the same time.

As it happens, adding information to the video – titles, arrows, animated arrows, and the like – is quite a bit more complicated and beyond the scope of the simplest editing programs. Most teachers tend to overdo "effects." Remember, your video is embedded in a multimedia platform that can deliver information in many ways. Keep your tapes simple. Think about first focusing students on what is important, and then adding complexity. Complexity cannot be avoided in most real-world situations. Recognizing an ecological relationship in the field is a great deal different than recognizing it in a clean video. For better learning, focus on the key points at first; then provide situations of increasing complexity.

There are at least five different ways in which sound can be used in multimedia programs:

• Earcons, sounds that indicate a particular operation
• Informational sound, the nature of which indicates some special meaning about the data
• Streaming audio, live radio-like sound
• Music
• Text to speech tracks in which text files are "spoken" in a computer voice

Sound is recognized as one of the emotionally most compelling media. It is the core means of human communication. A teacher can record the sounds made by a scientific instrument, or the calls of a bird, and put these to extremely good use. But in most Web teaching situations, sound is not very useful.

A teacher can record lectures, and have these audio tracks available over the Web. For visually impaired students, this may be particularly helpful. It is not clear how much the teacher stands to gain by having a program read text to learners, however.

Remember, it can take nearly as long to download sounds as to download movies! In some situations, however, sound may be well worth using.

Sound is a very complex area – nothing like delivering images where a few formats cover nearly all current situations and are available for nearly all platforms. If sound is mandatory for your pages, we recommend Webmaster Expert Solutions, by Morgan et al. [1996, Chapters 33 and 34] as a reference.

Sounds That Convey Information

John Flowers has experimented with the use of sound to convey information [Flowers and Haver, 1995]. In his work, as quantitative graphical information appears on the screen, there is an accompanying sound that informs the viewer of a summary of the information represented in that graph. This is a potentially powerful multimedia application.

Live audio is now broadcast over the Web. In the same way that video can be streamed, audio can be streamed. Many radio stations across the country have turned to streaming audio to increase their listening audience. If a real time broadcast of a lecture or event is desired, one way to approach the problem is streaming audio. A basic Web site can be converted to a virtual broadcast booth. A better approach might be to use streaming video, including a PowerPoint style presentation with the audio.

If spoken voice is important for learning, consider making a movie that consists of a series of still pictures with voice superimposed. In this "voice movie," you can include several images. Make a video containing the desired speech and replace the visual portion with still images using videoediting software. Alternatively, put an image in an HTML file, and link it to a corresponding sound file. If using stills with a video, make the inter-picture transitions slow cross-fades rather than abrupt cuts.

The quality of text to speech on computers has been improving steadily. To our ears, it’s up to the level of poor, possibly fair, but less than good. If you feel that spoken words are essential, consider electronic text to speech as an option. If possible, try this on your target audience. Our experience is that students quickly learn to impersonate the electronic voices, and have a great deal of fun doing so. They quickly pass the point of amusement, however, and get down to business.

When you include sound in your materials, one thing that is necessary is to be able to record and edit sounds conveniently. This requires some sound editing software.