Data management is a major component of any CCTV inspection program. The volume of digital files generated from ongoing video pipeline inspections can be massive, and, if not managed properly, could result in disaster.

In 2001, viewing and storage of inspection videos changed dramatically. The transition from VHS tapes to digital recordings was a significant upgrade, which made the computer the primary storage medium.

Digital videos can be quite large in size. An average 300-foot inspection using standard equipment can range in size from 200 to 500 megabytes. That means in an average day, 5 to 10 gigabytes of recordable storage may be needed for a single inspection unit. Side scan or fisheye camera inspections require even more space. A 300-foot run using this technology can take from 600 to 800 megabytes of memory.

Data files can add up very quickly and become unmanageable if not organized properly. For companies and agencies with multiple inspection vehicles, the volume of data can accumulate exponentially. A well-thought-out plan for storage and tracking is critical in preserving this all-important information.

As the computer's memory fills up, video files must be offloaded in order to make room for new inspections. Allowing a hard drive to reach its full storage capacity can have dire consequences. Even getting close to maxing out the memory could result in a total system crash. If this happens, all of the video files and all of the information on that hard drive will be completely lost.

There are services that can recover data from damaged hard drives, but they can be very expensive and aren't always successful.

The best way to keep this from happening is to limit the amount of information that can be written to a hard drive. By right clicking on the appropriate drive and then accessing "Properties" on your PC, a tab will appear called "Quota." This tool allows you to designate a limit on incoming data. For instance, if you place that limit at 80 percent, the hard drive will not accept files beyond that point. This procedure helps remind operators to download their files long before the critical point is reached.

Transferring data

Most inspection software programs allow you to designate file name parameters. Fields in the header can be selected as the basis for file names. Each video file must contain a unique pipeline identifier. Most of the time, the file name is the pipe ID itself, or a combination of both the upstream and downstream manholes. Sometimes the upstream manhole number is used exclusively.

If a pipeline is inspected more than once and the same file name is used, a conflict will exist within the database. In this case, a modifier should be added to ensure multiple files of the same line segment can reside within the database. Possible solutions include incorporating the date of the inspection into the file name or using the database survey number as the modifier.

Exporting into a GIS system database may require some outside programming help. Since most of the inspection software programs are Access-based, they can easily be set up to export into any existing GIS system. The key is to ensure that all of the critical data is transferred correctly and that each field is populated appropriately.

The process of offloading inspection files from the main computer has been greatly simplified with the existence of external hard drives. One- and 2-terabyte hard drives have become very affordable and have proven to be an ideal medium for transferring and backing up data. In fact, a single hard drive is capable of holding a video library of an entire collection system.

The process of transferring data can still be time consuming, but unlike DVDs, which had to be babysat, the hard drive can be left on its own to transfer data until it's complete. It can take up to 30 minutes to transfer 100 gigabytes of data, but that time can be reduced significantly by upgrading computer hardware. A faster processor, for example, will increase speed significantly. Increasing the RAM and installing a high-quality data cord can have favorable results as well. The type of USB port being used can also affect transfer speed. Fire Wire or SATA connections will improve transfer speeds significantly.

Backing up data

Backing up pipeline inspection files is critical. The cost of the hard drives and the time required to copy data is cheap insurance. Computers and hard drives crash, and backing up data is the best way to prevent loss.

The replacement cost of a hard drive can be staggering. If a 500-gigabyte hard drive crashes and no backup is available, over 2,000 video files could be lost. Re-inspecting those lines could cost well into six figures.

A good practice is to require daily backup of all files. This will ensure that inspection videos are never lost or compromised.

Backing up hard drives redundantly is highly recommended for both contractors and agencies. Having duplicate copies placed in two separate locations is the best way to ensure these important files are preserved. If a fire or other disaster destroys one hard drive, a second backup will still be available.

Many agencies today post their videos to servers. This is an ideal storage medium that allows access from any computer on the network. Video files are linked to the GIS system and can be viewed from the sanitary sewer layer. By simply clicking on a manhole-to-manhole segment, the information corresponding to that particular line is displayed and the linked inspection video can be viewed.

As a contractor, I guarantee my customers that I will maintain a backup of all CCTV pipeline inspection videos for a period of five years. This is good business and assures them that there is a triple backup in case both of their systems fail. But realistically, there is no reason to ever erase them.

Quality control

Data management cannot be addressed without mentioning quality control. Part of the process of managing data is ensuring that it is thorough, accurate and complete. Each individual file must be reviewed to ensure that its contents are entirely correct. We must make sure that every video is of acceptable quality and that each information field has been annotated accurately.

A well-developed data management program is the key to any successful CCTV pipeline inspection program. By adhering to each critical step, a program will run smoothly and effectively, ensuring that goals and objectives are met satisfactorily.