How Do You Process Drone Survey Data?

Interviewer:

[00:00:02] How do you process drone survey data? So after you're finished with your drone, you have all your footage shot, you've returned from the field. Now, now what happens?

Field data verification and backup

Bob Foley: Engineers With Drones

[00:00:14] Yeah. Okay, so we'll take a very slight step back. You don't leave the field without verifying that the data you need is present, and present in more than one location. So you never want to get back from a day in the field and find that your little SD card, which has, you know, 20,000 euros worth of captured data on it is now nowhere to be found. So whenever the drone lands, you're backing up your data on a second device so that you now have it in two separate places. That's the start of the processing journey. We're also verifying that the data is actually valid and correct and usable. We didn't leave a lens cap on or something silly like that and fly an hour's worth of mission and captured nothing but black images. So all that sort of thing is important before you ever even leave the site.

Because, one of the hardest things in drone land is actually mobilising to site. Data capture itself is relatively benign compared to actually the whole process of getting going and getting on site and setting everything up, etc. etc.

Two processing paths: reports vs technical deliverables

So it starts there. The next step is always to decide what sort of processing you're doing. So once again, in broad strokes, you got two big paths: either you're processing into some sort of report or you're processing into some sort of technical deliverable. Okay.

A report being something like a PDF or putting it on some sort of online platform or maybe it's just raw photography or videography that the client wants, but all of that is somebody's gonna take this imagery, this human readable imagery and turn it into something that another human can look at and make decisions on. Like is the roof of my building in trouble? Is there storm damage after forestry? Have the trees fallen over? How are my wind turbine blades? How are my electrical pylons? How was the surface of the spillways of my hydroelectric dam? All of these are not technical deliverables. They're imagery that another person will interpret or we will interpret into a report and present to another person.

Alrighty. In terms of that side of things, I can touch on that quickly because it tends to not really be heavy duty when it comes to the processing side.

For the basics, the very basics like imagery and video, the worst thing in the world you can hand somebody is a single folder with a thousand images in it and it starts with DJI 001 and ends with DJIs 1000.

Interviewer:

[00:03:18] Right

Bob Foley: Engineers With Drones

[00:03:19] You're like, what am I looking at? Where is this? You know, it's impossible to understand that. So one of our jobs is to make sure that when we present deliverables like that, that they're somewhat useful and readable straight out the gate.

So, if we have been asked to come in and present raw imagery of a building, let's say, we don't give them a folder with a thousand images. We give them 10 folders that says North facade, South facade, roof A, roof B, roof C, or whatever system that they use in their site for describing the various buildings and things like that. So as that anybody at all can approach this data set and say, okay, I'm interested in building B right now. Well, there's a folder called building B, click, Okay, there's north, south, east, west, roof, okay, make sense. I can go, I'm interested in the north side. We have issues there, double click, and now I've got 200 images of the north side of that building, and now I can go through those images and it'll make sense.

Avoiding "inspection through a straw"

And in the field, it's important that we data capture correctly for this person. So, what I mean by that is there's a term called conducting an inspection through a straw. And what that means is imagine I have a large object I want to inspect.

But if I have a really, really high zoom lens, really good camera, basically I'm conducting an inspection through a straw. So I'll find a problem with a bolt. There's a problem with a bolt. But I have no idea where it is in the entire structure. That's the problem of what we call conducting an inspection through a straw. For this reason, good inspectors know that you need to take a lot of what are called orientation or overview images.

So I'm inspecting the roof of a building, click, I'll start wide. This is the roof. Big wide shot of the roof. This is the side of the building. Okay. Now the person looking at the imagery knows where they are and they can identify any smaller defects that are taken at a high zoom within that larger image. You see? So all of that sort of thing goes into the processing to make sure that the data set stands on its own as an interpretable piece of knowledge that anybody can approach and utilize. So even the raw imagery capture side of things, it's important to actually present that well.

Same applies to video. If we were giving somebody videography of things, we tend not to do that as much for a multitude of reasons, but sometimes we do. If you are doing videography, it's very important again that orientation happens. Where is the start? Starting wide, starting orientating. We are in this area, we are looking at this particular piece of infrastructure and then getting close in and having a good look around. And then when finished, coming back out again, you know? So all of that's very important.

Even file naming - we rename most of our files to be applicable to the asset that they're dealing with. Turbine 1, Blade B, Leading Edge, you know. Pylon number 43, whatever, you know.

Interviewer:

[00:06:45] Yeah, yeah, I love that idea. I love the idea that someone could just land in on a folder and pick out one image and they would be able to ascertain through the folder name and the image name what they're looking at. Than what you were saying, just giving a dump of, know, file 0021 or whatever.

From raw data to inspection reports

Bob Foley: Engineers With Drones

[00:06:59] Yeah. Yeah, like, and that, as I said, we're talking about drone data processing here. This is at the most basic level. Take a thousand images of my structure and present them to me. It still requires processing. There's no world where you just go, here's a thousand images, deal with it. That's not a thing. You must present it usefully too, because you will never meet usually the person who is going to interpret that data. So it must stand on its own as understandable.

Let's see, let's see what else then. So yeah, that's raw imagery, raw video, things like that. Next step up is reporting. So with reporting, we still go through the initial process of turning raw data into a good folder structure. Once that's done, then if we are reporting, the client probably only wants a PDF. They don't want a thousand images. So if we are reporting, we will still go through the initial process, structure it all correctly so that we can then generate a report out of that.

Finding and selecting the best defect images

And then we will go through all of the imagery, find all of the defects. A lot of the time you'll have maybe 20 images of the same defect. So it's very important for the report generator to know or to be able to choose the best image to communicate the size, scale, extent, quality of any given defect correctly. What you don't want is an image that half captures a defect or captures it at such an angle that you can't really tell how big it is, how wide it is, how deep it is, how severe it is...

So if we have 20 images of a given defect from multiple angles, We'll first of all realise, okay, there is a defect in this area, then find the best image to communicate that defect. And it could be that we use a broad image, you know, a large image of a roof with a big hole in it. And then we use a tighter image in association with that or two or three tighter images to show the technical, it cracked along this line here or, you know, the structure was weak over here. You can see the wood is rotten or something like that. You know, so that applies to every single defect. So on a given structure, you will go through all the imagery, isolate out all the defects, find the best image or set of images to communicate any given defect, and then add those into the report. The report is always written out in this sort of instance. You always want at three key elements: classification, severity, and a technical description. The first thing is a classification of the defect.

A classification being broadly what it is. Is it lightning damage or is it erosion? That would be a wind turbine blade. Is it corrosion? Is it rot? Is it spalling of concrete? Something like that, but a broad classification.

The four or five point severity scale

You always want a severity. So, we typically use either a four or five point severity system, depending on the type of asset that we're working on. Some clients prefer four or some clients five, but fundamentally it starts at this is a piece of information that we are giving to you. It's not super important to do anything about it, but we thought you should be aware. That can be anything like grease stuck to a wind turbine tower. The world isn't going to end because of it, but at the same time, the client should realise that their blades are dirty or there's grease on the tower or something like that. The classic one in Ireland is, almost every wind turbine blade in Ireland is covered in green mildew because we live in Ireland and it's very moist here.

Interviewer:

[00:11:10] Warm at the moment too, like warm and damp.

Bob Foley: Engineers With Drones

[00:11:12] Yeah, so clients are not going to go cleaning all the wind turbine blades, but we will let them know, look, this is the condition of the surface of your blades. OK, yeah, that's level one. Level two would be a bit of a problem here. Next time you're doing something, you probably want to look at this. Level three is more like this is a fairly serious issue. I would probably be looking at conducting some sort of maintenance action on this.

And then depending on the levels that we're utilising on any given structure, Level four would be something along the lines of this is severely dangerous. There is distinct risk to human life or the structure if you continue the operation of this. The classic examples of that would be something like a slate on a roof that is clearly loose and hanging over O'Connell Street in Dublin or something like that. The next gust of wind could push that slate off and down onto the street and hurt somebody. So, that's the sort of thing. Something that will also very much be a phone call direct to the client saying, you've got a big problem, you need to deal with it straight away, etc. A wind turbine blade that clearly has a large crack down the trailing edge that may catastrophically deconstruct itself anytime soon. That sort of thing. Turbine needs to be shut down right now, any big wind, you could have a serious issue. So there's a level of severity in our reporting that does all that.

Yeah. So as I said, we get we get all the defects, we compile them, we classify them, we put a severity level against them. We also put a technical description, a piece of text that describes what the defect is, because a lot of the time, the next step onto that is somebody just generates a work list off of our report and they need a piece of text that describes the problem so that anybody reading into the future goes yeah I know what that is. And then all of that goes into a report there's a summary generated all of the images are of course put in and then that is presented to the client. The client may have a requirement to adjust the report depending on who they are and what they're looking to do with it.

Deliverables: PDF reports and spreadsheets

We also typically with that, a PDF report tends to be very much a human-centered product. That is to say, a human can read it, they can look at it, they can understand it. What a lot of clients like to have as well is a basic tabular spreadsheet with all of the defects listed that can be imported into a maintenance system easily as a line item of work to be carried out. So if there's 30 defects, they can instantly sort and filter them, delete the ones they don't wanna deal with at the moment, and import them into a maintenance system and send them to their maintenance technicians to be dealt with instantly. So, PDF for the humans, spreadsheet for the machines, let's say.

That's something pretty much standard give on a lot of our higher level technical stuff. On the reporting side, the asset inspection reporting side, that'll be about it. That'll be applying to wind turbines, electrical pylons, civil structures, buildings, roofs, all sorts of structures, radio masts, could be anything. That would be the typical sort of path it would take. That PDF plus spreadsheet is entirely usable by many engineers. It can be emailed to other people. It can be passed around. It can be imported. But anybody holding that particular document has a very good idea now of the status of the particular asset, whatever that may be.