Skip to main content

Deep Learning in Construction: What's Possible Now and in the Future

In the Summer of 2019, a report by Zion Market Research highlighted the tremendous potential of artificial intelligence (AI) and machine learning (ML) technologies in the construction industry.

According to the report, “the global AI-in-construction market was valued at $312 million in 2017 and is expected to reach $3.1 billion by 2024, a compound annual growth rate of 38.14% between 2018 and 2024.” 

Of course, the events of 2020 may result in some amendments to that forecast. But, the ability of AI and ML to support evolving workflows and address labor shortages can’t be ignored, especially for forward-thinking construction companies needing to navigate a potentially long road to recovery. So, we don’t expect interest in research and development to slow down too dramatically. 

With that in mind, we took some time to discuss one developing topic in AI and ML with two of our resident experts.

The following questions and answers were fielded by Bob Banfield, one of Trimble’s machine learning engineers, and Ross Stump, a software engineer who works closely with Bob. We’re diving into deep learning applications and how they are impacting the construction industry. 

What is deep learning and how does it relate to machine learning?

Bob: Machine learning includes many algorithms. Here’s a quick example: if you were looking to find out whether or not you are liable to get some type of disease, one type of learning algorithm might work its way through a tree of questions like, 'how old are you?' Then, 'okay, do you exercise?' And, so on. If you say yes, you go down one branch, and if you say no, then you go down another.

That's a perfectly valid machine learning algorithm. It's like the game 20 Questions you might've played as a kid, except in machine learning those questions are automatically generated.

Deep learning is a specific type of machine learning, which pretty much focuses on one of those machine learning algorithms, one called a neural network. What makes it “deep learning” is that there are many 'layers' in the neural network that you pass some sort of input through, such as an image, audio clip, or bit of text.

"Each layer is a very simple operation on its own, not like some other ML algorithms where you can have more complex operations going on. But, as the input passes through hundreds, thousands, or millions of 'layers', the cumulative effect is powerful."

It's modeled off of the brain in that each individual neuron in your brain is fairly pathetic. It's not very fast and it doesn't encode much information. But, you have a lot of them. That makes them really good for parallel processing and for taking all these little tiny operations and stringing them along through many neurons — just like the layers in the neural network — and that’s what lets you look at something and recognize "oh, that's a lion" almost instantly.

In deep learning, a computer model learns to perform classification tasks directly from images, text, or sound. 

What makes deep learning applications special as compared to other types of machine learning?

Bob: There are two kinds of conflicting ideas — what we call “models” — in machine learning. Neither one of them is bad, they're both necessary. The first one is something called bias. We’re familiar with the concept of being biased against someone based on their skin color, gender, whatever.

A heavily-biased model is based on the idea that you can make a generalization about someone with very little data. At the other extreme end is the assumption that everybody is perfectly unique, like snowflakes.

With this unbiased model, you can't make a decision about anyone because everyone is their own individual person.

The very biased model and the snowflake model will make many mistakes.

In the first case because it generalizes too much, limiting the number of questions that get asked, probably arriving at the wrong answer. In the latter case, because it doesn't ever generalize at all, the list of questions becomes almost exponential and it gets harder to arrive at a decision.

Ross: The magic of deep learning is that even as the model gets bigger, even as you're passing through more and more layers — asking more and more questions — the accuracy is not decreasing. That’s not normally what you would see in any other machine learning algorithm.

What are some practical applications of deep learning in construction?

1. Automated construction site equipment monitoring

Deep learning algorithm detects and classifies excavator actions. Source: ScienceDirect

Bob: One great example is site monitoring via camera. We already have the technology available to track equipment and vehicles. But, for it to work everywhere, all the time, everyone and everything on the site would have to be tied to one system with an RFID tag or GPS beacon.

But, if you’re running a busy construction site where you're handling all different types of subcontractors with their own equipment and vehicles, you're probably not going to have hardware on everything, so you can't just get those GPS readings.

With a deep learning algorithm trained to understand what an excavator is or what a wheel loader looks like, it could look at a real-time video feed and say, "Okay, the excavator is here."

Add in a database of license plates or other identifiers, and the algorithm can know if a specific excavator is here for a specific job.

Combine that with photogrammetry — using multiple cameras to achieve depth perception and spatial relationships just like your eyes do — and we can use deep learning on a construction site to say exactly where things are in the real world. 

Ross: Most construction foremen don’t know exactly what's going on at all times on a busy construction site, at least not for long. It’s simply not possible for one person to be in that many places at once.

At the same time, there are a lot of things that happen on a construction site that someone would like to get information on: How many loads happened that day, where the team was working that day, where they should have been working that day, if they weren't working at all or if they all took a long lunch and then eventually came back. 

"So, a camera-based site monitoring system with deep learning capability will allow you to automatically measure and eventually optimize whole site productivity." 

Bob: And, as great as they are, some of those GPS receivers they put on machines are $9,000 apiece. If you can do the same or better with a couple of site cameras, with 10 machines in view, you're talking about $90,000 worth of hardware you don’t need to buy to know where stuff is.

2. Building design optimization and automation

Ross: AI is already offering architects and engineers some really interesting options when it comes to improving building design.

Bob: For instance, solutions already exist to allow architects to design office spaces with a floor plan that maximizes productivity by taking into consideration factors like adjacency preference, so who sits by who, work style preference, noise, daylight, and views of the outside. That’s all data an existing staff could collect prior to a remodel or move to a new location, and it can be factored into the design. 

Ross: That sort of data-crunching ability offered by deep learning suggests tremendous potential for design optimization to grow, even in the short-term.

For example, a modeling system could interpret less-than-perfect drawings — even little doodles you make with a pen — and professionally render them based on relatively limited new input and a huge backlog of previous data. 

You could draw a rectangle in a certain position and the system fills in a complete window, or it brings up a list of 10 different windows you've used in the past, and you just select which one you want, then you move on. It’s so much quicker and simpler than digging through content and saying, “I need a window that’s made by so-and-so and is this big…” 

Bob: Taking it a step further, your modeling software could take those 2D points and instead of going into a 3D model, they can go straight into a 3D augmented reality viewer right out on the construction site. 

3. Vehicle maintenance and site data via audio analysis

Engine fault detection via a deep-learning algorithm that listens for certain sound patterns. Source: Hyundai Motor Group

Ross: Another really interesting concept being explored right now involves being able to gather data from the sound a machine makes as it works or the sound of its engine or other parts.

For example, a deep learning algorithm could analyze the sound of a bucket hitting rocks or gravel, and determine what material that is. Then, it can start making estimates about that material in terms of density or what we know about gravel or big rocks, and you can expand that out. Fuel consumption, daily load quotas, etc. 

Bob: And, listening to the engine or transmission, or whatever, is about preemptive maintenance. In the life of a Caterpillar excavator (which runs about $500,000 to buy), Caterpillar expects to make $500,000 in parts in preemptive servicing on that machine in its lifetime. So, double the value of the machine and that's how much you pay in parts.

There's this huge push for predictive maintenance and deep learning offers that almost human capability.

"We all probably know a mechanic who can just listen to a running vehicle and diagnose a problem. With an adequate database of audio references, deep learning and a microphone can give anyone that ability."

4. As-built site data

Ross: One of the most powerful benefits of BIM-powered field technology is the ability to bring digital information into the built environment in numerous ways. And, data from the built environment can be digitized and manipulated just as easily. The robotic total stations and laser scanners currently in use are already making quality control and future maintenance decisions so much easier than they used to be. 

Bob: Taking that concept a huge step further, deep learning can be combined with total stations, cameras, drones, or robots to completely automate onsite processes like scanning and quality inspections.

The input data would be the model that you're building out, the 3D model of the building, and a collection of images or images of the built environment. What you're looking for are pieces that weren't put in place, things that were installed incorrectly, but doing that all with camera and machine learning. 

What do you think is the future of deep learning in construction?

1. Predictive recommendations

Bob: One of the most compelling next steps would be seeing technology like the site monitoring system we were just describing evolve into a more predictive, pre-emptive tool to improve site optimization right from the planning stages on.

That’s quite a ways down the road at this point. But, once a system like that collects and analyzes enough data, it could reasonably say, for example, that those who have excavators on their construction site first finished faster under X, Y, and Z conditions. Then, when a job matched up well with those conditions, it could recommend scheduling the excavator first.

It’s really the same thing with the building design optimization angle. If you have the right type of network and you know a ton of data about your input parameters, this is where machine learning almost looks like magic.

"Could a deep learning machine actually design an entire building?" 

If it's seen so many buildings and knows all of this information about them, then it can kind of work backward and create the building from the data and the preferences. 

2. Self-driving heavy machinery

Bob: Another big field of opportunity is self-driving machines.

Of course, autonomous vehicles are already on the road, but there’s a pretty big difference between automating the morning commute and looking at a half-million-dollar bulldozer without a cab. But, there are compelling reasons to pursue the technology.

"There’s already a serious labor shortage, especially for highly skilled positions like the men and women who operate those big excavators."

In terms of the labor shortages there, and the mistakes they make from not seeing a power line that a neural network might have seen, I think that's going to change a lot faster than we originally thought.

What resources do you recommend for anyone who wants to learn more about deep learning and neural networks?

Ross: Here are a few of our favorite videos and articles: