This blog post was originally published at Retrocausal’s website. It is reprinted here with the permission of Retrocausal.
2020 was a pivotal year for everyone, and the manufacturing industry is no exception. The retirement of experienced workers, increase in temporary workforce, continuously changing processes and higher mix assembly driven by economic factors and customer demand have globally increased the number of manual assembly errors causing manufacturing companies to reevaluate their processes
These mistakes have increased rework and scrap costs, often measured by the “first time yield” metric, and are estimated to be behind 68% of overall quality challenges facing discrete manufacturing. At the same time, increasing profit margins by minimizing takt time is becoming a higher priority for manufacturers to survive potential years of economic recession and an industry in geographical, technological, and market segment transition.
Technology has responded to these needs by offering IoT, computer vision, artificial intelligence, and augmented reality-based solutions. However, a factory floor is a complex hybrid of humans, equipment, processes, infrastructure, and locations, and it is never trivial to assess whether such solutions are right for a particular operation. Correspondingly, long and failed pilots are the norm, and some operations fail to benefit from technology even after spending years and hundreds of thousands of dollars. At the same time, manufacturing leaders agree that without rapid integration of computer vision technology, an operation will soon lose its competitiveness. They are left wondering how to not only achieve success of a pilot but also evaluate its success.
In this article, we outline a holistic approach on setting up a digitized computer vision pilot for success and evaluate vision based systems for accuracy, ease of use and scalability.
Setting your pilots up for success
Our team at Retrocausal has years of experience deploying computer vision pilots by companies of all sizes, from small companies to enterprise manufacturing floors like Microsoft, Siemens, Toyota, Boeing, and Whirlpool.
Evaluating the success of a pilot can be difficult, however we’ve identified three fundamental elements that must go into planning.
- Clearly defined metrics: Production teams need to have concretely defined metrics, targets on these metrics, and a clear understanding of the ROI, before the pilot can be deployed. Oftentimes, innovation projects are set up without an understanding of baseline metrics. What is the first-time yield or manual assembly error rate for the process that is being “mistake-proofed”? What is the ROI in dollars if this error rate could be halved? What are the acceptable tolerances in terms of false positives or slow-downs to the line? We strongly recommend defining these relevant metrics, and setting expectations for both internal stakeholders and external vendors upfront. This streamlines the POC process and keeps all parties focused on the actual goals.
- Cross-department checks and balances: Solutions have a higher chance of success when there is accountability for the teams driving them from day one. Pilots that are done in R&D or innovation teams alone rarely manage to get deployed, despite having the potential to solve a significant pain point for the factory floor. However, both the manufacturing innovation team and the vendor loses their interest if pilots drag on for too long. That is why it is important to involve plant operations, quality engineering, or industrial engineering from the beginning of pilot discussions.
- Deployment planning before the pilot: A recurring observation in this space is that even successful pilots fail to add value to a manufacturing process, because nobody actually thought about the deployment process upfront. We find it crucial to involve all stakeholders, including the economic buyer in the pilot process very early in the discussion, and certainly before pilot kickoff. There should be a clear plan for rollout before the pilot initiates. Sometimes the findings of the pilot will require amendments to this rollout, and that is completely fine. However, failing to plan the final deployment, is planning to fail.
Criteria to evaluate your pilot
Assembly lines are pinnacles of human collaboration, where everyone from associates to supervisors, industrial engineers, quality engineers, design engineers, safety personnel, IT and facilities management, as well as finance teams come together to make production a success. That’s why, it is not at all surprising that various stakeholders have different criteria to evaluate any changes to the line.
In the following, we collect a list of criteria that present a holistic assessment of any new “digital” addition to the factory floor.
- Use cases- Live Task Guidance vs. Analytics vs. Traceability: Successful pilots involve multiple stakeholders in the conversation early-on, even if only one of them leads the evaluation. Often, quality control is the urgent pain point, whereas analytics and traceability are nice-to-haves, so quality teams may want to own the pilot whereas industrial engineering teams piggyback onto the process. This can also be helpful in terms of framing the early pilot to associates. At the same time, every vendor doesn’t provide every capability. For example, some have a limited offering focused on analytics alone. Even when a vendor provides a capability, for example automated procedure observation, it may be limited in the diversity of procedures it can work with. Some solutions can only track human body postures, which means that they may be limited to processes where tools and parts are held at fixed locations and process observation can be summarized by human movement observation only. (See more in our blog post here) Others need every frame of a video to be sent back to the cloud, so the communication lag alone makes it impossible to guide the worker in real-time. Nonetheless, having a clear understanding of what an operation can benefit from can narrow down the list of potential partners.
- Accuracy and false positives: Artificial Intelligence and computer vision techniques are being used in safety critical applications such as self-driving cars and medical diagnoses, which indicates that they can be versatile and robust. However, these techniques are far from a commodity, and most solutions will not be able to capture all worker mistakes (have less than 100% true positive rate). These solutions will also have a non-zero false positive rate which means that they will sometimes trigger false alerts or measure some cycles incorrectly.We find it valuable to measure accuracy and false positive rate explicitly. Manufacturers who have been successful in benefiting from such solutions, have designed careful experiments to measure accuracy. For instance, one manufacturer performed 100 “test” cycles of their assembly process, 50 “correctly” and 50 capturing a variety of mistakes. This helped them calculate metrics under controlled conditions, within an afternoon.If the manufacturing team has estimated the ROI in dollars for each mistake that gets avoided or for every second that is saved in takt time; they can immediately ascertain the overall ROI for the operation.
- Time Required to Program: An important metric to measure is the time needed to set up the solution for a new workstation environment. Processes can change rather frequently, and any solution which takes months to customize is probably not appropriate for the factory floor. This is a metric which should be discussed upfront in the earliest conversations, but also measured during the pilot.
- Cost per station – year 1 (including infrastructure costs): Obviously, it is important to ascertain cost per station as well as for the use of existing infrastructure. Unless these costs are significantly below the projected dollar ROI, the solution may be dead-on-arrival. In many cases, ROI and cost per station may both vary depending upon the scale of deployment. We know of manufacturers who have invested hundreds of thousands of dollars per station and one or more years building in-house solutions for their mission-critical processes. Obviously, as the scale of such deployments increases the ROI goes down, since they are no longer addressing workstations with the highest impact on the production only. At the same time, average costs should also go down. Again, it pays to discuss the scope, phases, and budget of deployment upfront.
- Recurring cost per year: Software-as-a-Service model has been universally adopted by the technology industry. These subscription-based models are new and sometimes uncomfortable to manufacturing operations, which are used to investing in technical solutions on a CAPEX-basis. However, these OPEX-based models provide the opportunity to spread the true cost of these solutions over years, while benefiting from continuous software updates and vendor support. It is important for manufacturers to understand such recurring costs. Often it helps to lock in prices by signing multi-year contracts.
- Out of Standard Operation Detection: In addition to measuring accuracy and false positives, manufacturers should also evaluate the technology under anomalous circumstances. A solution that relies exclusively on human body poses may fail when there are multiple people in the camera view, or when the unit being assembled on the workstation also moves. Is the solution robust to lighting variations? Is it able to handle situations where the camera moves? What if a worker changes the color of the gloves they wear? Some of these questions can be asked upfront, whereas others may need to be evaluated.
- Wrong Part Picking and Tool Use Detection: In certain situations, in addition to detecting “missed steps” a quality engineer may want to explicitly verify that the correct part was installed or the right tool was used for the job. It may be important to evaluate these issues in the POC alongside “missed step” detection.
- User Interface: Systems that will be used by ordinary line supervisors and manufacturing associates need to have easy to use user interfaces. Manufacturers should ensure that the technology being proposed isn’t just a raw computer vision prototype, but that it provides a practical mechanism to interact with the system. Does the UI provide a convenient way to pause the system? Does it provide a way to provide feedback to the system in case it detected an error incorrectly? Is the dashboard or notification accessible via supervisor’s tablet computer (e.g. iPad or Androids).
- Ability to stop the line: An important question to ask is whether a solution can be used as a true “Poka Yoke” system, i.e. to stop the line when a mistake is detected. Several vendors provide exclusively standalone solutions which will not talk to another system. Some vendors provide APIs which may be difficult to utilize towards integration, whereas others provide full-stack services, where their solutions engineers will work with a manufacturers’ IT team to come up with optimized solutions in a short period of time.
- High-mix assembly: Depending upon a manufacturer’s product offerings, it may make sense to explore what kinds of product or SKU variations they are able to support on the same workstation, and what it takes to enable these variations in the product. Again, different solutions vary significantly on this front. Some solutions strictly assume the same product or SKU to be assembled on each workstation. Others need integration with MES systems. Yet others provide off-the-shelf solutions e.g. through visual object identification (differentiating between an SUV and a hatchback to load appropriate work instructions) or bar code scanning.
- Worker privacy: It is important to be respectful of associates’ privacy and ensure their personally identifiable information is not being stored. Often cameras can be positioned in such a way that they only observe the work and not the worker. However, sometimes it is unavoidable to also capture the workers involved in the assembly process. In these cases, it is absolutely necessary that the platform provide support for facial blurring.
- Integration Complexity Rating: Integration complexity depends both on the vendor’s solution as well as on the Manufacturing Execution System (MES) deployed at the manufacturers’ facility. While manufacturing CIOs are driving a move towards de-customization, the majority of MES systems are homegrown and a reflection of the manufacturers’ own proprietary processes. Often for first deployment, manufacturers prefer standalone solutions, which can ingest a one-time download of SKUs and procedures, and start delivering value immediately. For expansions, or more complex operations, especially those involving high-mix assembly, manufacturers should be looking into whether their own MES systems provide a standard integration layer, and whether a relatively disjoint integration of the two systems e.g. merely through exchange of JSON files is possible. On the analytics front, they should check whether the vendor solution supports integration with off-the-shelf frameworks such as Snowflake or Databricks solutions. This often simplifies integration significantly.
- Infrastructure Security Requirements: Information security is an important consideration when digitizing valuable process information for a manufacturing operation. Manufacturers should ask whether the vendor utilizes leading cloud platforms such as Microsoft Azure and Amazon AWS, which provide robust security infrastructure as part of their offerings. Alternatively, if the infrastructure leveraged by the vendor is their own, it should be evaluated separately by the manufacturers’ IT security team. If the vendor possesses or is working towards industry standard certifications such as ISO 27001, that is a positive sign.
Conclusion
Manufacturing leaders and numerous studies [1,2,3] consistently highlight the importance of digitization. Computer vision based technologies provide an opportunity to capture previously invisible parts of a manufacturing process that traditional IoT solutions could not capture. However, it is important to evaluate these solutions before they can be deployed. The only way to extract value from your digital “exploration” dollars is to do a lot of preparation upfront. This includes rallying problem-owners and doing the homework on proper evaluation of pilots. We have outlined the most important criteria along which a new computer vision technology should be evaluated to ascertain its relevance to a manufacturing operation.
Zeeshan Zia
CEO & Co-Founder, Retrocausal
