Faculty of Computing: Design Study of a Smart Collaborative Robot

This report details the model-based system engineering of a smart collaborative robot designed for palletizing tasks. The project encompasses the integration of vision, voice control, and IoT capabilities to enhance the robot's functionality and safety. The report begins with an introduction to collaborative robots and their applications, followed by a literature review of relevant technologies and methodologies. It then outlines the project's objectives, including the development of obstacle detection, pathfinding using the A* algorithm, and the design of a gripper system. The methodology section describes the application of Model Based Systems Engineering (MBSE) for the design process, utilizing SysML diagrams and other tools for requirements specification, design specification, and the creation of block definition diagrams. The results and analysis section provides insights into obstacle detection techniques, including pre-processing, feature detection, and object classification, as well as the implementation of the A* algorithm for path planning. The report concludes with a summary of the findings and a discussion of potential future work.

Model Based System Engineering Of a Smart Collaborative Robot with Vision, Voice and IoT Capabilities

Table of Contents
Introduction......................................................................................................................................2
Aim & Objectives............................................................................................................................2
Literature Review............................................................................................................................3
Methodology....................................................................................................................................4
Requirements Specification.............................................................................................................5
Design Specification........................................................................................................................7
Design Concept and Final Design...................................................................................................9
Results and Analysis......................................................................................................................11
Obstacle detection......................................................................................................................11
Path finding using A* algorithm................................................................................................14
Conclusion.....................................................................................................................................20
References......................................................................................................................................21
1

Introduction
A 1995 research mission (General Motors Foundation) was the basic foundation of the
collaborative robots. The main target of the mission was to devise a method which would enable
the robots (or equipment resembling robots) to work with the humans in their specific
environments in a highly safe manner. Crucial robotic manufacturers (in the collaborative
market) such as KUKA, FANUC, Universal Robots, Motoman and ABB are continuously
working towards the development of responsive and smart robots. Along with making certain
improvements in the functionality of the app, the collaborative robotics movement is also
lowering down the required space in respect to a robotic unit. Numerous benefits are being
offered to the production lines, business houses as well as workers by the collaborative
technology [6]. Cobots have the capability of undertaking varied roles across different industries
as well as operating in almost all work environments along with human beings. Apart from this,
these collaborative machines are also capable of performing numerous tasks like commodity
packing, assembling, palletizing and others.
Aim & Objectives
This project deals with the development of a sharp, collective palletizing robot (in respect
to the manufacturing industry) who has the potential of working along with human beings. So
“safety of humans” must be an important component of the robot’s design. For this, features such
as voice-control, vision, IoT are crucial. Below we are going to mention some activities
(project’s objectives) which are crucial to be undertaken for the achievement of the above
discussed aim.
• Undertake the primary study in respect to the collaborative robot.
• Gather crucial info in regards to collaborative robot.
• Undertake the literature review
• Performance of the design creation process
• Performance of the substitute process of examination and selection
2

• Undertaking the task of design development and duplication
Literature Review
The collective robotics app. allows the robots and human beings to work together in a
free environment in an effective and safe manner without the fear of injuries. Numerous latest
software, sensors and EOATs in the collaborative robots (cobots) aid them in quick and easy
detection of any interruptions in the work place so that they can be timely adapted to. Generally,
cobots have round type shape[5]. They come without any internal wires and motors as well as
pinch points. Also, any irregular force being applied to their joints can be easily detected by them
during motion. Apart from this, such programming can also be done to these robots which will
enable them to stop or reverse their positions whenever they face any human contact. Being
guided by hand, their programming as well as implementation is known to be made highly
simplified. This means that once the worker directs the robots of the desired paths, they can
easily and automatically repeat them[4].
Author Name Intext
citation
Year Method Performance
Lars Bretzner, Ivan
Laptev and Tony
Lindeberg
[7] 2002 Hierarchical Models,
Multi-Scale Color
Features, and Particle
Filtering
color prior: 86.5%
No color prior: 45%
Junqiu Wang and
Yasushi Yagi
[8] 2009 Adaptive Mean Shift 64.11%
Jifeng Ning [9] 2009 Mean Shift, Joint Color-
Texture Histogram , and
Local Binary Pattern
JCTH :8.22
LBP:10.78
Mean Shift:2.83
Kaihua Zhang et al [10] 2013 Active Feature Selection 83%
Lokesh Selvaraj and
Balakrishnan
[11] 2014 IP-HMM 97.14%
3

Ganesan
Jiangyang Zhang,
Shangwen Li, and C.-
C. Jay Kuo
[12] 2014 Video retargeting system
and Compressed-domain
94.81%
Methodology
Model based systems engineering is a systems engineering technique which tends to
utilize visual modeling as the initial source of info exchange. When compared with the old info
exchange, MBSE promises to remove the exchange of any unimportant info. This is because it
depends on the abstract models which store only crucial data. This way, it gets simple for the
engineering teams to discuss the design intent, get aware of the influence of the changes in
design as well as examine the design of the system before they start to build it.
The first cycle begins with the Concept-of-Operation stages, setting up the cases,
essentials and plans. Either a SysML modeling tool, a requirements management tool or some
amalgamation of both might be used to undertake it. This is the essence of the primary effort at
the system architecture, by choice in the SysML tool in place of MS Office (or Vizio).
After multiple starting cycles, the development process is known to achieve expansion in
regards to people and tools. Huge levels of efforts are put in the right-hand quadrant by the
domain engineers utilizing CAD or programming environments along with the accompanying
PLM/ALM repositories. However, the SysML model may stay as a very necessary part of the
TSM. The distribution of the system design data across multiple varying tools turns the Synthesis
process highly difficult. Creating such a graph of connection among the tools which would offer
help in recollection of the discrete engineering efforts is the first job of an MBE platform like
Syndeia.
Next job of the MBE platform is the development of a model which is transformed to
special examination as well as duplication tools which offer regularity with the present
4

synthesized TSM. The last job of the MBE platform is to bring the simulation and examination
outcomes inside the V&V quadrant so as to give a picture of the system development’s ongoing
position.
With the advancement of the system development, extra tools and people are added up in
the process. More comprehensive designs and thoroughly authenticate interpretations are made.
This is followed up by the execution of the non-digital activities related to building,
incorporation and testing. At this point, the MBE Wheel takes the image of an onion consisting
of multiple layers. This leads to a rise in the expectations on the MBE platform. In place of
creating new analysis models, consideration should be given to the fact that the framework is
capable of comparing the current models and improving them as needed. The growth of
connections is posing a huge challenge in regards to efficient tracking of links across the TSM.
Requirements Specification
The palletizer robot which has been presented here is known to move a particular box
from A to B place during which it navigates and prevents crashes. For goal achievement, the
below mentioned tasks have been faced:
 Usage of contact and sonar sensors for environment perception.
 Path planning strategy on the basis of A* algorithm.
 Performance of the path of robot and interaction of the robot with the changing obstacles.
The force and velocity needed for acquiring and moving an object from one location to
another (the servomotors not to be forced), was taken as the base for gearbox designing. Total 8
gears i.e. 1 of 36 teeth, 3 of 16 teeth and 4 having 24 teeth were utilized by the gearbox.
5

Design Specification
Use case diagram
6

State machine diagram
7

Activity diagram
8

In the diagram, A* algorithm has been utilized by the action “compute trajectory to the
box” for receiving an already defined environment’s map along with all the fixed obstacles on it.
The programming of the action is undertaken in 2 stages on the DaNI robot: 1st: the robot
moving to the position of box, 2nd: the robot moving to final pallet. Once the robot comes to the
box position after performing 1st stage, the gripper will be configured and positioned to take the
box under the action “set gripper ready” whose programming was done on the NXT Lego. The
trajectory’s 2nd stage followed by the robot moving to the “go to final pallet” action is
performed once the indication regarding the gripper taking the box is given by the sensors. The
robot finds out the gripper which is followed up leaving the box on the pallet properly. After this,
the robot returns to the starting position and the process is repeated again for other boxes. The
sonar sensor finds the dynamic obstacles during which the robot stops, however, the path is
continued on no detection of obstacles.
Design Concept and Final Design
Block Definition Diagram
Girders, gearwheels, 4 touch sensors, angle brackets, bricks, 3 servomotors which are a
part of the Lego kit are required by the robotic arm assembly. The arm’s mechanical part is
9

moved up and down by 2 servomotors. The griper is opened and closed by 3rd servomotor. One
touch sensor (at arm’s base) senses the arm’s lower position whereas the 2nd touch sensor senses
the higher position the arm can reach. Located on the griper, the 3rd and 4th sensor measure the
opening/closing degrees under servomotor’s control. The above figure offers the final robotic
arm’s 3D model whereas the below figure deals with the real robotic arm. To ensure proper
holding of the object, the gripper has 2 jaws.
10

Results and Analysis
Obstacle detection
The developers can resort to the below mentioned 4 phrase strategy for the purpose of
overcoming these issues:
 Pre-processing- After collecting the data from sensors and cameras, the data gets
converted to a highly beneficial form. In this case, for the purpose of voice command
identification and record mikes, IoT sensors as well as 3 cameras were utilized.
 Detection of Feature- The pre-processed data is used to fetch the important features like
edges, corners and others.
 Detection and classification of Object: The features are used for object identification
which is followed up by classification based on known feature maps.
 Tracking & navigation of object- The identified objects are exposed to time tracking. As
the robot tends to navigate, this may be inclusive of both the objects as well as dynamic
environment viewpoints.
 Servos can be controlled; decisions can be taken and other top-level robot-based tasks
can be undertaken based on the generated data.
Real world data is collected by the robot with the help of multiple cameras and sensors.
However, for meeting the set goals exact predictions and measurements are needed, for which
this data may prove to be raw. So, in order to clean the data to make it more beneficial, Digital
Signal Processing might be resorted to. For instance, resizing, contrast enhancement and gamma
correction can be used in case of cleaning the image data.
Proper planning as to how much and how quick the image data is to be collected is a
must. Since 2 stereo images can be supported by the robot, so it’s crucial for the system to
undertake 2 planes process simultaneously. Numerous resolution configurations in the range of
16-32 megapixels as well as frame rates (In 30-60fps range) are also supported. Also, high &
low-speed connectors can be utilized for collection of varied frequency and bit rates related
11

sensor data. The lowest resolutions and sample rate offering the needful data for the app. Must be
utilized for reducing the overhead involved in data processing. Extraction of features can be done
with clean data. The 4 general features expected by the vision developers in the visual data are
given below:
 Edges: Group of points existing between 2 regions
 Ridges: Curve having a ridge point
 Corners: Feature resembling a point having a localized 2D structure
 Blobs: areas of interest
12

13

Path finding using A* algorithm
Being at the map’s bottom in the starting, the unit wishes to reach the top. Nothing in the
region being scanned by it tells that it should not be going up, so it tends to move forward. Since
it senses a problem close to the top, it tends to change the direction. After this, it’s way is near
the “U-shaped” problem which is known to follow the red-colored path. Contrarily, a much
larger area (depicted in light blue) would have been scanner by the pathfinder, however a smaller
path (blue) was found, resulting in not dispatching the unit inside the obstacle having concave
shape. However, you have the option of expanding a movement algorithm so that it can operate
near traps like presented above[3]. Either the development of concave obstacles is to be avoided
or their convex hulls are to be marked as risky.
14

Instead of waiting till the final moment when a problem is found out, Pathfinders allow
you to plan for future. An exchange is known to exist between planning along with pathfinders
as well as responding with movement algorithms. Though Planning is slower in nature but it can
offer fantastic outputs. Contrarily, movements having a fast nature can face problems. Future
planning is known to value less due to the dynamic game world[2]. So, my advice is to utilize
both of them, say, movements in the case of localized area, quickly changing as well as shorter
paths and; pathfinding in the case of bigger picture, longer paths and obstacles which are
changing slowly.
Dijkstra’s Algorithm and Best-First-Search
Dijkstra’s Algorithm is known to operate by dropping by all the vertices present in the
graph which is initiated by the starting point of the object. After this, it continuously analyses the
nearest unexamined vertex, adding up its vertices to the group of vertices which are to be
analyzed. Starting from the initial point, it is known to achieve its goal by growing in an outward
direction. Provided no one of the edges are known to have a cost in negative, Dijkstra’s
Algorithm is sure shot to find out a shortest way (path) from the initial point to the main goal. In
the successive diagram, the initial point is represented by the pink square, the goal is indicated by
the blue square and the teal areas are there to represent the areas which have been scanned by
Dijkstra’s Algorithm. Since the lightest of the teal areas are the most away from the initial point,
they are known to create the exploration’s “Frontier”.
15

The Greedy Best-First-Search algorithm is known to operate in somewhat same way
only. However, it has some approximation of the distance of a vertex from the goal. This
estimate is known as heuristic. So, this works by making the selection of that vertex which is
nearest to the goal rather than the vertex which is nearest to the initial point. Though there is no
guarantee of finding a shortest path in this algorithm still it manages to operate faster as
compared to Dijkstra’s Algorithm. The reason behind this is that it makes use of the heuristic
function which helps it find its way to the goal faster. As an instance, if the goal lies in the south
of the initial point, then the main target of the Greedy Best-First- Search algorithm will be those
paths which go in the southward direction[1]. In the successive diagram, black is known to
indicate a low heuristic value whereas yellow is known to indicate a higher heuristic value. It
clearly showcases paths can be found much quicker with the Greedy Best-First-Search when
compared with Dijkstra’s Algorithm.
The simplest case (i.e. there are no obstacles in the map and the shortest path is known to
be a straight line only) is being represented by both of these instances. Let us now take the
concave obstacles which had been discussed in the preceding section. Dijkstra’s algorithm is
known to undertake hard work but it definitely helps in finding out a shortest path:
16

Contrarily, less work is undertaken by the Greedy Best-First-Search but it does not offer
a really good path to us:
Being “Greedy” in nature Greedy Best-First-Search tends to run towards the goal even if
the path is wrong. As it only takes into consideration the cost involved in reaching the goal (and
not the cost of the entire path), it continues its movement even when the path turns lengthy.
How would a combo of both sound? A* was created in year 1968 for the amalgamation of
formal and heuristic approaches. It’s a bit surprising that heuristic approaches operate to offer an
estimated way of problem solving without any guarantee of offering the best possible solution.
However, A* has been developed above heuristic. So, even though there is no guarantee
involved in heuristic, a shortest path is guaranteed by A*.
17

Now I will concentrate on A* algorithm. Due to its flexible nature as well as wide scope,
A* is the most preferred option.
In the case which has concave obstacle, A* is known to find a good path similar to the
Dijkstra’s Algorithm:
18

Combining the info segments being used by both Dijkstra’s and Greedy Best-First-Search
algorithm is its success secret. In the typical terminology being utilized in reference to A*, the
exact cost of the path starting from the initial point to any specific vertex is indicated by g(n).
Similarly, the heuristic approximated cost starting from vertex n to the goal is indicated by h(n).
In the above shown diagrams, the vertices which are far from the main goal are indicated by
yellow (h) whereas the vertices which are far from the initial point are indicated by teal (g). A*
creates a balance between both by moving from the initial point to the goal. Every single time it
uses the main loop to analyze the vertex which is known to have the least f(n) = g(n) + h(n).
The remaining project part deals with the exploration of heuristic design, representation
of map, execution and other sections which are crucial to pathfinding in regards to games. While
sections are properly developed, others are still not complete.
19

Conclusion
Numerous considerations are involved when an object is to move from one location to
another freely: mechanism for object carrying, path planning strategy, mobile robot, integrating
all the inclusive systems. The robotic arm can achieve completion in this chapter. To implement
this sort of project some considerations are:
 While implementing the route, the ultrasonic sensor values must be validated.
 Implementation of the route in 2 stages: Reaching the box and placing in pallet
 Having a viewpoint to enhance the precision of the navigation, which consists of
including a video camera to track the process of palletization.
 Connection of the sensors with the IoT platform will help in streamlining the process.
For limiting the access to the danger zone, engineering controls (like fixed barriers, light
curtains, pressure mats and others) are utilized. Though these devices cut off the chances of
danger exposure, they don’t tend to lower down the possible injury severity. The least liked
prevention measures are those of administrative controls (like signs, visual warning lights,
audible alarms, trainings and procedures) because of their heavy reliance on human beings. As
20

per OSHA, proper safety training must be given to those workers who program, run, maintain as
well as undertake repair or robots/ robot systems. Also, they must be enabled to show their
proficiency in respect to job performance in a safe manner. Since enough protection is not
offered by practice or administrative controls, so the provision as well as use of Personal
Protective Equipment (like gloves, hearing protections, safety glasses, respirators and others)
must be properly monitored by the employers.
References
[1]C. Huang, Z. Huang, J. Hu, Z. Wu and S. Wang, "A MDE-Based Approach to the Safety
Verification of Extended SysML Activity Diagram", Journal of Software, vol. 10, no. 1, pp.
56-70, 2015. Available: 10.17706/jsw.10.1.56-70.
[2]Y. Xu and L. Wu, "An Automatic Test Case Generation Method based on SysML Activity
Diagram", IOP Conference Series: Materials Science and Engineering, vol. 563, p. 052075,
2019. Available: 10.1088/1757-899x/563/5/052075.
[3]F. Vicentini, "Terminology in safety of collaborative robotics", Robotics and Computer-
Integrated Manufacturing, vol. 63, p. 101921, 2020. Available:
10.1016/j.rcim.2019.101921.
[4]S. Papatheodorou, A. Tzes and Y. Stergiopoulos, "Collaborative visual area
coverage", Robotics and Autonomous Systems, vol. 92, pp. 126-138, 2017. Available:
10.1016/j.robot.2017.03.005.
[5]R. Jarvis and A. Zelinsky, Robotics research. Berlin: Springer, 2003.
[6]Y. Bai, Z. Chen, H. Bin and J. Hu, "Collaborative design in product development based on
product layout model", Robotics and Computer-Integrated Manufacturing, vol. 21, no. 1,
pp. 55-65, 2005. Available: 10.1016/j.rcim.2004.05.005.
[7]L. Bretzner, I. Laptev and T. Lindeberg, "Hand gesture recognition using multi-scale colour
features, hierarchical models and particle filtering", Proceedings of Fifth IEEE International
21

Conference on Automatic Face Gesture Recognition. Available: 10.1109/afgr.2002.1004190
[Accessed 8 January 2020].
[8]Junqiu Wang and Y. Yagi, "Adaptive Mean-Shift Tracking With Auxiliary Particles", IEEE
Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 39, no. 6, pp.
1578-1589, 2009. Available: 10.1109/tsmcb.2009.2021482 [Accessed 8 January 2020].
[9]J. NING, L. ZHANG, D. ZHANG and C. WU, "ROBUST OBJECT TRACKING USING
JOINT COLOR-TEXTURE HISTOGRAM", International Journal of Pattern Recognition
and Artificial Intelligence, vol. 23, no. 07, pp. 1245-1263, 2009. Available:
10.1142/s0218001409007624 [Accessed 8 January 2020].
[10]K. Zhang, L. Zhang, M. Yang and Q. Hu, "Robust Object Tracking Via Active Feature
Selection", IEEE Transactions on Circuits and Systems for Video Technology, vol. 23, no.
11, pp. 1957-1967, 2013. Available: 10.1109/tcsvt.2013.2269772 [Accessed 8 January
2020].
[11]L. Selvaraj and B. Ganesan, "Enhancing Speech Recognition Using Improved Particle
Swarm Optimization Based Hidden Markov Model", The Scientific World Journal, vol.
2014, pp. 1-10, 2014. Available: 10.1155/2014/270576 [Accessed 8 January 2020].
[12]J. Zhang, S. Li and C. Kuo, "Compressed-Domain Video Retargeting", IEEE Transactions
on Image Processing, vol. 23, no. 2, pp. 797-809, 2014. Available:
10.1109/tip.2013.2294541 [Accessed 8 January 2020].
22