Build a dependable baseline
First, make the arm pick and place.
I built the workcell and a scripted controller so the gripper had to lift simulated parts through finger contact. That gave the project a working reference before adding AI.
← All projectsTell it what
Independent project · Ubuntu / ROS 2 · 2026
Tell it what
to pick up.
Camera-Guided Robot Arm Sorting
Ask for a component in everyday language. The robot looks at the camera, checks which part you mean, picks it up and places it in a tray. It checks the result before calling the task complete.
A simulated robot working with known red, green and blue parts. Built on Ubuntu and ROS 2; not yet tested on physical hardware.
The workcell in action
Three parts.
Two trays.
Select the right part, grasp it, carry it and check its destination. These recorded tasks show the same camera-guided pipeline working with different components and trays.
Red → left tray
Find the red component and place it in the left tray.
Green → right tray
Select the green component and carry it to the right tray.
Blue → pick up → left
Verify the pickup, then respond to the follow-up destination.
Recorded simulation footage. Clips play independently and use sampled frames; playback is not a real-world speed measurement.
Language meets vision
A request becomes
a robot action.
Ask for the red box. The vision-language model selects it in the camera image, then the controller picks it up. Fresh full-camera colour and depth observations check that it is resting inside the left tray.
What the camera checks
The VLM selects the part from the initial image. After release, separate vision software uses three fresh colour and depth observations to check that the part fits inside the known tray boundary, rests at the expected height and stays still.
Before the pickup
The full camera view contains the parts and both trays. A numbered crop of this image goes to the VLM for target selection.
After the release
A saved placement observation shows the red part in the left tray. Green marks the known tray boundary; yellow marks the part’s footprint measured from colour and depth.What the final planned tests showed
10/10Parts placed correctly
15/15Planned checks passed
0Unexpected collisions in these tests
Ten tests asked the robot to place a part. Five more checked whether it asked for clarification, refused a moved target or stopped while holding a part. All 15 passed in the final planned test set. An earlier set passed 14 of 15 and helped identify a problem with waiting for camera feedback. These are small simulation tests, not a promise of performance on any object.
How the project developed
Start with a working robot.
Then give it eyes and words.
Learn from an approach that failed
Copying movements was not enough.
A model trained to imitate demonstrations struggled to complete tasks. I kept those failed experiments and changed direction: a learned model now helps calculate robot movement, while feedback checks the arm’s progress.
Connect language to what the camera sees
Ask for a part. Check the answer.
A vision-language model sees the camera image and request together. Separate checks verify its proposed target, and depth sensing estimates where to grasp. Missing or ambiguous parts lead to clarification.
Handle the moments that go wrong
Success needs more than reaching the tray.
The software checks the grasp and placement, allows one retry after a missed grasp, and stops when the target moves or feedback fails. Testing exposed waiting and timing problems that shaped the final controller.
What it is ready to show
A parts-sorting
simulation prototype.
The project connects an everyday request to a visible, checked robot action. It demonstrates the engineering behind an AI-enabled workcell, from a conventional baseline to camera-guided manipulation.
What still needs work
The scene has three known parts, two fixed trays and a calibrated camera. Arbitrary objects, clutter and a physical robot need further development and testing. The simulation runs slower than real time, and its Stop button is not a hardware emergency stop.
My contribution
I developed the workcell, control, perception, learning pipeline, ROS interfaces and evaluation with AI-assisted coding. The Panda robot assets come from Google DeepMind’s MuJoCo Menagerie; Qwen provides the pretrained vision-language model.
A closer look at the engineering Vision, learning and robot control ↓
From words to a checked target
Qwen3-VL sees the request and a numbered camera image. Independent colour and geometry checks validate its proposed target, and calibrated RGB-D sensing locates the grasp. An unclear or missing part leads to clarification rather than an arbitrary pick.
From a target to movement
A PyTorch model learns the robot’s kinematics from 60,000 synthetic samples. Conventional feedback uses its predicted Jacobian to calculate joint movement. The task sequence and gripper control remain engineered; the vision-language model does not directly drive the joints.
From movement to a result
ROS 2 connects the camera, joint feedback, controller and MuJoCo simulation on Ubuntu. Contact, progress and placement checks determine completion. The final planned simulation evaluation passed 10/10 placement cases and 15/15 checks overall, including clarification and stopping behavior.
The scene uses known parts and calibrated simulated sensors. Continuous-operation timing still needs further qualification before broader deployment. Foundations: Python, ROS 2 Humble, MuJoCo, PyTorch, Qwen3-VL and llama.cpp.