All Posts – Siddharth Jha

My summer at Pittsburgh

Posted on August 15, 2017 by Siddharth JhaLeave a comment

I spent my summer working at the Robust Adaptive Systems Lab at Carnegie Mellon University’s Robotics Institute. My work was centered around visual inertial state estimation for a quadcopter, focussed on loop closures and robustly calculating the transforms between camera images. I was able to do justice to my project, and am currently working remotely for developing a better version of the framework.

Robotics Institute was exactly as anticipated, with lots of like minded people working days and nights performing robotics research, something I wish I could do. I had a lot of fun working with the amazing team at RASL, and learned a way lot more than expected. I read papers everyday, maintained my documentation, wrote codes of sizes I’d never done before and produced results! This was the life I always wanted.

Apart from the work, I also had a lot of fun exploring Pittsburgh and around. We went for 2 trips, one covering Washington DC and Philadelphia and the second one covering NYC (just to watch the 4th of July celebrations). Pittsburgh had it’s share of attractions as well and I had fun exploring the Aquarium + Zoo, Mt. Washington, Point State Park, Schenley Park, Cathedral of Learning and maybe a couple of more places. The people of the ‘Burgh were very welcoming and I had tremendous amusement talking to people from other cultures. And yeah, I got lost in the city as well with no connectivity in hand. That was a unique tale, never to be forgotten.

Sick panorama from Mount Washington

The hangout place-to-be was the RI lounge. Coffee, PS4, Fußball, Pizza. That was the life. Wonderful times were spent there with Manash beside me. I found great friends in my roommates at Shadyside as well. Exploring food joints, watching Football + Cricket + Tennis, grocery shopping, discussing stuff, cards! It wouldn’t have been great had Nikhil and Nikunj not been there!

I saved the best for the last, the concerts! Tool! Megadeth! Meshuggah! Tesseract! What a metal summer this turned out to be! I got a chance to see my beloved bands play live and had great music discussions with people around me. Dave Mustaine was literally 3m away from me <3. What an experience!

I will miss CMU tremendously. And Oakland, Shadyside, Squirrel Hill, East Liberty, Downtown, Waterfront, Lawrenceville… And the diners. And the amazing people of the ‘Burgh. Thank you everyone for a memorable 10 weeks of my life. Hope I make it there again.

An update blog post

Posted on March 21, 2017 by Siddharth JhaLeave a comment

It’s been a long while since I posted on this blog, and consequently February ’17 ended up being the first month since Nov ’15 (when I first started writing on this blog) in which I did not post. All those clichéd combo breaker thoughts aside, it was an uphill journey for me for the most part of early 2017. And proud as I may be, I might have endured just as well.

I’d cut to the chase, I have been selected as a research assistant at the Robust Adaptive Systems Lab, Robotics Institute, Carnegie Mellon University for the period May 10, 2017 to July 25, 2017. I’d be reporting to Prof. Nathan Michael, and my work would revolve around the “refinement, optimization, and evaluation of an online monocular visual-inertial state estimation methodology that leverages a window-based optimization to compute a maximum posterior estimate of the system state.” Simply speaking, I’d have to implement a better online Visual Inertial Odometry framework on an ARM PC mounted on a quadcopter.

Why I call this selection an uphill task is because of the sheer time and effort it required. Since early December 2016, I’ve had about 10 voice calls with people at RASL (including 5 with Prof. Nathan), read over 10 research papers related to the field, compiled the entire code on my PC and added a few bug-fixes to the same. Add to that the agony of getting hospitalized for a week in mid February (it was an infection of some sort). I finally got the confirmation on March 5 and that has waived a humongous amount of stress off my head.

Needless to say, it’s a dream position for me. The sheer number of times I’ve dreamt about being at RI is uncountable, and with a project that totally matches my field of interest, that’s like cake with icing on it. And since I loathe overtly sweet stuff, much more like hot sauce on my noodles. I currently am busy planning out the logistics for my trip, and trying to rush through the visa procedure. I depart from IGI on May 09.

Here’s hoping for a great productive summers! I’d aspire to turn my PhD at CMU dreams into a reality with this!

Working with awesome sensors at AGV

Posted on January 11, 2017 by Siddharth JhaLeave a comment

I haven’t been posting for a while, with the university registrations and stabilizing into the monotony of the life taking some time. I have resumed work with AGV, now working on sensors more commonly found on real world cars due to the ongoing Rise Prize project. As a result, I now am working primarily with the Carnegie Multisense S7 stereo camera. I’d try to implement robust bumper detection and resulting creep and crawl control soon and am also working on stereo visual odometry.

Coloured PointCloud from Multisense

Also a few new sensors have arrived in the lab, like the shiny new SICK LMS511 LiDaR and the Novatel GPS sensor. I had fun interfacing these and testing out the raw data.

SICK LiDaR scan

Also, we installed and ran Autoware on the AGV PC and ran the inbuilt SLAM module using the Velodyne, and the results were impressive. With the new Husky UGV about to arrive and the work on the DuneBuggy almost ready for test, I anticipate awesomeness ahead!

Visual Homing Project: Methodology and Results

Posted on December 29, 2016 by Siddharth Jha1 Comment

The visual homing project at IIT Bombay was a success and we were able to achieve homing on a quadcopter. The road to this point wasn’t smooth though. It was the third complete rewrite of the algorithm that finally worked. I earlier had written about the panoramic stitching algorithm in which the positions in the room were calculated on the basis of the corresponding matched feature position in a stitched image. The problem was, there were many parts of the room exactly similar to each other.

After that I attached 16 different feature points in the entire lab for the robot to identify and use as markers so that repetitions in the room may be avoided and then moved on to the panorama approach, but soon realized that finding keypoints would be a more robust approach.

So I wrote down a C++ code to detect labels in the rooms, using blob detection and colour segmentation in HSV colour space. The overall code was working okay and with a few tweaks here and there, I was able to detect labels pretty well. Although there were (and still are) some issues with lighting unsolved.

On top of that was the planner code, which was verified using MATLAB and just had to be converted to C++. Also there had to be a manual override cum teleoperation code that had to run in case the algorithm didn’t perform as expected. I wrote the former in C++ and latter in Python (the goodness of ROS 😀 ).

Image divided into regions of interest by Green and Red lines, Blue lines denote labels

My initial approach for detecting 3 labels per image was to segment the image into sections having enough keypoints and then applying the image processing over again. It was not a great approach and I eventually shifted to processing the keypoints already obtained, which turned out to be better. Also, initially I was using the rotZ readings of the ARDrone v2, but they tend to drift and make the algorithm worthless. So I shifted to noisier Magnetometer readings which did not drift.

Finally, after a lot of testing and change of approaches, the last one worked out pretty well and here’s a screenshot of it running and giving command velocity commands.

Best approach, directly using the already obtained keypoints

As usual, all the code is uploaded on a Github repository, which is now public. I would also upload a video of it running on the quadcopter soon. I learned a lot in the process and my first experience with aerial robotics was rather good. I hope to turn this initiation into a passion soon.

Visual Homing Project: Homing algorithm simulation and convergence test.

Posted on December 25, 2016 by Siddharth Jha1 Comment

As a part of the homing project, I’ve been working on a paper by Liu et. al. concerning a bearing only method for visual homing. I have now implemented everything on ROS and am now in the testing stages for the implementation on a quadcopter. But before all of that, I had to test for convergence of the algorithm, because the paper is not the most lucid one you’d find.

So I have written a small piece of code on MATLAB that when run, produces the proof of convergence of the algorithm.

Here are the code(s):

Algo.m

clc;
clear;
close all;
 
global u;
 
 
Tsamp = 1;
 
 
X0 = [60 30]';
 
G = [12 -42]';
u = [0 0]';
 
L1 = [150 30]';
L2 = [-100 -180]';
L3 = [-170 -60]';
 
theta_1 = newTan(L1(2)- G(2), L1(1)-G(1));
theta_2 = newTan(L2(2)-G(2), L2(1)-G(1));
theta_3 = newTan(L3(2)-G(2), L3(1)-G(1));
 
beta_21 = theta_2 - theta_1;
beta_21s = newTan(sin(beta_21),cos(beta_21));
 
beta_32 = theta_3 - theta_2;
beta_32s = newTan(sin(beta_32),cos(beta_32));
 
beta_13 = theta_1 - theta_3;
beta_13s = newTan(sin(beta_13),cos(beta_13));
 
path = zeros(2,1);
path(:,1) = X0;
 
for k=1:1000
    [t,X] = ode45('holModel',Tsamp, path(:,end));
    theta_1 = newTan(L1(2)- X(end,2), L1(1)- X(end,1));
    theta_2 = newTan(L2(2)- X(end,2), L2(1)- X(end,1));
    theta_3 = newTan(L3(2)- X(end,2), L3(1)- X(end,1));
 
    beta_21 = theta_2 - theta_1;
    beta_21 = newTan(sin(beta_21),cos(beta_21));
 
    beta_32 = theta_3 - theta_2;
    beta_32 = newTan(sin(beta_32),cos(beta_32));
 
    beta_13 = theta_1 - theta_3;
    beta_13 = newTan(sin(beta_13),cos(beta_13));
 
    OP1 = L1 - X(end,:)';
    OP1 = OP1./norm(OP1);
    OP2 = L2 - X(end,:)';
    OP2 = OP2./norm(OP2);
    OP3 = L3 - X(end,:)';
    OP3 = OP3./norm(OP3);
 
    M = [cos(beta_13/2)*(beta_13 - beta_13s) 0 cos(beta_13/2)*(beta_13 - beta_13s);
        cos(beta_21/2)*(beta_21 - beta_21s) cos(beta_21/2)*(beta_21 - beta_21s) 0;
        0 cos(beta_32/2)*(beta_32 - beta_32s) cos(beta_32/2)*(beta_32 - beta_32s)];
    N = [OP1 OP2 OP3]';
 
    V = 2.*M*N;
 
    u = -(V(1,:) + V(2,:) +V(3,:))';
 
    path = [path X(end,:)'];
 
    if(norm(path(:,end) - G)&lt;.1)
        break;
    end
 
end
 
figure(1); plot(path(1,:),path(2,:)); hold on;

newTan.m

function [ phi ] = newTan( sinTheta, cosTheta )
%UNTITLED Summary of this function goes here
%   Detailed explanation goes here
 
 phi = atan2(sinTheta, cosTheta) + (atan2(sinTheta, cosTheta) &lt; 0)*2*pi;
 
end

holModel.m

function dydt = holModel(t,X0)
 
global u;
 
dydt =  u;
 
end

Which upon running produce the proof of convergence of the algorithm. Such as this, where start point is (60,30) and end point is (-12,-42)

Visual Homing Project: A video panorama stitching method

Posted on December 24, 2016 by Siddharth Jha1 Comment

I’ve been working on the Visual Homing project at IITB for more than two weeks, and have made significant progress as well. I’ve not been documenting stuff, which is rather unusual. Although, I regularly am uploading the codes on Github. Right now it is a private repository but I’d be making it public rather soon.

This post corresponds to an earlier portion of my work, in which I had thought about stitching a 360 degre panoramic image and then developing a pixel x-coordinate versus the rotation angle, and it pretty much does work assuming constant angular velocity while taking the video. For the project purposes, this approach is no longer relevant though.

For the stitching, the simplistic approach was:

Detect features using SURF algorithm, calculate feature vectors.
Match features between consecutive images, find regions of images that are common.
Compensate for changes in vertical direction.
Add common region from the left image along with the region not present in the right one, and then append the region not present in the left image to form the stitched image.

The algorithm does not take care of any sort of colour-tone matching or the change in lighting conditions.

The entire code written in C++ and OpenCV 2.4 is:

#include "opencv2/opencv_modules.hpp"
#include "stdio.h"
#include "iostream"
#include "opencv2/core/core.hpp"
#include "opencv2/features2d/features2d.hpp"
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/nonfree/features2d.hpp"
 
#define Y_THRESHOLD 5
#define DIV_FACTOR 2.7
 
using namespace cv;
using namespace std;
 
int distx,disty;
Mat prevframe,frame;
Mat img_1,img_2;
 
int main( int argc, char** argv )
{
  int iterator=0;
  VideoCapture cap("source.mp4"); 
  std::cout&lt;&lt;cap.get(CV_CAP_PROP_FRAME_COUNT);
  for(iterator=0;iterator&lt;cap.get(CV_CAP_PROP_FRAME_COUNT);iterator++)
  {
    cap.read(frame);
    img_2=frame;
    if(img_1.empty())
    {
      prevframe=img_2;
    }
    img_1 = prevframe;
 
    int minHessian = 400;
    SurfFeatureDetector detector( minHessian );
    std::vector keypoints_1, keypoints_2;
 
    detector.detect( img_1, keypoints_1 );
    detector.detect( img_2, keypoints_2 );
 
 
    SurfDescriptorExtractor extractor;
    Mat descriptors_1, descriptors_2;
    extractor.compute( img_1, keypoints_1, descriptors_1 );
    extractor.compute( img_2, keypoints_2, descriptors_2 );
 
 
    FlannBasedMatcher matcher;
    std::vector&lt; DMatch &gt; matches;
    matcher.match( descriptors_1, descriptors_2, matches );
 
    double max_dist = 0; double min_dist = 100;
 
    for( int i = 0; i &lt; descriptors_1.rows; i++ )
    { double dist = matches[i].distance;
      if( dist &lt; min_dist ) min_dist = dist; if( dist &gt; max_dist ) max_dist = dist;
    }
 
 
    std::vector&lt; DMatch &gt; good_matches;
 
    for( int i = 0; i &lt; descriptors_1.rows; i++ )
    { if( matches[i].distance &lt;= max(2*min_dist, 0.05) )
      { good_matches.push_back( matches[i]); }
    }
 
 
 
    int less_dist=0;
    Mat img1cut,img2cut;
    for( int i = 0; i &lt; (int)good_matches.size(); i++ )
    { 
    	Point2f apt=keypoints_1[(int)good_matches[i].queryIdx].pt;
    	Point2f bpt=keypoints_2[(int)good_matches[i].trainIdx].pt;
 
 
    	distx=abs(apt.x-bpt.x);
    	disty=abs(apt.y-bpt.y);
 
    	if(disty&lt;Y_THRESHOLD) { less_dist++; } if(less_dist&gt;(int)good_matches.size()/DIV_FACTOR)
    	{
    		Rect img1roi(0,0,apt.x,img_1.rows);
    		img1cut=img_1(img1roi);
    		Rect img2roi(bpt.x,0,img_2.cols-bpt.x,img_1.rows);
    		img2cut=img_2(img2roi);
    		break; 
 
    	}
 
    }
 
    Size sz1 = img1cut.size();
    Size sz2 = img2cut.size();
 
 
    Mat result(sz1.height, sz1.width+sz2.width, CV_8UC1);
    Mat left(result, Rect(0, 0, sz1.width, sz1.height));
    img1cut.copyTo(left);
    Mat right(result, Rect(sz1.width, 0, sz2.width, sz1.height));
    img2cut.copyTo(right);
    imshow("left", left);
    imshow("right", right);
    Mat concat;
    hconcat(left,right,concat);
    imshow("result", concat);
    cout&lt;&lt;concat.cols&lt;&lt;endl;
    prevframe=concat;
 
     if(waitKey(30) == 27) 
       {
                break; 
       }
 
  }
 
}

#include "opencv2/opencv_modules.hpp" #include "stdio.h" #include "iostream" #include "opencv2/core/core.hpp" #include "opencv2/features2d/features2d.hpp" #include "opencv2/highgui/highgui.hpp" #include "opencv2/nonfree/features2d.hpp" #define Y_THRESHOLD 5 #define DIV_FACTOR 2.7 using namespace cv; using namespace std; int distx,disty; Mat prevframe,frame; Mat img_1,img_2; int main( int argc, char** argv ) { int iterator=0; VideoCapture cap("source.mp4"); std::cout<<cap.get(CV_CAP_PROP_FRAME_COUNT); for(iterator=0;iterator<cap.get(CV_CAP_PROP_FRAME_COUNT);iterator++) { cap.read(frame); img_2=frame; if(img_1.empty()) { prevframe=img_2; } img_1 = prevframe; int minHessian = 400; SurfFeatureDetector detector( minHessian ); std::vector keypoints_1, keypoints_2; detector.detect( img_1, keypoints_1 ); detector.detect( img_2, keypoints_2 ); SurfDescriptorExtractor extractor; Mat descriptors_1, descriptors_2; extractor.compute( img_1, keypoints_1, descriptors_1 ); extractor.compute( img_2, keypoints_2, descriptors_2 ); FlannBasedMatcher matcher; std::vector< DMatch > matches; matcher.match( descriptors_1, descriptors_2, matches ); double max_dist = 0; double min_dist = 100; for( int i = 0; i < descriptors_1.rows; i++ ) { double dist = matches[i].distance; if( dist < min_dist ) min_dist = dist; if( dist > max_dist ) max_dist = dist; } std::vector< DMatch > good_matches; for( int i = 0; i < descriptors_1.rows; i++ ) { if( matches[i].distance <= max(2*min_dist, 0.05) ) { good_matches.push_back( matches[i]); } } int less_dist=0; Mat img1cut,img2cut; for( int i = 0; i < (int)good_matches.size(); i++ ) { Point2f apt=keypoints_1[(int)good_matches[i].queryIdx].pt; Point2f bpt=keypoints_2[(int)good_matches[i].trainIdx].pt; distx=abs(apt.x-bpt.x); disty=abs(apt.y-bpt.y); if(disty<Y_THRESHOLD) { less_dist++; } if(less_dist>(int)good_matches.size()/DIV_FACTOR) { Rect img1roi(0,0,apt.x,img_1.rows); img1cut=img_1(img1roi); Rect img2roi(bpt.x,0,img_2.cols-bpt.x,img_1.rows); img2cut=img_2(img2roi); break; } } Size sz1 = img1cut.size(); Size sz2 = img2cut.size(); Mat result(sz1.height, sz1.width+sz2.width, CV_8UC1); Mat left(result, Rect(0, 0, sz1.width, sz1.height)); img1cut.copyTo(left); Mat right(result, Rect(sz1.width, 0, sz2.width, sz1.height)); img2cut.copyTo(right); imshow("left", left); imshow("right", right); Mat concat; hconcat(left,right,concat); imshow("result", concat); cout<<concat.cols<<endl; prevframe=concat; if(waitKey(30) == 27) { break; } } }

The constants Y_THRESHOLD and DIV_FACTOR need to be tuned with relation to how shaky the video feed is. Very low values might give segmentation faults, I haven’t done anything to correct that because as mentioned earlier, I am no longer using this approach in my project. Too high values would give unconvincing results. The minHessian value can also be changed. ”

The result is stored in a Mat concat, which may be used for further processing or may be saved using imwrite.

Here are a few results.

Input video 1: (I trimmed the video as it was too long, also the few black frames at the beginning are a pain.)

Resultant stitched image 1:

Input Video 2:

Resultant stitched image 2:

Input video 3:

Resultant stitched image 3:

The occurrence of segmentation faults when using shaky videos is still an issue. The blurriness of the image is due to the rotation in the video being too fast for the camera, but the lines that appear are the algorithm’s fault.

I hope this code may find some use to someone.

Computational Neuroscience Term Project Completed!

Posted on December 5, 2016 by Siddharth JhaLeave a comment

Today marks the end of the most demanding, yet interesting course I’ve ever taken at IITKGP. EC60007: Computational Neuroscience was challenging, frustrating but overall a really informative course (not a pun). Information theory overall is a great field in itself, and it was a sort of a primer to that as well. Taught by Prof. Sharba who is a really cool yet somewhat feared person, it was an example of how courses should be.

The final term project of the course was due 5 days after the end semesters (i.e. December 5, today) and no surprises here, despite repeated warnings from the professor, we had happily postponed our efforts. And when the actual time came, saying that hell broke loose would be an understatement.

The project in itself seemed straightforward. We had spiking data from 4 neurons for 20 seconds each and 50 total repetitions for each set. Now the plan of action was to predict a Linear-Nonlinear-Poisson model on the data, similar to machine learning, but using a much more fundamental mathematical approach. And a few extra parts.

The six parts were:

Gaussian Estimation
PSTH
Poisson Estimation
STA
Nonlinearity
Victor & Purpura Distance Metric

It was much of a war of nerves, with computation times increasing as the parts kept piling in. Add to that the travelling involved in the past few days, and here’s the recipe for ultimate frustration. But somehow, we pulled it off. I worked mostly in collab with Shailesh, and we somehow got it all figured out (or so we presume).

Here’s the Github repository of the entire course, including all the three projects. It also contains the problem statements and other helpful resources.

So after two nightouts (not in succession), hours and hours of coding and hours of waiting 😛 , here’s our final submission attached as a Google doc.

Hope all this effort helps me out, and maybe others too, someday!

Going to IIT Bombay for the winters.

Posted on November 25, 2016 by Siddharth JhaLeave a comment

After staying back in KGP for two times in a row, I’m going to IIT Bombay this winter for a project.

The project is titled “3D Homing for quadcopters using visual servoing”. I’d be working under Prof. Leena Vachhani of the Systems & Control engineering department.

I’d be working on image processing, motion planning and low level control on an UAV and would try to implement this paper over a course of one month, from December 4th to January 4th. I’d be working on the Parrot AR Drone v2.

I hope to complete my task well and add some much needed UAV experience to my robotics profile. I anticipate a fruitful month ahead! I’d keep updating my blog upon completion of tasks.

Bumper Detection ConvNet: Results

Posted on November 3, 2016 by Siddharth JhaLeave a comment

I had written in my previous post that I had written the Convolutional Neural Network to detect bumpers in a scene. The testing worked great and the ConvNet was showing an accuracy of 99.2% on the test dataset by the fifth epoch. The overall network was quite small and overall the task is something that even a simple CV code would have done. But since I’m not that experienced with ML, and wanted to do a good segmentation task, and this one fit perfectly. Anyhow, I put the predictions on the test and with a bit of fine tuning, the results are satisfactory (not awesome), and may help the group in the future.

So, here’s how our net performs.

With better tuning and maybe better fitting, maybe it would perform great. I’d update this post then, maybe.

The entire code is hosted on Github

Bumper Detection ConvNet ready for frame by frame analysis.

Posted on October 31, 2016 by Siddharth Jha1 Comment

Talk about having the quietest Diwali (the festival of lights in India), the entire evening of mine was spent on the paper(s?) I’m working on, updating my resume and searching for procurements. Meanwhile, I also took some time out to finally complete the ConvNet and the code for classifying segments of a video live stream as bumpers and background. I studied few resources where similar classification was done, and I wanted my net to be a bit smaller than AlexNet because the classification was not THAT difficult.

Sample Positive from the dataset

Sample Negative from the dataset

I finally designed a network using an awesome tool I recently discovered, named ConvNet Designer. It lets you add Convolution, Pooling, Linear, ReLU, Fully connected and Output layers using a GUI and finally rolls out a TensorFlow code for the same. Although I was specifically using Torch, the overall experience of using the tool was fantastic and saved me a lot of calculation time.

~~I’d add the screenshot of the net and~~ it’s result on a full size image by tomorrow.

View of the designed Net. The input is a 256x256x3 image (3 is the channel count)

I had some issues with CUDA versions and forwarding the image as a CudaTensor. But in the end, I managed to get a 99.2% test case accuracy on the test set by the fifth epoch. The overall net was quite fast (more than 100 tests per second) and was generalizing well.

With the control systems research, the motor still remains doubtful, and hence we have to wait for the repaired motor. Once it’s here, I guess it won’t be too long a job. I really feel I should be able to complete it by the end of this week.

The link to the dataset.

Happy Diwali. Maybe.