如何使用立体相机创建好的深度贴图？

Question

我在一个项目上学习了很长时间。我的目标是从立体相机的图像中获取深度图，并仅过滤人体，以便计算人体内部的数量

我正在尝试校准我的相机，连续校准1-2个月。然而，当我在校正对上画极线时，结果还不够好（我附上了校正对的结果）。我现在正在工作，我的校准结果非常好，并试图从视差图中获得深度图。我已经录制了一个图像序列，.avi文件，当我试图从这个视频中获取深度贴图时，当我尝试这样做时，我面临着一个不稳定的局面。在前一帧中为白色的斑点在下一帧中可能为黑色。所以我不能仅仅通过过滤差异来计算人数。我使用SGBM从校正图像中获取深度。我仍然被认为是这个项目的业余爱好者。我愿意接受任何建议。（如何进行更好的校准？更好的视差图？更好的深度图？）

这是深度图，并对其进行了校正：

整流对线和极线

我已经校准了我的相机，几乎有600双，并且改进了它。我的总体平均误差是.13像素，35对图像

minDisparity=-1,
        numDisparities=2*16,  # max_disp has to be dividable by 16 f. E. HH 192, 256
        blockSize=window_size,
        P1=8 * 3 * window_size,
        # wsize default 3; 5; 7 for SGBM reduced size image; 15 for SGBM full size image (1300px and above); 5 Works nicely
        P2=32 * 3 * window_size,
        disp12MaxDiff=12,
        uniquenessRatio=1,
        speckleWindowSize=50,
        speckleRange=32,
        preFilterCap=63,
        mode=cv2.STEREO_SGBM_MODE_SGBM_3WAY

这是我的块匹配参数

Answer 1

为什么要使用距离图来检测人类？在我看来，这是一个目标检测问题

无论如何，在目前获取距离图的技术水平下，我推荐基于人工智能的模型

像NeRF这样的模型已经取得了惊人的成果

• 谷歌提供了ARCore，它提供了一个深度图，但基于一个摄像头
• Nvidia拥有this project
• This，基于视频的Nerf实现三维重建

在几周内，我将致力于此，我想实现一个在TensorFlowLite中运行的模型，通过立体相机实现深度图

Answer 2

为了改进视差贴图的结果，可以实现后期过滤，下面是一个教程（https://docs.opencv.org/master/d3/d14/tutorial_ximgproc_disparity_filtering.html）。我还使用了一个额外的斑点过滤器和选项来填补缺失的差异。python实现如下所示：

stereoProcessor = cv2.StereoSGBM_create(
                minDisparity=0,
                numDisparities = max_disparity, # max_disp has to be dividable by 16 f. E. HH 192, 256
                blockSize=window_size,
                P1 = p1,       # 8*number_of_image_channels*SADWindowSize*SADWindowSize
                P2 = p2,    # 32*number_of_image_channels*SADWindowSize*SADWindowSize
                disp12MaxDiff=disp12Maxdiff,
                uniquenessRatio= uniquenessRatio,
                speckleWindowSize=speckle_window,
                speckleRange=speckle_range,
                preFilterCap=prefiltercap,
               # mode=cv2.STEREO_SGBM_MODE_HH# numDisparities = max_disparity, # max_disp has to be dividable by 16 f. E. HH 192, 256
                
        )
        
        #stereoProcessor = cv2.StereoBM_create(numDisparities=16, blockSize=15)
        
        # set up left to right + right to left left->right + right->left matching +
        # weighted least squares filtering (not used by default)

        left_matcher = stereoProcessor
        right_matcher = cv2.ximgproc.createRightMatcher(left_matcher)

        #Image information 
        height, width, channels = I.shape

        frameL= I[:,0:int(width/2),:]
        frameR = I[:,int(width/2):width,:]

        # remember to convert to grayscale (as the disparity matching works on grayscale)

        grayL = cv2.cvtColor(frameL,cv2.COLOR_BGR2GRAY)
        grayR = cv2.cvtColor(frameR,cv2.COLOR_BGR2GRAY)

        # perform preprocessing - raise to the power, as this subjectively appears
        # to improve subsequent disparity calculation

        grayL = np.power(grayL, 0.75).astype('uint8')
        grayR = np.power(grayR, 0.75).astype('uint8')

        # compute disparity image from undistorted and rectified versions
        # (which for reasons best known to the OpenCV developers is returned scaled by 16)

        if (wls_filter):

            wls_filter = cv2.ximgproc.createDisparityWLSFilter(matcher_left=left_matcher)
            wls_filter.setLambda(wls_lambda)
            wls_filter.setSigmaColor(wls_sigma)
            displ = left_matcher.compute(cv2.UMat(grayL),cv2.UMat(grayR))  # .astype(np.float32)/16
            dispr = right_matcher.compute(cv2.UMat(grayR),cv2.UMat(grayL))  # .astype(np.float32)/16
            displ = np.int16(cv2.UMat.get(displ))
            dispr = np.int16(cv2.UMat.get(dispr))
            disparity = wls_filter.filter(displ, grayL, None, dispr)
        else:

            disparity_UMat = stereoProcessor.compute(cv2.UMat(grayL),cv2.UMat(grayR))
            disparity = cv2.UMat.get(disparity_UMat)
        
        speckleSize = math.floor((width * height) * 0.0005)
        maxSpeckleDiff = (8 * 16) # 128

        cv2.filterSpeckles(disparity, 0, speckleSize, maxSpeckleDiff)
        
        # scale the disparity to 8-bit for viewing
        # divide by 16 and convert to 8-bit image (then range of values should
        # be 0 -> max_disparity) but in fact is (-1 -> max_disparity - 1)
        # so we fix this also using a initial threshold between 0 and max_disparity
        # as disparity=-1 means no disparity available

        _, disparity = cv2.threshold(disparity,0, max_disparity * 16, cv2.THRESH_TOZERO)
        disparity_scaled = (disparity / 16.).astype(np.uint8)

        # fill disparity if requested

        if (fill_missing_disparity):

            _, mask = cv2.threshold(disparity_scaled,0, 1, cv2.THRESH_BINARY_INV)
            mask[:,0:120] = 0
            disparity_scaled = cv2.inpaint(disparity_scaled, mask, 2, cv2.INPAINT_NS)

        # display disparity - which ** for display purposes only ** we re-scale to 0 ->255
        disparity_to_display = (disparity_scaled * (256. / self.value_NumDisp)).astype(np.uint8)
        

Answer 3

快速浏览以下内容：

• GBM中的P1、P2参数应为平方，计算如下：

  P1 = 8*3*blockSize**2
  P2 = 32*3*blockSize**2
  

• 立体声GBM支持彩色图像，请尝试跳过灰度转换。如果使用灰度，则应删除P1、P2参数中的*3乘数。这是图像通道数，其中灰度为1

• 您使用的是cv2.STEREO_SGBM_MODE_3WAY，它速度更快，但精确度较低。为了获得更好的结果但速度较慢，请尝试使用cv2.STEREO_SGBM_MODE_SGBM（默认为5个邻居）或cv2.STEREO_SGBM_MODE_HH（8个邻居）

• 您的图像具有不同的曝光，如果可能，请尝试固定相机的AWB/增益，以便一致地捕获图像