用Python改进AR应用中两帧视频之间的单应性估计

Question

我应该使用OpenCV库进行帧到帧的比较来改进Phyton中的AR应用程序。我们必须在一本书的封面上投射一个图像，而这个图像必须在现有的视频中被检测到

其思想是在两个连续帧之间采用单应性，以保持第一帧和当前帧之间的单应性的更新，从而投影AR层。 我在单应性的正确估计中发现了问题。它似乎在单应性的每次更新时都会收集错误，可能是由于矩阵的乘法，每次帧比较都会重复一次。输出视频的结果是AR层的位置越来越差

如何解决使用frame2frame方法的问题

以下是代码的相关部分：

[...]

#################################

img_array = []
success = True
success,img_trainCOLOUR = vid.read()
kp_query=kp_ref
des_query=des_ref

#get shapes of images
h,w = img_ref.shape[:2]
h_t, w_t = img_trainCOLOUR.shape[:2]
M_mask = np.identity(3, dtype='float64')
M_prev=M_ar

#performing iterations until last frame
while success :

    #obtain grayscale image of the current RGB frame
    img_train = cv2.cvtColor(img_trainCOLOUR, cv2.COLOR_BGR2GRAY)

    # Implementing the object detection pipeline
    # F2F method: correspondences between the previous video frame and the actual frame 
    kp_train = sift.detect(img_train)
    kp_train, des_train = sift.compute(img_train, kp_train)
    
    #find matches 
    flann = cv2.FlannBasedMatcher(index_params, search_params)
    matches = flann.knnMatch(des_query,des_train,k=2)
    
    #validating matches
    good = []
    for m ,n in matches:
        if m.distance < 0.7*n.distance:
            good.append(m)

    #checking if we found the object
    MIN_MATCH_COUNT = 10
    if len(good)>MIN_MATCH_COUNT: 

        #differenciate between source points and destination points
        src_pts = np.float32([ kp_query[m.queryIdx].pt for m in good ]).reshape(-1,1,2)
        dst_pts = np.float32([ kp_train[m.trainIdx].pt for m in good ]).reshape(-1,1,2)
    
        #find homography between current and previous video frames
        M1, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC, 5.0)
        #matchesMask = mask.ravel().tolist()

        #updated homography between M_mask which contains  from first to current-1 frame homography
        # and the current frame2frame
        M_mask=np.dot(M_mask,M1)
        #updated homography between M_prev which contains img_ar layer and first frame homography,
        # and from first to current-1 frame homography and the current frame2frame
        M = np.dot(M1, M_prev)
        
        #warping the img_ar (transformed as the first frame)
        warped = cv2.warpPerspective(img_arCOLOUR, M, (w_t, h_t), flags= cv2.INTER_LINEAR)
        warp_mask = cv2.warpPerspective(img_armask, M, (w_t, h_t), flags= cv2.INTER_LINEAR)
        
        #restore previous values of the train images where the mask is black
        warp_mask = np.equal(warp_mask, 0)
        warped[warp_mask] = img_trainCOLOUR[warp_mask]
         
        #inserting the frames into the frame array in order to reconstruct video sequence
        img_array.append(warped)
                   
        #save current homography for the successive iteration
        M_prev = M
        #save the current frame for the successive iteration
        img_query=img_train

        #warping the mask of the book cover as the current frame
        img_maskTrans = cv2.warpPerspective(img_mask, M_mask, (w_t, h_t), flags= cv2.INTER_NEAREST)

        #new sift object detection with the current frame and the current mask 
        # to search only the book cover into the next frame      
        kp_query=sift.detect(img_query,img_maskTrans)
        kp_query, des_query = sift.compute(img_query, kp_query)

        #reading next frame for the successive iteration
        success,img_trainCOLOUR = vid.read()

[...]

这里有输入数据、完整代码和输出： https://drive.google.com/drive/folders/1EAI7wYVFy7SbNZs8Cet7fWEfK2usw-y1?usp=sharing

谢谢你的支持

Answer 1

您的解决方案会漂移，因为您始终与上一个图像匹配，而不是与固定的参考图像匹配。保持其中一个图像不变。此外，SIFT或任何其他基于描述符的匹配方法对于短基线跟踪来说都是多余的。您只需检测兴趣点（Shi Tomasi GoodfeatureToTrack或Harris角点）并使用Lucas Kanade进行跟踪