查找包含图像的子文件夹

Question

获取包含文件的子文件夹路径的最有效方法是什么。例如，如果这是我的输入结构

inputFolder    
│
└───subFolder1
│   │
│   └───subfolder11
│       │   file1.jpg
│       │   file2.jpg
│       │   ...
│   
└───folder2
    │   file021.jpg
    │   file022.jpg

如果我通过getFolders(inputPath)， 它应该以包含图像的文件夹列表的形式返回输出['inputFolder/subFolder1/subFolder11','inputFolder/folder2']

目前，我正在使用我的库TreeHandler，它只是os.walk的包装器来获取所有文件

import os
from treeHandler import treeHandler
th=treeHandler()
tempImageList=th.getFiles(path,['jpg'])
### basically tempImageList will be list of path of all files with '.jpg' extension

### now is the filtering part,the line which requires optimisation.
subFolderList=list(set(list(map(lambda x:os.path.join(*x.split('/')[:-1]),tempImageList))))

我认为这样做可以更有效率

提前谢谢

Answer 1

• 拆分路径的所有部分并重新连接它们似乎会降低效率

• 查找最后一个“/”实例的索引和切片的速度要快得多

  def remove_tail(path):
      index = path.rfind('/') # returns index of last appearance of '/' or -1 if not present
      return (path[:index] if index != -1  else '.') # return . for parent directory
  .
  .
  .
  subFolderList = list(set([remove_tail(path) for path in tempImageList]))
  

• 已在AWA2数据集文件夹（50个文件夹和37322个图像）上验证

• 观察到的结果快了大约3倍
• 使用列表理解增强可读性
• 已处理父目录具有映像的情况（这将导致现有实现出错）

添加用于验证的代码

import os
from treeHandler import treeHandler
import time

def remove_tail(path):
    index = path.rfind('/')
    return (path[:index] if index != -1  else '.')

th=treeHandler()
tempImageList= th.getFiles('JPEGImages',['jpg'])
tempImageList = tempImageList
### basically tempImageList will be list of path of all files with '.jpg' extension

### now is the filtering part,the line which requires optimisation.
print(len(tempImageList))
start = time.time()
originalSubFolderList=list(set(list(map(lambda x:os.path.join(*x.split('/')[:-1]),tempImageList))))
print("Current method takes", time.time() - start)

start = time.time()
newSubFolderList = list(set([remove_tail(path) for path in tempImageList]))
print("New method takes", time.time() - start)

print("Is outputs matching: ", originalSubFolderList == newSubFolderList)

Answer 2

import os
import glob

original_path = './inputFolder/'

def get_subfolders(path):
    return [f.path for f in os.scandir(path) if f.is_dir()]

def get_files_in_subfolder(subfolder, extension):
    return glob.glob(subfolder + '/*' + extension)

files = []
subfolders = [original_path] + get_subfolders(original_path)
while len(subfolders) > 0:
    new_subfolder = subfolders.pop()
    print(new_subfolder)
    subfolders += get_subfolders(new_subfolder)
    files += get_files_in_subfolder(new_subfolder, '.jpg')