Linux在bash/python脚本中将下一个“n”个文件从一个文件夹复制到另一个文件夹

Question

我们有17k个文件，名称为file1.csv、file2.csv、file3.csv…file17000.csv。 所有这些文件都应该从一个文件夹复制到另一个文件夹。 目标是创建LinuxBash或python脚本，以每5分钟复制所有这些文件除以csv文件的“n”个数，并防止复制已经复制的文件的“n”个数

这个想法是：

copy file1.csv file2.csv file3.csv  file4.csv  file5.csv to destination_dir
sleep for 300 seconds
copy file6.csv file7.csv file8.csv file9.csv file10.csv to destination_dir
sleep for 300 seconds
...
copy file16996.csv file16997.csv file16998.csv file16999.csv file17000.csv to destination_dir

对于少量文件，我们使用下面的脚本在2个范围之间复制文件：

#!/bin/bash
source_dir='/source_dir'
target_dir='/target_dir'
echo "beginning number:$1"
echo $1
echo "finite number:$2"
echo $2
for f in $(eval ls $source_dir/file{$1..$2}.csv);
do
cp $f $target_dir
done

有人能建议如何正确地指向脚本以使用下一个“n”个csv文件吗

如有任何意见和建议，将不胜感激

Answer 1

使用bash：

max=$(for i in printf file*.csv;do echo $i;done | grep -Eo '[[:digit:]]+' | tail -1)  # Work out the maximum file number
n=5                                                                                   # Set the batch number of files to copy in one go
for ((i=1;i<=max;i=i+$n));                                                            # Loop from one to max file in batches of n
do 
  sleep 300
  p=$(($i+($n-1)));                                                                   # Set the upper limit for batch file copying
  for ((k=i;i<=p;k++));
  do
     cp "file$k.csv" destination_dir                                                  # Copy files using lower and upper limits of files for this pass
  done
done

Answer 2

一个bash版本，只需根据需要添加apt batch_size变量即可

#!/bin/bash
source_dir='/source_dir'
target_dir='/target_dir'

all_csv_files=`ls -1v $source_dir/file*.csv`
batch_size=5
sleep_break=300

file_counter=0
echo Found ${#all_csv_files[@]} files

for f in "${all_csv_files[@]}"
do
    cp $f $target_dir
    let file_counter++
    if [ $file_counter == $batch_size ] 
    then
        echo Take a break `date`
        file_counter=0
        sleep $sleep_break
    fi
done

echo Done