遍历列表并比较值

Question

我正在尝试遍历一列日期列表，如果第二个日期比第一个日期长10分钟或更长，则表示“1”或“0”；如果第三个日期比第二个日期长10分钟或更长，则表示“1”或“0”等

我很抱歉，如果这个问题已经得到回答，我似乎找不到任何帮助

列表大小都不一样。有人知道我该怎么做吗

df = df_data_collective.groupBy("customer_id").agg(
    F.expr("collect_list(start_dt)").alias("start_times")
)

这将输出客户id和日期时间列表，如下所示

['2020-04-02T08:15:50+01:00', '2020-04-02T08:15:53+01:00', '2020-04-02T08:15:56+01:00', '2020-04-02T08:16:01+01:00', '2020-04-02T08:16:07+01:00', '2020-04-02T08:21:05+01:00', '2020-04-02T08:21:17+01:00', '2020-04-02T08:21:30+01:00', '2020-04-02T08:21:43+01:00', '2020-04-02T08:21:49+01:00', '2020-04-02T08:22:11+01:00', '2020-04-02T08:22:16+01:00', '2020-04-02T08:24:02+01:00', '2020-04-02T08:24:09+01:00', '2020-04-02T08:24:37+01:00', '2020-04-02T08:36:26+01:00', '2020-04-02T08:39:25+01:00', '2020-04-02T08:39:41+01:00', '2020-04-02T08:39:52+01:00', '2020-04-02T08:40:18+01:00', '2020-04-02T08:40:27+01:00', '2020-04-02T08:40:33+01:00', '2020-04-02T08:40:49+01:00', '2020-04-02T08:41:03+01:00', '2020-04-02T08:41:29+01:00', '2020-04-02T08:42:00+01:00', '2020-04-02T08:42:23+01:00', '2020-04-02T08:42:57+01:00', '2020-04-02T08:44:43+01:00', '2020-04-02T08:44:49+01:00']

我对for循环有非常基本的了解，但仍在培训中，希望看看是否有人能提供任何建议

Answer 1

from datetime import datetime, timedelta

dt_str_list = ['2020-04-02T08:15:50+01:00', '2020-04-02T08:15:53+01:00',
               '2020-04-02T08:15:56+01:00', '2020-04-02T08:16:01+01:00',
               '2020-04-02T08:16:07+01:00', '2020-04-02T08:21:05+01:00',
               '2020-04-02T08:21:17+01:00', '2020-04-02T08:21:30+01:00',
               '2020-04-02T08:21:43+01:00', '2020-04-02T08:21:49+01:00',
               '2020-04-02T08:22:11+01:00', '2020-04-02T08:22:16+01:00',
               '2020-04-02T08:24:02+01:00', '2020-04-02T08:24:09+01:00',
               '2020-04-02T08:24:37+01:00', '2020-04-02T08:36:26+01:00',
               '2020-04-02T08:39:25+01:00', '2020-04-02T08:39:41+01:00',
               '2020-04-02T08:39:52+01:00', '2020-04-02T08:40:18+01:00',
               '2020-04-02T08:40:27+01:00', '2020-04-02T08:40:33+01:00',
               '2020-04-02T08:40:49+01:00', '2020-04-02T08:41:03+01:00',
               '2020-04-02T08:41:29+01:00', '2020-04-02T08:42:00+01:00',
               '2020-04-02T08:42:23+01:00', '2020-04-02T08:42:57+01:00',
               '2020-04-02T08:44:43+01:00', '2020-04-02T08:44:49+01:00']


dt_list = [datetime.strptime(dt_str, '%Y-%m-%dT%H:%M:%S%z')
           for dt_str in dt_str_list]

minute_10 = timedelta(minutes=10)
flags = [1 if dt_list[i] - dt_list[i-1] > minute_10 else 0
         for i in range(1, len(dt_list))]

Answer 2

您可以使用str.split()方法：

from datetime import datetime, timedelta

lst = ['2020-04-02T08:15:50+01:00', '2020-04-02T08:15:53+01:00', '2020-04-02T08:15:56+01:00', '2020-04-02T08:16:01+01:00', '2020-04-02T08:16:07+01:00', '2020-04-02T08:21:05+01:00', '2020-04-02T08:21:17+01:00', '2020-04-02T08:21:30+01:00', '2020-04-02T08:21:43+01:00', '2020-04-02T08:21:49+01:00', '2020-04-02T08:22:11+01:00', '2020-04-02T08:22:16+01:00', '2020-04-02T08:24:02+01:00', '2020-04-02T08:24:09+01:00', '2020-04-02T08:24:37+01:00', '2020-04-02T08:36:26+01:00', '2020-04-02T08:39:25+01:00', '2020-04-02T08:39:41+01:00', '2020-04-02T08:39:52+01:00', '2020-04-02T08:40:18+01:00', '2020-04-02T08:40:27+01:00', '2020-04-02T08:40:33+01:00', '2020-04-02T08:40:49+01:00', '2020-04-02T08:41:03+01:00', '2020-04-02T08:41:29+01:00', '2020-04-02T08:42:00+01:00', '2020-04-02T08:42:23+01:00', '2020-04-02T08:42:57+01:00', '2020-04-02T08:44:43+01:00', '2020-04-02T08:44:49+01:00']

def s(d):
    h,m,s = d.split(':',2)
    h = int(h[-2:])*60*60
    m = int(m)*60
    s = int(s[:2])
    return h+m+s

c = [1 if s(lst[i-1])-s(d) >= 600 and i else 0 for i,d in enumerate(lst)]

print(c)

输出：

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0]

Answer 3

首先，必须将start_dt转换为timestamp格式，然后在收集列表后，我们可以使用transform(with index as i)函数和unix_timestamp获得所需的输出。（转换从spark2.4开始提供）

from pyspark.sql import functions as F

df.show() #sample dataframe

#+     -+            -+
#|customer_id|start_dt                 |
#+     -+            -+
#|1          |2020-04-02T08:15:50+01:00|
#|1          |2020-04-02T08:15:53+01:00|
#|1          |2020-04-02T08:15:56+01:00|
#|1          |2020-04-02T08:16:01+01:00|
#|1          |2020-04-02T08:16:07+01:00|
#|1          |2020-04-02T08:21:05+01:00|
#|1          |2020-04-02T08:21:17+01:00|
#|1          |2020-04-02T08:21:30+01:00|
#|1          |2020-04-02T08:21:43+01:00|
#|1          |2020-04-02T08:21:49+01:00|
#+     -+            -+
only showing top 10 rows


df.withColumn("start_dt", F.to_timestamp('start_dt',"yyyy-MM-dd'T'HH:mm:ss'+'SS:SS"))\
  .groupBy("customer_id").agg(F.collect_list('start_dt').alias('start_times'))\
  .withColumn("start_times", F.expr("""transform(start_times,(x,i)-> IF(i>0 and (unix_timestamp(x)-\
                                                                     unix_timestamp(start_times[i-1])>=600),1,0))"""))\
  .show(truncate=False)

#+     -+                                             +
#|customer_id|start_times                                                                               |
#+     -+                                             +
#|1          |[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]|
#+     -+                                             +