如何使用正则表达式或索引从非日期列派生日期？

Question

我有一个数据帧和dict，如下所示

df = pd.DataFrame({
    'subject_id':[1,2,3,4,5],
    'age':[42,56,75,48,39],
    'date_visit':['1/1/2020','3/3/2200','13/11/2100','24/05/2198','30/03/2071'],
    'a11fever':['Yes','No','Yes','Yes','No'],
    'a12diagage':[36,np.nan,np.nan,40,np.nan],
    'a12diagyr':[np.nan,np.nan,2091,np.nan,np.nan],
    'a12diagyrago':[6,np.nan,9,np.nan,np.nan],
    'a20cough':['Yes','No','No','Yes','No'],
    'a21cough':[np.nan,'Yes',np.nan,np.nan,np.nan],
    'a22agetold':[37,np.nan,np.nan,46,np.nan],
    'a22yrsago':[np.nan,6,np.nan,2,np.nan],
    'a22yrtold':[np.nan,2194,np.nan,np.nan,np.nan]

 })
df['date_visit'] = pd.to_datetime(df['date_visit'])
disease_dict = {'a11fever' : 'fever', 'a20cough' : 'cough','a21cough':'cough'}

此数据框包含有关患者医疗状况和诊断日期的信息

但是正如您所看到的，诊断日期不是直接可用的并且我们必须根据包含诸如age、yr、ago、diag等关键字的列来推导它，这些关键字出现在条件列的下一个5-6列中（例如：a11fever）。在这个条件列之后查找接下来的5列，您将能够获得派生条件所需的信息日期。相似对于其他条件，如cough

我希望我的输出如下所示

我试过下面的方法，但没用

df = df[(df['a11fever'] =='Yes') | (df['a20cough'] =='Yes') | (df['a21cough'] =='Yes')]  
# we filter by `Yes` above because we only nned to get dates for people who had medical condition (`fever`,`cough`)
df.fillna(0,inplace=True)
df['diag_date'] =  df["date_visit"] - pd.DateOffset(years=df.filter('age'|'yr'|'ago'))  # doesn't help throws error. need to use regex here to select non-na values any of other columns
pd.wide_to_long(df, stubnames=['condition', 'diag_date'], i='subject_id', j='grp').sort_index(level=0)
df.melt('subject_id', value_name='valuestring').sort_values('subject_id')

请注意，我知道疾病的列名。我不知道的是实际的列名，从中我可以获得导出日期所需的信息。但我知道它包含像age、ago、yr、diag这样的关键字

diag_date是通过从date_vist列中减去derived date得到的。你知道吗

规则屏幕截图

例如：subject_id = 1因发烧于1/1/2020到医院就诊，他在36（a12diagage）或6年前（a12diagyrago）被诊断出发烧。我们知道他现在的年龄和访问日期，所以我们可以选择从任何一列中减去

如您所见，我无法找到如何基于regex选择列并减去它

Answer 1

用途：

#get of columns with Yes at least one value
mask = df[list(disease_dict.keys())].eq('Yes')
#assign mask back
df[list(disease_dict.keys())] = mask
#rename columns names by dict
df = df.rename(columns=disease_dict).max(axis=1, level=0)
#filter out False rows
df = df[mask.any(axis=1)]
#convert some columns to index for get only years and condition columns
df = df.set_index(['subject_id','age','date_visit'])

#extract columns names - removing aDD values
s = df.columns.to_series()
df.columns = s.str.extract('(yrago|yrsago)', expand=False).fillna(s.str.extract('(age|yr)', expand=False)).fillna(s)

#replace True in condition columns to column names
ill = set(disease_dict.values())
df.loc[:, ill] = np.where(df[ill].values, np.array(list(ill)), None)

#replace columns names to condition
df = df.rename(columns = dict.fromkeys(ill, 'condition'))

---

#create MultiIndex - only necessary condition columns are first per groups
cols = np.cumsum(df.columns == 'condition')
df.columns = [df.columns, cols]
#reshape by stack and convert MultiIndex to columns
df = df.stack().rename(columns={'age':'age_ill'}).reset_index().drop('level_3', axis=1)
#subtract ages
df['age_ill'] = df['age'].sub(df['age_ill'])
#priority yrago so yrago is filling missing values by age_ill
df['yrago'] = df['yrago'].fillna(df['yrsago']).fillna(df['age_ill']).fillna(0).astype(int)
df = df.drop(['yrsago','age_ill'], axis=1)

#subtract years
df['diag_date1'] =  df.apply(lambda x: x["date_visit"] - pd.DateOffset(years=x['yrago']), axis=1)
#replace years
mask1 = df['yr'].notna()
df.loc[mask1, 'diag_date'] = df[mask1].apply(lambda x: x["date_visit"].replace(year=int(x['yr'])), axis=1)
#because priority yr then fillna diag_date by diag_date1
df['diag_date'] = df['diag_date'].fillna(df['diag_date1'])

df = df.drop(['diag_date1','age','date_visit','yr','yrago'], axis=1)

---

print (df)
   subject_id condition  diag_date
0           1     fever 2014-01-01
1           1     cough 2015-01-01
2           2     cough 2194-03-03
3           3     fever 2091-11-13
4           4     fever 2190-05-24
5           4     cough 2196-05-24