Python中对数似然函数和梯度函数的实现

Question

关于科学论文https://arxiv.org/abs/1704.04289，我正试图实施第7.3节中提到的优化超参数。特别是论文第25页的等式35

负对数似然函数似乎比通常的逻辑回归更复杂。我尝试按照下面的代码为log reg实现负对数似然和梯度下降

import numpy as np
import pandas as pd
import sklearn
import matplotlib.pyplot as plt
%matplotlib inline

#simulating data to fit logistic regression model
np.random.seed(12)
num_observations = 5000

x1 = np.random.multivariate_normal([0,0], [[1, .75],[.75, 1]], num_observations)
x2 = np.random.multivariate_normal([1,4], [[1, .75],[.75, 1]], num_observations)

simulated_features = np.vstack((x1, x2)).astype(np.float32)
simulated_labels = np.hstack((np.zeros(num_observations), np.ones(num_observations)))

plt.figure(figsize=(12,8))

plt.scatter(simulated_features[:,0], simulated_features[:,1], c = simulated_labels, alpha = .4)

#add a column of ones to deal with bias term
ones = np.ones((num_observations*2, 1))

Xb = np.hstack((ones, simulated_features))

#Activation Function

def sigmoid(scores):
    return 1 / (1 + np.exp(-scores))


#log-likelihood function

def log_likelihood(features, target, weights):
    #model output
    scores = np.dot(features,weights)
    nll = np.sum(target*scores - np.log(1 + np.exp(scores)))
    return nll

def log_reg(features, target, num_steps, learning_rate):
    weights = np.zeros(features.shape[1])
    
    for step in range(num_steps):
        score = np.dot(features, weights)
        predictions = sigmoid(scores)
        
        #update weights with gradient
        error = target - predictions
        gradient = np.dot(features.T, error)
        weights += learning_rate * gradient
        
        if step % 10000 == 0:
            print(log_likelihood(features, target, weights))
        
    return weights

我在这里面临的最大挑战是实现本文中等式中的lambda、DK、θ（DK）和θ（dyn）项。从理论上讲，我理解这个实现，并且我能够在一篇论文中手工解决它，但是我发现在使用一些模拟数据（如我的代码所示）的情况下在python上实现它很困难。有谁能指导我如何实现这一点