C#Unity在Unity中从相机中提取像素阵列并在python中显示

Question

我试图做两件事：

1. 从相机中检索84 x 84 x 3像素阵列
2. 将此数组发送到python脚本，在那里可以显示它

我被1）卡住了，所以目前我只是将bytearray写入一个txt文件，然后用python打开该文件。然而，我不断得到这张图片，这让我相信bytearray实际上并没有用像素数据正确实例化。 [] 1

以下是C#unity代码：

using System.IO;
using UnityEngine;
using MLAgents.Sensors;
// using UnityEngine.CoreModule;

public class TrainingAgent : Agent, IPrefab
{

    public Camera cam;
    private RenderTexture renderTexture;
    public int bytesPerPixel;
    private byte[] rawByteData;
    private Texture2D texture2D;
    private Rect rect;


    public override void Initialize()
    {

        renderTexture = new RenderTexture(84, 84, 24);
        rawByteData = new byte[84 * 84 * bytesPerPixel];
        texture2D = new Texture2D(84, 84, TextureFormat.RGB24, false);
        rect = new Rect(0, 0, 84, 84);
        cam.targetTexture = renderTexture;
    }

    public override void CollectObservations(VectorSensor sensor)
    {
        run_cmd();
    }

    private void run_cmd()
    {

        cam.Render();
        // Read pixels to texture
        RenderTexture.active = renderTexture;
        texture2D.ReadPixels(rect, 0, 0);
        Array.Copy(texture2D.GetRawTextureData(), rawByteData, rawByteData.Length);
        File.WriteAllBytes("/Path/to/byte/array/Foo.txt", rawByteData); // Requires System.IO
    }
}

下面是python代码：

from PIL import Image
import numpy as np
fh = open('/Path/to/byte/array/foo.txt', 'rb')
ba = bytearray(fh.read())
data = np.array(list(ba)).reshape(84,84,3)
img = Image.fromarray(data, 'RGB')
img.show()

任何帮助都将不胜感激，因为我不知道哪里出了问题，我的调试尝试似乎是徒劳的

Answer 1

我不确定（不知道python的细节），但我不认为^{}是您在这里想要使用的

您更愿意使用^{}（前^{}）导出实际的图像二进制文件，例如JPG

Encodes this texture into JPG format.

The returned byte array is the JPG "file". You can write them to disk to get the JPG file, send them over the network, etc.

This function works only on uncompressed, non-HDR texture formats. You must enable the texture's Read/Write Enabled flag in the Texture Import Settings. The encoded JPG data will have no alpha channel.

然后将其作为实际图像文件加载到python中

在您的代码中，这相当于将最后两行C#替换为：

rawByteData = ImageConversion.EncodeToJPG(texture2D);
File.WriteAllBytes("/Path/to/jpg/foo.jpg", rawByteData);