DoD xccdf XML文件的（Python）XML到CSV转换

Question

我开始编写一个python脚本，从DISA SCAP检查工具（SCC）获取XML输出，并将其转换为csv，这样我就可以轻松地将其接收到Splunk进行存储/分析。下面是XML的摘录

<cdf:Benchmark id="xccdf_mil.disa.stig_benchmark_Mozilla_Firefox_Windows" style="SCAP_1.2" resolved="1" xsi:schemaLocation="http://checklists.nist.gov/xccdf/1.2 http://scap.nist.gov/schema/xccdf/1.2/xccdf_1.2.xsd http://cpe.mitre.org/dictionary/2.0 http://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd http://cpe.mitre.org/language/2.0 http://scap.nist.gov/schema/cpe/2.3/cpe-language_2.3.xsd" xmlns:cdf="http://checklists.nist.gov/xccdf/1.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:dc="http://purl.org/dc/elements/1.1/">
      <cdf:status date="2020-01-24">accepted</cdf:status>
      <cdf:title>Mozilla Firefox Security Technical Implementation Guide</cdf:title>
      <cdf:description>The Mozilla Firefox Security Technical Implementation Guide (STIG) is published as a tool to improve the security of Department of Defense (DoD) information systems. Comments or proposed revisions to this document should be sent via e-mail to the following address: disa.stig_spt@mail.mil</cdf:description>
      <cdf:notice id="terms-of-use"></cdf:notice>
      <cdf:reference href="https://cyber.mil">
            <dc:publisher>DISA</dc:publisher>
            <dc:source>STIG.DOD.MIL</dc:source>
      </cdf:reference>
      <cdf:plain-text id="release-info">Release: 1.5 Benchmark Date: 24 Jan 2020</cdf:plain-text>
      <cdf:platform idref="cpe:/a:mozilla:firefox:::~~~windows~~"></cdf:platform>
      <cdf:version update="http://iase.disa.mil/stigs">001.005</cdf:version>
      <cdf:metadata>
            <dc:creator>DISA</dc:creator>
            <dc:publisher>DISA</dc:publisher>
            <dc:contributor>DISA</dc:contributor>
            <dc:source>STIG.DOD.MIL</dc:source>
      </cdf:metadata>
      <cdf:Profile id="xccdf_mil.disa.stig_profile_MAC-1_Classified">
            <cdf:title>I - Mission Critical Classified</cdf:title>
            <cdf:description>&lt;ProfileDescription&gt;&lt;/ProfileDescription&gt;</cdf:description>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15768" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15771" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15772" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15774" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15775" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15776" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15778" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15779" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15983" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15985" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15986" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-19742" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-19743" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-19744" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-64891" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-79053" selected="true"></cdf:select>
      </cdf:Profile>
      <cdf:Profile id="xccdf_mil.disa.stig_profile_MAC-1_Public">
            <cdf:title>I - Mission Critical Public</cdf:title>
            <cdf:description>&lt;ProfileDescription&gt;&lt;/ProfileDescription&gt;</cdf:description>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15768" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15771" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15772" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15774" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15775" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15776" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15778" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15779" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15983" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15985" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-15986" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-19742" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-19743" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-19744" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-64891" selected="true"></cdf:select>
            <cdf:select idref="xccdf_mil.disa.stig_group_V-79053" selected="true"></cdf:select>
      <cdf:Group id="xccdf_mil.disa.stig_group_V-15768" Id="xccdf_mil.disa.stig_group_V-15768">
            <cdf:title>FireFox Preferences – Verification</cdf:title>
            <cdf:description>&lt;GroupDescription&gt;&lt;/GroupDescription&gt;</cdf:description>
            <cdf:Rule id="xccdf_mil.disa.stig_rule_SV-16707r1_rule" weight="10.0" Id="xccdf_mil.disa.stig_rule_SV-16707r1_rule" severity="medium">
                  <cdf:version update="http://iase.disa.mil/stigs">DTBF050</cdf:version>
                  <cdf:title>FireFox is configured to ask which certificate to present to a web site when a certificate is required.</cdf:title>
                  <cdf:description>&lt;VulnDiscussion&gt;When a web site asks for a certificate for user authentication, Firefox must be configured to have the user choose which certificate to present. Websites within DOD require user authentication for access which increases security for DoD information. Access will be denied to the user if certificate management is not configured.&lt;/VulnDiscussion&gt;&lt;FalsePositives&gt;&lt;/FalsePositives&gt;&lt;FalseNegatives&gt;&lt;/FalseNegatives&gt;&lt;Documentable&gt;false&lt;/Documentable&gt;&lt;Mitigations&gt;&lt;/Mitigations&gt;&lt;SeverityOverrideGuidance&gt;&lt;/SeverityOverrideGuidance&gt;&lt;PotentialImpacts&gt;&lt;/PotentialImpacts&gt;&lt;ThirdPartyTools&gt;&lt;/ThirdPartyTools&gt;&lt;MitigationControl&gt;&lt;/MitigationControl&gt;&lt;Responsibility&gt;System Administrator&lt;/Responsibility&gt;&lt;IAControls&gt;&lt;/IAControls&gt;</cdf:description>
                  <cdf:reference>
                        <dc:publisher>DISA</dc:publisher>
                        <dc:identifier>205</dc:identifier>
                        <dc:type>DPMS Target</dc:type>
                  </cdf:reference>
                  <cdf:ident system="http://iase.disa.mil/cci">CCI-001274</cdf:ident>
                  <cdf:fixtext fixref="F-15985r1_fix">Set the value of "security.default_personal_cert" to "Ask Every Time".  Use the Mozilla.cfg file to lock the preference so users cannot change it.</cdf:fixtext>
                  <cdf:fix id="F-15985r1_fix"></cdf:fix>
                  <cdf:check system="http://oval.mitre.org/XMLSchema/oval-definitions-5">
                        <cdf:check-content-ref name="oval:mil.disa.stig.mozilla.firefox.windows:def:20" href="U_Mozilla_Firefox_Windows_V1R5_STIG_SCAP_1-2_Benchmark-oval.xml"></cdf:check-content-ref>
                  </cdf:check>
            </cdf:Rule>
      </cdf:Group>
<cdf:TestResult id="xccdf_mil.disa.stig_testresult_scap_mil.disa.stig_comp_U_Mozilla_Firefox_Windows_V1R5_STIG_SCAP_1-2_Benchmark-xccdf.xml---xccdf_mil.disa.stig_profile_MAC-1_Classified-1" start-time="2020-04-28T16:34:31" version="001.005" end-time="2020-04-28T16:34:31" test-system="cpe:/a:spawar:scc:5.3">
            <cdf:benchmark id="xccdf_mil.disa.stig_benchmark_Mozilla_Firefox_Windows" href="#scap_mil.disa.stig_comp_U_Mozilla_Firefox_Windows_V1R5_STIG_SCAP_1-2_Benchmark-xccdf.xml"></cdf:benchmark>
            <cdf:organization>NIWC Atlantic</cdf:organization>
            <cdf:profile idref="xccdf_mil.disa.stig_profile_MAC-1_Classified"></cdf:profile>
            <cdf:target-id-ref name="SCC_DESKTOP-LAISPM9" href="" system="http://scap.nist.gov/schema/asset-identification/1.1"></cdf:target-id-ref>
            <cdf:platform idref="cpe:/a:mozilla:firefox:::~~~windows~~"></cdf:platform>
            <cdf:rule-result idref="xccdf_mil.disa.stig_rule_SV-16707r1_rule" weight="10.0" version="DTBF050" severity="medium" time="2020-04-28T16:34:31">
                  <cdf:result>fail</cdf:result>
                  <cdf:ident system="http://iase.disa.mil/cci">CCI-001274</cdf:ident>
                  <cdf:fix id="F-15985r1_fix"></cdf:fix>
                  <cdf:check system="http://oval.mitre.org/XMLSchema/oval-definitions-5">
                        <cdf:check-content-ref name="oval:mil.disa.stig.mozilla.firefox.windows:def:20" href="#scap_mil.disa.stig_comp_U_Mozilla_Firefox_Windows_V1R5_STIG_SCAP_1-2_Benchmark-oval.xml"></cdf:check-content-ref>
                  </cdf:check>
            </cdf:rule-result>
            <cdf:rule-result idref="xccdf_mil.disa.stig_rule_SV-16710r3_rule" weight="10.0" version="DTBF105" severity="medium" time="2020-04-28T16:34:31">
                  <cdf:result>fail</cdf:result>
                  <cdf:ident system="http://iase.disa.mil/cci">CCI-000381</cdf:ident>
                  <cdf:fix id="F-15988r3_fix"></cdf:fix>
                  <cdf:check system="http://oval.mitre.org/XMLSchema/oval-definitions-5">
                        <cdf:check-content-ref name="oval:mil.disa.stig.mozilla.firefox.windows:def:40" href="#scap_mil.disa.stig_comp_U_Mozilla_Firefox_Windows_V1R5_STIG_SCAP_1-2_Benchmark-oval.xml"></cdf:check-content-ref>
                  </cdf:check>
            </cdf:rule-result>
</cdf:TestResult>
</cdf:Benchmark>

这是我到目前为止得到的。它只是写入我在“xccdf_head”中指定的值。有更多的值需要提取，但我想我希望在构建之前先写入格式

import csv
import xml.etree.ElementTree as ET

# file to read
tree = ET.parse("./test.xml")
root = tree.getroot()

# create file for writing
xccdf_out = open('./test.csv', 'w')
csvwriter = csv.writer(xccdf_out)
xccdf_head = ['ID']
csvwriter.writerow(xccdf_head)

for group in root.findall('cdf:Benchmark'):
        row = []
        group_id = group.find('cdf:Group').find('cdf:title').text
        row.append(group_id)
        csvwriter.writerow(row)
xccdf_out.close()

我正在寻找一个可以从两个部分（cdf:Group和cdf:Result）捕获以下内容的输出

'cdf:groupid', cdf:ruleid','cdf:ruleseverity','cdf:title','cdf:description','cdf:result'

…所以理想情况下，它看起来是这样的：

xccdf_mil.disa.stig_group_V-79053, xccdf_mil.disa.stig_rule_SV-93759r3_rule, medium, Background submission of information to Mozilla must be disabled, VulnDiscussion>There should be no background submission(...),fail

Answer 1

另一种方法

from simplified_scrapy import SimplifiedDoc,utils
html = utils.getFileContent('test.xml') # Get xml data from file
doc = SimplifiedDoc(html)

rows = [['cdf:groupid', 'cdf:ruleid','cdf:ruleseverity','cdf:title','cdf:description','cdf:result']]
groups = doc.selects('cdf:Group') # Get all groups
for group in groups:
  rule = group.select('cdf:Rule') # Get the rule
  # Get result by rule id
  result = doc.select('cdf:TestResult').getElement('cdf:rule-result',attr='idref',value=rule.id).select('cdf:result>text()')
  rows.append([group.id,rule.id,rule.severity,rule.select('cdf:title').text,rule.select('cdf:description').text,result])

utils.save2csv('./test.csv',rows) # Save to csv file

这里有更多的例子：https://github.com/yiyedata/simplified-scrapy-demo/tree/master/doc_examples